Hack Hydra, Track 1 — Enterprise Context + Ontology.
Turn 38,600 noisy enterprise artifacts into a queryable ontology in HydraDB, then answer questions over it — lookups, multi-hop reasoning, and correctly refusing when the answer is not in the data.
HERB's own paper says the bottleneck is retrieval, not generation: the best agentic RAG scores 32.96, and "existing methods struggle to conduct deep searches and retrieve all necessary evidence."
The reason is structural, and it is a single number: a HERB question needs a median of 52 supporting artifacts (min 11, max 683). Top-k vector or lexical retrieval cannot rank its way to 52 needles in 38,600 documents. We measured the lexical ceiling ourselves — see the BM25 row in the results table: even retrieving 500 artifacts per question, BM25 reaches only 0.550 recall, at 0.072 precision.
A graph does not rank. It walks to the complete connected evidence set:
38,600 artifacts -> ~1,300 per product -> ~5 conversations -> ~52
measured, exact recall 1.000, in-database measured: median 5 of ~132,
70.6% cited in full
Scope reduction is the mechanism, and the first stage is exact rather than approximate: every one of the 1,514 questions names its product, and we resolve that slot deterministically at 1514/1514, with no model and no embeddings.
The second stage happens at routing, not retrieval, and only where it earns its place.
Retrieval keeps perfect recall at the product bound (precision 0.058). Narrowing to
conversations — the real unit of evidence, measured — helps pr and hurts person and
company, so it is applied per question type. Honest remaining weakness: person sits at
0.303 against an oracle of 0.548 and pr at 0.240 against 0.932, so most of the
remaining loss is still retrieval breadth rather than answer logic.
1. Abstention is structural, not a confidence threshold.
46% of HERB (699 questions) is unanswerable by design. We abstain when the typed evidence
set is empty — ABSTAIN ⟺ E_typed(q) = ∅ — never on a model score.
This is testable, and we tested it. 19.4% of the benchmark (293 questions) is answerable
if and only if a role → commented-on → document edge exists: the same question template
appears 93 times answerable and 200 times unanswerable across 50 (product, document) slots,
and in zero of those slots is the same role on both sides. A top-k system always returns
something and must hallucinate on all 200. A graph can prove the edge is absent — in
0.11 s for all 293, batched.
The sharpest case: "demo URLs for <product>'s competitor products" is answerable for
exactly 10 products and unanswerable for 20, with zero products in both and zero exact-string
collisions. No rule that reads the question text can work. The structural discriminator —
a Url whose host is not sf-internal.slack.com and whose path contains /demo — scores
TP=10, FP=0, TN=20, FN=0 across all 30 products and recovers 30/30 gold URLs.
The design conclusion we would put in front of a judge: the precision of structural abstention equals the precision of your edge typing. Budget ingest effort on typed edges, not on abstention logic.
2. Identity is resolved structurally — because names are provably useless here. 530 employees share only 98 distinct names. "Hannah Taylor" is 10 different people; 98.9% of employees share a name with someone else. Fuzzy matching or an LLM asked "same person?" over-merges catastrophically and never notices.
We measured something stronger: 0 of 1,514 questions contain any employee name — not a
full name, not one of the 10 first names, not one of the 10 last names. Every person
reference in HERB is already an explicit eid_ string. So we do no clustering at all,
which makes transitive over-merging structurally impossible rather than merely avoided.
The masked-label ablation in src/hyrev/resolve/ablation.py is kept as a robustness
demonstration, not a pipeline stage.
3. Ontology alignment — the half of the brief most teams skip.
Instance-based field matching plus greedy 1-to-1 per source pair across
EnterpriseRAG-Bench's nine real sources (164 core fields), emitted as SSSOM with provenance
and per-mapping justification: results/erb_alignment.sssom.tsv.
We also checked the literature we were about to build on, and it did not survive: the
published claim that stable matching lifts OAEI Anatomy F1 from 0.182 → 0.832 does not
hold. Reproducing it, the isolated gain from the 1-to-1 constraint is +0.0197 F1, the
"stable" matcher is bit-identical to plain greedy, and linear_sum_assignment is a trap
that costs 0.832 → 0.687. We kept the constraint because it is 11 ms and flattens the
threshold cliff — not because the headline was true.
HydraDB is not a store we read rows out of. Retrieval itself runs inside the database
as a single algo.MSpaths call per question; the client only unpacks the returned paths.
| Capability | HydraDB feature | What breaks without it |
|---|---|---|
| Complete evidence sets (median 52) | algo.MSpaths batched many-to-many traversal |
falls back to top-k ranking — the published 32.96 ceiling |
| Structural abstention | bounded path existence; negatives cost the same as positives | abstention becomes a confidence threshold, i.e. a guess |
| Product/release scoping | typed edges + automatic property indexes | scope becomes a similarity heuristic |
| Identity without names | AUTHORED / MENTIONED_IN / PARTICIPATED_IN edges |
name matching, which provably over-merges here |
Measured on this machine: the full graph loads in 24 s, and 683×683 many-to-many
traversal at depth 3 runs in 1.6 s. There is no sourceValues cardinality cap — 12,000
values parse and execute in 623 ms.
Reported in two stages, because quoting only the first while answering from the second would be misleading.
Stage 1 — graph traversal (algo.MSpaths, executed in-database). This is what the
go/no-go gate tests: can the graph reach the complete evidence set at all?
| system | recall | precision | mean retrieved |
|---|---|---|---|
| BM25 lexical, k=10 | 0.062 | 0.269 | 10 |
| BM25 lexical, k=100 | 0.239 | 0.126 | 100 |
| BM25 lexical, k=500 | 0.550 | 0.072 | 500 |
| graph traversal | 1.000 | 0.058 | 1298 |
Gate (recall ≥ 0.85 and above BM25): PASS. Gold evidence averages 73.3 artifacts per question; traversal p50 709 ms.
Stage 2 — after per-type routing. This is the set answers are actually built from. It trades evidence recall for answer quality, deliberately and measurably: end-to-end F1 goes 0.306 → 0.352 with routing on, and retrieval precision 0.058 → 0.101.
| recall | precision | mean retrieved | |
|---|---|---|---|
| stage 1, product scope | 1.000 | 0.058 | 1298 |
| stage 2, routed | 0.686 | 0.101 | 617 |
Per question type, stage 2:
| type | n | recall | precision |
|---|---|---|---|
| company | 32 | 0.692 | 0.236 |
| content | 58 | 1.000 | 0.030 |
| person | 64 | 0.515 | 0.121 |
| pr | 41 | 0.464 | 0.070 |
| url | 5 | 1.000 | 0.072 |
How the second reduction was found, and what it cost. We measured the gold set's shape rather than guessing at filters: a question cites a median of 5 conversations out of ~132 per product, and 70.6% of touched conversations are cited in full (median purity 1.000). So the unit of evidence is the conversation, not the message.
Narrowing is then applied per question type, because one setting cannot serve all:
companynarrows to conversations referencing aCUST-id — 2.4× precision at identical answer F1, the cleanest win here;prroutes through theMENTIONS_PRrelation (Slack says "PR#7",/pull/24) for 0.100 → 0.240;persontakes a light lexical narrowing for 0.255 → 0.275;urlis left alone because a typed property already answers it.Three narrowings measured and rejected, kept with their numbers so nobody re-derives them: filtering to the artifact kind the question names (
prrecall 1.000 → 0.081 — the question's noun is the answer type, not the evidence type), a release date-window filter (precision +0.001 for recall −0.179), and message-levelCUST-filtering (precision 5× but recall 1.000 → 0.182).Remaining weakness, stated plainly:
prsits at 0.240 end-to-end against an oracle of 0.932, so evidence selection — not answer logic — is still the dominant loss.
| type | n | end-to-end F1 | oracle F1 (gold citations) | retrieval cost |
|---|---|---|---|---|
| company | 32 | 0.579 | 0.664 | +0.085 |
| person | 64 | 0.303 | 0.548 | +0.245 |
| pr | 41 | 0.240 | 0.932 | +0.692 |
| url | 5 | 0.437 | 1.000 | +0.563 |
| all | 142 | 0.352 | 0.701 | — |
llm_calls = 0, spend $0.00. The oracle column is the
same projection given gold citations, so the gap is exactly what retrieval costs.
F1 is extraction-free: HERB's harness pipes prose through gpt-4o to extract ids
first, and we emit exact sets instead, so these are an upper bound on that harness.
The end-to-end numbers above route each question to its answer projection using
HERB's gold type field. The shipped API cannot do that — it has only the
question text — so that column is a ceiling, not the pipeline. Both are now
measured:
| routing | ALL | company | person | pr | url |
|---|---|---|---|---|---|
gold type (ceiling) |
0.352 | 0.579 | 0.303 | 0.240 | 0.437 |
classify() (deployable) |
0.332 | 0.579 | 0.259 | 0.240 | 0.437 |
HYREV_CLASSIFY=1 python -m hyrev.eval.end_to_end reproduces the second row. |
|||||
The gap is 0.020 because classify() (src/hyrev/query/qtype.py) agrees with the |
|||||
| gold label on 795 of 815 answerable questions — 97.5%, and 100% on company, | |||||
content, pr and url. It was 80.9% before two regex bugs were fixed: \bpr\b |
|||||
| never matches "PRs", which sent 156 of 167 pr questions down the url route, and | |||||
| `\burl | link | demo\bparses as(\burl) |
(link) | (demo\b)`, so bare "link" matched | |
| "hyperlink". |
238 of the 815 answerable questions are prose. They are answered by a local model
(qwen2.5:7b via ollama), which keeps the project's zero-spend property intact:
$0.00 total API spend, ingest to answer.
| n | mean judge score | empty answers | spend |
|---|---|---|---|
| 58 | 52.2 / 100 | 7 | $0.00 |
Known defect: 7 of 58 answers came back empty and score 0. The distribution is bimodal — 37 of 58 score 60+ (median 72), while 21 fail outright, the empties among them. That is a generator bug worth fixing, not a tuning problem, and it drags the mean down from a median of 72 to a mean of 52.2.
Indicative only — do not read this as a benchmark result. The answers are generated and judged by the same local 7B model, which is self-evaluation. It is not the paper's gpt-4o judge and is not comparable to the published 32.96.
results/content.jsoncarriesindicative_only: true. What it does show is that the content path works end to end on real retrieved evidence, at no cost.
| TP | FP | TN | FN | accuracy | gold URLs recovered |
|---|---|---|---|---|---|
| 10 | 0 | 20 | 0 | 1.000 | 30/30 |
Rule: a Url whose host is not sf-internal.slack.com and whose path contains
/demo. The untyped variant — "does any demo URL exist" — scores 33%.
Every number above is reproducible via just eval, from committed configs in results/.
Measurement discipline. Retrieval is tuned on a frozen 200-question dev subset whose
ids are committed in results/dev_subset.json. We never tune on the full set and report the
same number.
One honest caveat about the retrieval metric. HERB never scores retrieval — citations
is carried through its code but never compared against anything, and the published 32.96 is
an answer score. Our recall figure is therefore a metric we define, and it is not
comparable to any published number. That is precisely why the BM25 baseline is here: it is
the only baseline that exists.
Requires Docker and uv. ~1 GB RAM for the stack.
git clone https://github.com/yrevash/hyrev && cd hyrev
# 1. datasets (gitignored, not redistributed — see licences below)
mkdir -p .research/repos && cd .research/repos
git clone --depth 1 https://github.com/SalesforceAIResearch/HERB.git
cd ../..
# 2. HydraDB + MinIO
just up
# 3. python env
uv venv --python 3.12 hv && uv pip install --python hv/bin/python -r requirements.txt
# 4. build the graph, then evaluate
just ingest
just evalUse MinIO, not
CLOUD_PROVIDER=local. Onlocal, oncegraph-indexerpublishes a generation for a cell, every subsequent write to that cell fails permanently (PutMode::Update not yet implemented by LocalFileSystem).just upuses MinIO and avoids this.graph-indexeris mandatory — it is worth 8–15× on traversal.
- HydraDB — https://github.com/hydra-db/hydradb (AGPL-3.0), used as a database service via Bolt/HTTP. We do not vendor or link its code.
- Salesforce HERB — https://huggingface.co/datasets/Salesforce/HERB (CC-BY-NC-4.0). Choubey et al., "Benchmarking Deep Search over Heterogeneous Enterprise Data", arXiv:2506.23139. Not redistributed; cloned at setup. HERB was generated with GPT-4o; per its terms it "should not be used to develop models that compete with OpenAI." We train no models.
- EnterpriseRAG-Bench — https://github.com/onyx-dot-app/EnterpriseRAG-Bench, arXiv:2605.05253. Used for the ontology-alignment demonstration.
- Libraries:
rank-bm25,fastapi,uvicorn. HydraDB is reached over its HTTP query API usingurllibfrom the standard library — no driver dependency.
See HANDOVER.md for what remains (demo video, submission form, push) and the honest list of known weaknesses.
This project is MIT licensed — see LICENSE.