kestrel: Stop asking the model that hallucinates to verify its own hallucinations

[grloper]

kestrel

A research engine that can't cite what a source never said.

_{dsh-deep-research · DeepSeek Harness plugin & standalone library · Zero runtime dependencies}

Repository · Quick Start · Live Demo · Catalog PR #361 (Merged)

---

Repository: https://github.com/grloper/dsh-deep-research
By grloper (@grloper)

Most research tooling optimizes for plausible prose with URLs attached. Two failure modes survive any amount of prompt engineering because they are structural rather than stylistic:

1. The cited page often does not state the sentence attached to it. Nobody checks entailment, so a URL near a claim is treated as proof.
2. "12 sources agree" is usually one press release and eleven syndications. Source count is used as a proxy for truth, and independence is never computed.

Kestrel attacks both with code rather than with prompting.

---

1. A fabricated quote fails indexOf

The entailment judge is required to return a verbatim quote. That quote is then located in the fetched source text by substring match — exact first, then normalized for whitespace and punctuation, with the match projected back to original character offsets.

If the quote cannot be located, the citation is rejected and the claim is demoted to UNVERIFIED.

No model is asked whether the citation is good. Asking the thing that hallucinates to check its own hallucination is circular; String.indexOf is not. A dishonest judge cannot manufacture support — it can only fail the gate.

Here is the demo catching a fabrication that differs from the truth by one digit:

ADMITTED  (genuine)    "its Q3 revenue reached 42 million dollars"
          kind=EXACT span=[40,81]

REJECTED  (fabricated) "its Q3 revenue reached 91 million dollars"
          kind=FAILED  best token overlap 0.75 < 0.85

Token overlap is 0.75 — comfortably high enough to slip past any similarity threshold a naive implementation would pick. The exact-match gate rejects it regardless.

---

2. Corroboration is counted in origins, not documents

3-gram shingles → MinHash → LSH candidate filtering → pairwise Jaccard. Above the syndication threshold a document is a verbatim copy; above the derivation threshold with a shared long quote it is a rewrite. Those edges form a lineage DAG, cycles are condensed with Tarjan's algorithm, and the roots are counted.

Edge direction is fixed by min(publishedAt, waybackFirstSeen), because publishers rewrite their own dates and the archive timestamp is the one they do not control.

On the demo corpus (one press release, six verbatim syndications, four rewrites, one independent regulatory filing, one correction):

Raw source count      : 13
Independent origins   : 3

Every source-counting tool reports "13 sources agree." Three actually do.

---

3. A contradiction is not averaged away

SUPPORTED   89% confidence   ICS 2 / 12 sources
  Northwind Robotics reported Q3 revenue of 42 million dollars.

PARTIAL     40% confidence   ICS 1 / 11 sources
  Northwind Robotics grew 18 percent year over year.

Eleven sources "support" the 18% growth claim. One primary filing retracts it. Confidence is capped by the contradiction rather than inflated by the crowd — the opposite of what majority-vote retrieval does.

---

4. It never silently produces nothing

The three mechanisms above are all gated on a judge returning a verbatim quote that is then anchored mechanically. That gate is right — but it makes the fallback judge load-bearing in a way that is easy to get catastrophically wrong, and this engine got it wrong.

When a host had no llm service, the judge fell back to a stub returning NEUTRAL with an empty quote for every document. The Tribunal admits only SUPPORTED/PARTIAL/CONTRADICTED evidence, so that stub discarded 100% of everything retrieved. The loop then spent its full round budget, found nothing by construction, and reported "0 resolved". If you activated deep_research and felt like nothing happened, this is why.

A stub that guarantees zero output is not graceful degradation. It is a silent failure. The LLM-free path is now a real judge:

• IDF-weighted sentence selection over the document's own sentences, so a term appearing in every sentence carries little weight while a rare, specific term carries a lot. A page that merely mentions the subject no longer scores like one that addresses the assertion.
• Normalized figure matching — 8, 8% and 8.0 compare equal. (The first cut of this had a real bug: it extracted "8%" and compared it against a claim's "8", so the numeric bonus silently never fired. Caught by reading a failing test's actual values instead of assuming the test was wrong.)
• Stance detection weighing negation cues (debunked, failed to replicate, found no) against affirmation cues, so refutation is captured rather than collapsed into support.
• Structurally honest — the quote is selected from the document, never generated, so it passes the anchoring gate by construction and cannot fabricate a citation. Below a relevance floor it returns an honest NEUTRAL.

Alongside it, a capability preflight separates three failures that used to look identical:

Condition	Before	Now
No search service	empty "0 resolved" report	explicit "research could not run: no web-search capability"
No LLM	empty "0 resolved" report	full run via the lexical judge, degradation stated in the trace
Genuinely no evidence	empty "0 resolved" report	unresolved, with the queries that were tried

A capability problem is now reported as a capability problem, never as an absence of evidence. Verified end to end: with no LLM at all a run admits real anchored evidence and cites real sources; a lying LLM judge still cannot inject a fabricated quote; and an irrelevant corpus is reported as unresolved rather than invented into support.

Same pass also fixed: question decomposition that severed predicates ("Compare Rust and Go for backend services" became two corrupted fragments issued verbatim as search queries), status buckets that could print 0 resolved · 0 contested · 0 unresolved for work that had actually run, deep/forensic modes quitting at round 2 of 6/12 because follow-up queries never varied, and paywalled sources being dropped entirely despite carrying quotable snippet text.

---

Tools registered in DSH

Tool	Purpose
verify_text	Fact-check any block of text claim by claim
deep_research	Adversarial investigation with a prosecutor hunting disconfirming evidence (quick/standard/deep/forensic)
compare_sources	Given N URLs on one story, how many are genuinely independent
check_source	Source type, credibility signals, publication-date reliability
research_recall	Query prior findings, excluding those past their volatility horizon

Freshness is per-claim by volatility class — a mathematical constant and a stock price must not expire on the same schedule (5 years / ~6 months / 24 hours).

---

Try it in 60 seconds — no API key, no network

git clone https://github.com/grloper/dsh-deep-research.git
cd dsh-deep-research
npm run demo

The demo runs the real pipeline over a fixed corpus. Every number above is its actual output.

Install into DeepSeek Harness

dsh plugin --profile web add github:grloper/dsh-deep-research

---

Engineering highlights

• Zero runtime dependencies. Nothing to audit, no supply chain risks.
• 252 tests, CI matrix passing on Node 20, 22, and 24 across Ubuntu and Windows.
• Every capability is injected — search, fetch, and LLM arrive through adapters, so the engine is fully testable offline. With no LLM available it falls back to a deterministic lexical judge (see below) rather than going quiet, and a missing search service is reported as a missing search service instead of an empty result.
• Browser-Host RPC: The browser never adjudicates. The UI calls harness.handle RPC back to the host engine and renders only what was proved mechanically.
• Persistent evidence store migration: Automatic zero-loss migration moves legacy .dsh/veritas/ evidence to .dsh/kestrel/, preserving SQLite WAL and SHM sidecars so prior findings never get lost across updates.

---

Why "Kestrel"?

A kestrel hunts by hovering — holding station in the air, dead still, until it sees exactly what is there. Then it commits, once.

That is the opposite of how research agents usually behave: grab the first ten results, summarize confidently, attach URLs. This engine is built to hold position over the evidence and only strike when the quote is actually there.

Feedback and issues welcome at github.com/grloper/dsh-deep-research!