feat: lexical citation verifier — flag bogus [N] cites in synthesized answers#22
Merged
Merged
Conversation
… answers
After the final synthesis, every [N] reference in the answer body is checked
against the extracted text of source N. Sentences are split, claim tokens
are extracted (lowercased, stop-words dropped, numbers and digit/letter
boundaries preserved), and recall is scored against each cited source.
Multi-cite sentences ([1][3]) are supported only when EVERY cited source
clears the threshold — a bogus cite buried in an otherwise-true sentence
is still flagged. No second LLM "judge" pass; this is pure-function lexical
scoring with zero new deps.
New flags: --strict-cites, --cite-min-recall=<0..1> (default 0.4),
--no-verify-cites. Env vars mirror. Verbose stream gets a verify line per
unsupported sentence; --json output gains a `verification` key; markdown
output appends a "## Citation health" footer only when something fails
(clean runs stay clean). AgentResult.verification + usage.{citationsTotal,
citationsSupported} added for library consumers.
39 new unit tests across all five exported helpers in verify.ts, plus
2 agent-loop integration tests (end-to-end bogus-cite flagging and
verifyCitations:false skip path), plus 4 CLI/config tests covering the
new flag plumbing. Suite goes from 212 → 251.
v0.5.0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
After the final synthesis, deepdive now checks every
[N]reference in the answer body against the extracted text of source N. Catches the dominant failure mode of cited-answer tools: a confident sentence pointing at a source that doesn't actually support it.[1][3]is supported only when every cited source clears the recall threshold — a bogus[3]buried in an otherwise-true sentence is still flagged.5h→["5","h"],5-hour→["5","hour"]) so numeric anchors survive paraphrase. Numbers and length-≥3 alphas survive; everything else gets dropped.## Citation healthfooter in the markdown, one warning line per unsupported sentence in--verbose, averificationkey in the--jsonpayload.What's added
src/verify.ts—verifyCitations+ four exported helpers, all pure--strict-cites,--cite-min-recall=<0..1>(default0.4),--no-verify-citesDEEPDIVE_STRICT_CITES,DEEPDIVE_CITE_MIN_RECALL,DEEPDIVE_NO_VERIFY_CITESverify.doneagent eventAgentResult.verification: VerificationReport | undefined+usage.{citationsTotal, citationsSupported}for library consumersWhat it explicitly is not (v1)
Test plan
test/verify.test.mjscovering all five exported helpers + faithful/hallucinated/multi-cite/threshold/strip-sources cases[1]for content not in the source →unsupported.length === 1) andverifyCitations: falseskip pathnpm run typecheckcleannpm run buildcleannpm test— 251/251 passing (up from 212)--helpsmoke test shows the three new flags