Skip to content

v0.1.0 — the gate goes first

Choose a tag to compare

View all tags
@tonydzi tonydzi released this 04 Aug 19:28
· 17 commits to master since this release
v0.1.0
096cedd

Catch fabricated RAG citations before they reach the user. First tagged release — the code has been in the open since 24 July; from now on it has a version you can pin.

What it does

Two stages, and the cheap one goes first:

  1. The verbatim gate (gate.py) — pure re + substring matching over a normalized form. Zero dependencies, zero model calls. It answers the prior question a judge model never asks: does this quote physically exist in the document it is attributed to? Fabrications, frankenquotes (every word real, the sentence never written) and misattributed real quotes are rejected here, and misattributed stays a separate verdict from not_found because they need different fixes upstream.
  2. The skeptical judge (judge.py) — for quotes that do exist, a burden-of-proof prompt: the verdict starts at unsupported, outside knowledge is inadmissible, and support must match population, direction, magnitude and certainty or it caps at partial. Framework-agnostic — it plugs into whatever LLM your pipeline already speaks.

14 tests, each one a named failure mode rather than a coverage number — including test_audit_fabrication_never_calls_judge, which is the whole economic argument for the ordering.

In this release

  • verbatim_citation_gate.gate — audit_citation, normalization, the three rejection verdicts.
  • verbatim_citation_gate.judge — the burden-of-proof prompt and verdict parsing.
  • Docs site: https://palo-alto-ai-research-lab.github.io/verbatim-citation-gate/
  • New here: CI now runs the suite on every push and weekly, on Linux/macOS/Windows × Python 3.9/3.11/3.13. Until today the tests existed and nothing ran them.

Known limits — read these before you rely on it

They are open issues, not footnotes:

  • #1 — non-Latin scripts. normalize() keeps only [a-z0-9%.], so in Cyrillic/CJK the comparison degenerates to digits and a fabricated quote can come back found. In those scripts the tool does not yet do its job. This is the top of the queue.
  • #4 — a one-word quote returns found; there is no minimum-evidence rule yet.
  • #2 — the corpus is re-normalized on every call (2.4s per miss on 300 docs).
  • #3 — no py.typed, so downstream type checkers see nothing.
  • #8 — not on PyPI yet; install from git.

Install

pip install "git+https://github.com/Palo-Alto-AI-Research-Lab/verbatim-citation-gate@v0.1.0"

What's next

The roadmap board is public and every open row is a claimable issue: https://github.com/users/Palo-Alto-AI-Research-Lab/projects/1 — issues labelled accepted are scoped, free to take, nobody is on them. Comment "claiming this" and it is yours for 7 days. You keep the copyright to your code; no CLA, ever.

From here on, every noticeable change ships as a new minor release — so this feed, not the commit graph, is the honest record of how mature this is.

Full Changelog: https://github.com/Palo-Alto-AI-Research-Lab/verbatim-citation-gate/commits/v0.1.0

Assets 2
Loading