ragcite

Deterministic, framework-free evals for RAG and tool-using agents: retrieval metrics, citation grounding, abstention scoring, and tool-trajectory scoring.

No LangChain, no vector store, no LLM client is imported by this package. You bring your own retriever and (optionally) your own judge callable — ragcite scores what they return.

Install

Not on PyPI yet — install from the repository:

pip install "git+https://github.com/pcbeingused333/ragcite"
pip install "ragcite[cli] @ git+https://github.com/pcbeingused333/ragcite"  # + rich, for a live progress bar

Retrieval: hit@1, recall@k, MRR

import ragcite

def my_retriever(query: str) -> list[str]:
    # your retrieval code — return ranked chunk/doc ids, most relevant first
    ...

cases = ragcite.load_retrieval_cases("qa.jsonl")   # {"id", "query", "expected_ids"} per line
report = ragcite.evaluate_retrieval(my_retriever, cases)

print(report.hit_at_1_rate, report.recall_at_k, report.mrr)
print(ragcite.markdown_table([report]))

Grounding: did the answer cite what was actually retrieved?

No LLM call, no similarity score — a set-membership check between the ids an answer cites and the ids the retriever actually returned. Catches a fabricated citation regardless of how fluent or plausible it reads.

result = ragcite.check_grounding(
    id="q1",
    answer="The breach window is 72 hours [art-33].",
    retrieved_ids=["art-33", "art-5"],
)
result.grounded      # True
result.fabricated    # set() — empty means nothing was invented

Abstention: does it decline when it should?

Feed it questions your knowledge base genuinely can't answer, plus the system's real answer and what it actually retrieved. A judge model — never the system under test — classifies each reply as abstained / hedged / answered.

def my_judge_fn(prompt: str) -> str:
    # call whatever LLM client you already use; return its raw text reply
    ...

ragcite.check_judge_independence(model="llama-3.3-70b", judge_model="gpt-oss-120b")

cases = [ragcite.AbstentionCase(id="q1", question="...", answer="...", contexts=[...])]
report = ragcite.evaluate_abstention(cases, my_judge_fn)

report.abstained_rate
report.worth_reading   # everything that wasn't a clean, correct abstention

ragcite.parse_json_object / ragcite.strip_reasoning are the parsing helpers evaluate_abstention uses internally — reusable on their own for any LLM-as-judge prompt that asks for JSON back, including from reasoning models that inline a <think> block in the reply.

Tool trajectories: did the agent call the right tools, and are its numbers real?

For agents that call tools (booking, pricing, inventory, MCP servers) rather than just retrieving text. No LLM here either — tool selection, ordering, and arguments are checked structurally, and any money or quantity the final answer states is checked against what the tools actually returned.

import ragcite

trajectory = ragcite.Trajectory(
    calls=[ragcite.ToolCall(name="check_stock", args={"sku": "A1"}),
           ragcite.ToolCall(name="book", args={"sku": "A1", "qty": 5})],
    results=["on_hand: 80"],
    answer="Booked 5 units. 75 remain.",  # "75" was never in a tool result
)
case = ragcite.TrajectoryCase(
    id="order-1",
    trajectory=trajectory,
    required_tools=["check_stock", "book"],
    order_constraints=[("check_stock", "book")],
)
report = ragcite.evaluate_trajectories([case])
report.pass_rate            # 0.0 — see why:
report.failing[0].failures  # ["ungrounded_count: 75 not backed by any tool result"]

From the command line

pip install "git+https://github.com/pcbeingused333/ragcite"        # core: zero dependencies
pip install "ragcite[cli] @ git+https://github.com/pcbeingused333/ragcite"  # + rich

ragcite retrieval qa.jsonl --retriever mymodule:my_function --html report.html
ragcite grounding answers.jsonl --min-grounded-rate 0.95
ragcite abstention cases.jsonl --judge mymodule:my_judge
ragcite trajectory cases.json --min-pass-rate 0.9

grounding takes {"id", "answer", "retrieved_ids"} per line and needs no model and no network — it is a set check, so it is the one you can afford to run on every commit.

mymodule:my_function is an import path — a function in a module already importable from where you run the command, taking a query string and returning a ranked list of ids. Prints the same report as the library call, followed by rule-based recommendations, and (with --html) writes a self-contained report page. No rich? It falls back to a plain [3/20] scoring… line — the run still works, just without the bar.

Recommendations, without an LLM call

recs = ragcite.advise(report)          # works on any report type in this library
print(ragcite.render_text(recs))
# [HIGH] recall@k is 42% — the right passage often isn't retrieved at all...

Every rule here just turns reasoning already in this README into code — a big gap between recall@k and hit@1 means reranking, not chunking; an "answered" verdict in an abstention run is the expensive failure, not a "hedged" one. It tells you which number to look at first, never why it moved — that part still needs you.

HTML report

open("report.html", "w").write(ragcite.render_html([retrieval_report, abstention_report]))

One static file: the tables plus the recommendations above, no server and no JS. ragcite retrieval ... --html report.html writes the same thing.

CI gates: make the run actually fail

ragcite retrieval qa.jsonl --retriever mymodule:my_function --min-hit-at-1 0.8 --min-recall 0.85

Without a --min-* flag the command always exits 0 — it's a report, not a gate, and stays that way by default. Set one and a bad run exits 2, so a CI job can actually block a merge instead of only printing a table nobody reads. Same idea as a library call:

violations = ragcite.check_thresholds(report, min_hit_at_1=0.8)
ragcite.render_violations(violations)   # "FAIL: hit@1 is 62%, below the required 80%"

check_thresholds dispatches on report type, and every subcommand has the flags for its own:

Command	Gate flags
ragcite retrieval	--min-hit-at-1, --min-recall, --min-mrr
ragcite grounding	--min-grounded-rate
ragcite abstention	--min-abstained-rate, --max-answered-rate
ragcite trajectory	--min-pass-rate

Real-world example

examples/dogfood_gdpr.py runs evaluate_retrieval against the actual corpus and 25-question set from rag-chatbot-portfolio, the project this library's core was extracted from — not synthetic test fixtures. It scores two retrievers over the same 414 real GDPR provisions:

Retriever	hit@1	recall@k	MRR
stdlib-tfidf, k=10	4/25 (16%)	11/25 (44%)	0.22
production embeddings (BAAI/bge-small-en-v1.5, 1500/200, k=4) — 433 chunks	13/25 (52%)	17/25 (68%)	0.59

Going from the keyword retriever to the production embedding one changes a single function in that script — the point of scoring a callable rather than integrating with a vector store.

Run with --verify and the same index is scored twice: once by ragcite, once by that project's own independent metrics implementation. They match exactly (13/25, 17/25, 0.59), and the script exits non-zero if they ever stop matching. See examples/README.md for what that does and doesn't prove.

Why this instead of Ragas / TruLens

Ragas and TruLens are heavier and answer a different, harder question: "is this answer faithful to what it read?" — which needs an LLM to decompose the answer into claims. That's real and worth having, and nothing here stops you from also running one of those.

ragcite covers what's underneath that question and doesn't need an LLM to answer it:

• Retrieval (hit@1 / recall@k / MRR) is pure string/id matching — free, deterministic, safe to run in CI on every commit.
• Grounding (did it cite something it never retrieved?) is a set check, not a similarity score — it can't be fooled by a fabricated claim that happens to read well.
• Abstention does need a judge, but the judge is yours — bring any provider, and check_judge_independence stops the system-under-test and the judge from silently becoming the same model over time.

If retrieval or grounding is broken, a faithfulness score built on top of them is measuring noise. Run these first.

Status

v0.6.0 — retrieval, grounding, abstention, and tool-trajectory scoring, extracted and generalized from two production eval harnesses (a RAG agent and an MCP tool-calling agent), plus a CLI covering all three evals, a rule-based advisor, a static HTML report, and a CI threshold gate on every subcommand. 166 tests, CI on Python 3.9–3.12, plus a real-world (non-CI) example that scores the original project's actual corpus and question set with its actual embedding retriever, and checks ragcite's numbers against that project's own independent metrics code.

License

MIT