Repository navigation
What's new in v0.2
- RAG module (
agent_trace_evals/rag/, numpy + fastembed, CPU only): BM25 and dense retrieval (multilingual MiniLM), hybrid fusion (RRF), cross-encoder rerank, context packing, citation check, lexical faithfulness, and one JSONL trace per query. - Rerankers:
BAAI/bge-reranker-base(MIT) is the default.jina-reranker-v2-base-multilingualis included as a comparison only, because its licence is non-commercial (CC-BY-NC-4.0). - Refusal gate: low rerank scores are refused before the LLM is called (bge threshold −6.5).
- End-to-end benchmark with a free LLM over 30 queries. The bge pipeline with the gate scored answer accuracy 1.00, refused 4/4 unanswerable queries with 0 false refusals, faithfulness 0.885, and latency p50/p95 4.9/7.9 s. Retrieval with bge rerank: R@1 0.846, MRR 0.917.
- Also new: a stdlib RAG eval baseline (
rag_eval.py), a strict-TS tool router, and a TS port of the retrieval and metrics. - Tests: 42 Python and 15 TypeScript tests, all offline; they re-check the published numbers against the recorded traces.
Caveats: a 64-document synthetic corpus, 30 queries, and one run per configuration. The thresholds and settings were tuned on these same queries; there is no held-out set. bge rerank does not beat dense retrieval alone on this set. This is a demonstration of the engineering pattern, not a benchmark.
Data in rag_data/ is CC0 (see rag_data/LICENSE-DATA); the code is MIT.