Skip to content

v0.2.0: RAG module + TS router

Latest

Choose a tag to compare

@MarkusIsaksson1982 MarkusIsaksson1982 released this 16 Oct 13:23
· 7 commits to main since this release

What's new in v0.2

  • RAG module (agent_trace_evals/rag/, numpy + fastembed, CPU only): BM25 and dense retrieval (multilingual MiniLM), hybrid fusion (RRF), cross-encoder rerank, context packing, citation check, lexical faithfulness, and one JSONL trace per query.
  • Rerankers: BAAI/bge-reranker-base (MIT) is the default. jina-reranker-v2-base-multilingual is included as a comparison only, because its licence is non-commercial (CC-BY-NC-4.0).
  • Refusal gate: low rerank scores are refused before the LLM is called (bge threshold −6.5).
  • End-to-end benchmark with a free LLM over 30 queries. The bge pipeline with the gate scored answer accuracy 1.00, refused 4/4 unanswerable queries with 0 false refusals, faithfulness 0.885, and latency p50/p95 4.9/7.9 s. Retrieval with bge rerank: R@1 0.846, MRR 0.917.
  • Also new: a stdlib RAG eval baseline (rag_eval.py), a strict-TS tool router, and a TS port of the retrieval and metrics.
  • Tests: 42 Python and 15 TypeScript tests, all offline; they re-check the published numbers against the recorded traces.

Caveats: a 64-document synthetic corpus, 30 queries, and one run per configuration. The thresholds and settings were tuned on these same queries; there is no held-out set. bge rerank does not beat dense retrieval alone on this set. This is a demonstration of the engineering pattern, not a benchmark.

Data in rag_data/ is CC0 (see rag_data/LICENSE-DATA); the code is MIT.