Jev RAG: vector-free local retrieval benchmark (BM25 vs agentic lexical + Jev)

[aifabrice]

I built Jev RAG, an independent local-first knowledge search application, to test how far a retrieval pipeline can go without first building an embedding index or running a vector database.

Repository: https://github.com/aifabrice/jev-rag
Interactive benchmark: https://aifabrice.github.io/jev-rag/

What the pipeline does

The default path is deliberately small and inspectable:

1. Index a local folder with SQLite FTS5/BM25.
2. Shortlist candidate passages locally.
3. Ask Jev batched Noul questions about passage relevance and whether the passage contains concrete answer-supporting evidence.
4. Let Python own thresholds, ordering, caching, and the final evidence boundary.
5. Stream a cited answer from the configured answer model.

I also added an agentic lexical mode: a planning model proposes multiple bounded keyword searches, local BM25 executes them, RRF fuses the results, and Jev reranks the final candidate pool. It still creates no embedding index. An optional embedding-hybrid mode is included as a comparison baseline.

Complete NFCorpus result

The public evaluation uses the complete BEIR NFCorpus test split: 3,633 documents and 323 queries.

Pipeline	nDCG@10	MRR@10	Recall@10
BM25 top 50 + Jev	0.362468	0.593023	0.164474
Agentic lexical top 50 + Jev	0.430969	0.644041	0.204138
Hybrid top 50 + Jev	0.444327	0.654583	0.214907

Agentic lexical + Jev was 0.013358 nDCG@10 behind hybrid + Jev. I do not claim they are equivalent: a 20,000-sample paired bootstrap for agentic minus hybrid produced a 95% interval of [-0.025480, -0.001697], still favoring the hybrid pipeline.

Exact commands, JSON summaries, observed provider usage/cost, latency caveats, and per-query outputs are published here:

https://github.com/aifabrice/jev-rag/blob/main/benchmarks/NFCORPUS_RESULTS.md

What I would value feedback on

• Is per-passage Noul the right primitive for this reranking boundary, or would a listwise Choice stage be a better default?
• Is the candidate-pool comparison fair enough to support the architectural conclusion?
• Which public dataset should be the next evaluation target beyond NFCorpus?
• Where should the application draw the line between a local-first product and a remote decision layer?

The project documents exactly what text is sent to Jev, the planner, embedding provider, and answer model. It is not fully offline, and it is not affiliated with or endorsed by TypeSafe AI, OpenRouter, or MiniMax.

I am the author and maintainer; criticism of the benchmark protocol is especially welcome.

[aifabrice]

Benchmark update (2026-09-30): I expanded the comparison beyond the three original paths and kept the same NFCorpus corpus, 323 test queries, qrels, and metric implementation.

A few results changed how I would frame the design:

• Agentic Hybrid + local Jev/retrieval rank fusion is now the strongest measured path at 0.450750 nDCG@10 and 0.220885 Recall@10.
• A unified Jev Passage Gate was a useful negative result: with fixed cookbook-inspired thresholds it excluded 84.4% of candidates and reached 0.376298 nDCG@10, below bare Hybrid (0.396712) and ordinary Hybrid + Jev (0.444327).
• Two-level Jev Line Search had the strongest first-hit behavior (nDCG@1 0.554180), but its full-corpus cold provider cost was USD 4.553288, so I now treat it as an experimental deep-search path rather than the default.

The seven selectable pipelines, exact configurations, costs, latency caveats, and reproduction commands are summarized here:
https://aifabrice.github.io/jev-rag/seven-pipelines.html

The main question I am still trying to answer is where the Jev boundary should sit: independent batched Noul judgments per passage, a listwise Choice stage, or a retrieval-prior fusion performed locally after Jev. Feedback from anyone testing the same boundary on another corpus would be especially useful.