A hybrid-retrieval RAG assistant over a 216-document York University policy corpus, that cites its sources and refuses when the corpus can't support an answer. Runs fully locally — no API keys, no cloud, $0.
The GIF is sped up 8×. Real measured latency is 3.2 s of retrieval plus local CPU generation, which is slow and honestly reported below rather than hidden. Everything shown is the running system answering real queries — retrieval, citations and refusals are genuine, nothing is mocked up. Reproduce with
bash scripts/run_demo.sh(needs Ollama).
| Grounded answer, with citations | Honest refusal when the corpus can't support an answer |
|---|---|
 |
 |
The default path is linear: retrieve with both a lexical and a dense index, fuse the ranked lists, rerank, gate, then answer with citations. The adaptive branches exist behind flags and are off by default — see What still doesn't work.
flowchart TD
Q[User query] --> BM25["BM25 lexical · top-20"]
Q --> VEC["Vector dense · top-20"]
BM25 --> RRF["RRF fusion · k=60"]
VEC --> RRF
RRF --> CE["Cross-encoder rerank · top-6"]
CE --> GATE{"Refusal gate<br/>score ≥ threshold?"}
GATE -->|yes| LLM["LLM answer + citations"]
GATE -->|no| REF["Refuse: insufficient evidence"]
style GATE fill:#009E73,color:#fff
Every stage emits a ScoredChunk carrying its own ScoreType, because BM25,
cosine, RRF and cross-encoder scores are on four incompatible scales. The gate
picks its threshold from the score's type; comparing them naively refused 100% of
queries (see DECISIONS.md ADR-11).
| Component | Technology |
|---|---|
| LLM | llama3.2:3b via Ollama (local, CPU) |
| Embeddings | all-MiniLM-L6-v2 via SentenceTransformers (local, ~90 MB) |
| Reranker | cross-encoder/ms-marco-MiniLM-L-6-v2 (local, ~117 MB) |
| Vector store | ChromaDB (persistent, cosine HNSW) |
| BM25 | bm25s (persisted inverted index) |
| Web | Next.js 16 + Auth.js + Prisma/SQLite |
| API | FastAPI (SSE streaming), internal-only |
| Deployment | Docker Compose · CI via GitHub Actions |
Hand-written, paraphrased eval set. Corpus: 216 documents → 2,864 chunks. Methodology and caveats: docs/EVALUATION.md.
These numbers are also reproduced, and extended to 248 questions with component-level failure attribution, by a separate harness: york-rag-eval. It runs this system behind an adapter, splits every failure into a retrieval or generation cause, and gates regressions in CI.
| Metric | Value | n | 95% CI |
|---|---|---|---|
| Hit-rate@5 (a chunk of the right policy in the top 5) | 78% | 36 | [64%, 92%] |
| End-to-end success (hit and gate open) | 44% | 36 | [30%, 60%] |
| False refusal (retrieved, but gate closed) | 36% | 36 | [22%, 52%] |
| Reranker demotion (in pool, ranked out of top-5) | 17% | 36 | [8%, 32%] |
| Retrieval miss (neither retriever had it) | 3% | 36 | [0%, 14%] |
| Out-of-scope refusal | 70% | 40 | [55%, 82%] |
Retrieval is not the bottleneck — 35 of 36 questions have the correct policy somewhere in the candidate pool. The refusal gate and the reranker account for essentially all of the failures.
Per-stage ablation, hit-rate@5 (data/eval/ablation_report.md):
| Mode | Hit-rate@5 | 95% CI |
|---|---|---|
bm25_only |
50% | [33%, 67%] |
vector_only |
78% | [64%, 92%] |
hybrid_rrf |
78% | [64%, 89%] |
full_rerank |
78% | [64%, 92%] |
Hybrid retrieval does not measurably beat dense retrieval alone here. Vector, hybrid and reranked all land on the same number, and BM25 alone is 28 points behind. In the failure analysis, 7 failing questions had the answer only in the vector candidate list and 0 only in the BM25 list. On paraphrased queries over this corpus the lexical half is carrying its weight in latency, not in results. It is retained because it is nearly free (0.9 ms) and lexical matching is the plausible failure mode for exact policy-number or defined-term lookups, which this eval set does not probe — but that is a hypothesis, not a measurement.
At these sample sizes the intervals are tens of points wide. Differences narrower than the intervals are not established, and nothing here should be read as a precise figure.
Retrieval costs 3.2 s per query on CPU, of which the cross-encoder is 98%
(BM25 1.0 ms, vector 60 ms, RRF 0.2 ms, cross-encoder 3,091 ms). LLM generation is
timed separately and dominates end-to-end. Full breakdown:
data/eval/latency_report.md.
cp .env.example .env # set API_SHARED_SECRET, or export ALLOW_UNAUTHENTICATED=1
pip install -e ".[dev]" && ollama pull llama3.2:3b
python -m scripts.ingest # index data/documents/ (~10 min on CPU)
python -m api.main & # RAG service on 127.0.0.1:8000
cd web && npm ci && npx prisma migrate dev && npm run dev # UI on :3000Docker: docker compose up --build (only the web app publishes a host port).
Legacy Streamlit UI: SERVICE=streamlit, or docker compose --profile streamlit up.
Chunks were sized at 1000 tokens of cl100k_base — OpenAI's tokenizer, which no
model in this system uses. The embedder truncates at 256 WordPiece tokens, so most
of every chunk was silently discarded before embedding and could never be
retrieved. Nothing errored. Nothing warned. The index was just quietly wrong, and
so was every number measured on it.
Fixing it (size chunks with the embedding model's own tokenizer; Settings now
refuses to construct if chunk_size exceeds the embedder's usable context) moved
end-to-end success from 28% to 44% — and reversed both of this project's
previously published "negative results":
| Claim, pre-fix | Post-fix reality |
|---|---|
| "The cross-encoder is net-negative for recall" | No longer negative — but the gain is inside the noise (78% vs 78% hit-rate@5) |
| "The fitted gate threshold degenerates to τ ≈ 0" | It doesn't; τ is meaningful on every fold |
Both were artefacts. A cross-encoder scoring the first third of each chunk really does demote correct answers, and a signal that weak really does have nothing to threshold on — the measurements were faithfully reporting a broken pipeline. Note the correction cuts both ways: the reranker is no longer harmful, but that is not the same as useful, and the honest current answer is "we can't tell at this sample size". Every number here was regenerated rather than carried forward.
- Out-of-scope refusal is 70%, not 100%. The earlier "100%" came from 6 probes. On 40 harder ones — other universities' policies, external legislation, plausible-sounding but non-existent York policies, injection attempts — the gate answers about a third of questions it has no evidence for.
- BM25 contributes nothing measurable on this eval set (see the ablation above). Hybrid retrieval is a defensible default, not a demonstrated win.
- Fitting the cross-encoder threshold is worse than leaving it at 0.3 (Youden J 0.048 vs 0.200). The vector similarity signal is the one that beats the baseline (J 0.282). That's a lead, not a shipped change.
- Adaptive reranking is now indistinguishable from plain reranking — identical to three decimals. It adds a branch and a flag for no measured benefit, so it stays default-off.
- Everything above has overlapping confidence intervals. At n=36 / n=40 these are directional readings, not settled facts.
Evaluation. 36 in-scope + 40 out-of-scope questions — small, wide intervals, every figure an estimate. Ground truth is document-level, so the metric asks "did a chunk of the right policy surface?", not "did the best chunk surface?"; recall@k is capped by that label style and must not be compared across chunk sizes (docs/EVALUATION.md).
Auth. No email verification (emailVerified is in the schema and never
written) and no password reset. Sessions are revocable — each JWT is paired with
a Session row that is checked on every request, so "sign out everywhere"
invalidates live tokens immediately (web/lib/sessions.ts). The cost is one
indexed lookup per authenticated request.
Operations. Rate limiting and the inference cap are both in-process, so more
than one web replica needs Redis. TRUSTED_PROXY_DEPTH defaults to 0, meaning
x-forwarded-for is ignored — set it to the number of proxies you control or
rate limiting keys on "unknown". Single-process, single-model inference with no
batching; concurrency is capped at 2 in-flight retrievals, past which the API
returns 503.
Corpus. Scanned PDFs are not OCR'd — pages with no extractable text are logged and skipped. RAGAS evaluation needs an OpenAI key and is the one part of the project that is not $0 / local; it is not installed by default.
Citation rendering. The model is instructed to cite inline as
[Source: <filename>, page N, <chunk_id>], which is verbose and duplicates the
citation panel below the answer — visible in the screenshot above. It is that
explicit because it makes citations checkable in the eval; a friendlier format is
a prompt change that would need the grounding behaviour re-measured, so it has not
been made casually.
Tests, linting, and contribution workflow: CONTRIBUTING.md. Design rationale and the decisions that were reversed after measurement: DECISIONS.md.
src/ retrieval engine (chunking, bm25, vectorstore, retrieval, reranker, llm,
evaluation) · api/ internal FastAPI service · web/ Next.js app, the only
host-facing service · app/ legacy Streamlit UI · scripts/ ingestion and every
benchmark · data/eval/ labels, result JSONs, generated reports.
The three places a plausible-looking change silently breaks the system:
src/chunking/chunker.py (chunk length is denominated in the embedding model's
tokenizer), src/retrieval/scored.py (score scales are not interchangeable), and
src/config/settings.py (which enforces the first of those at startup).
MIT — see LICENSE. The York University policy documents under
data/documents/ are York's property and are not covered by it.