Upload your product docs, get a chat endpoint that answers with citations — and a faithfulness score.
A multi-tenant API (one document collection per customer). What sets it apart from the hundreds of "chat with your docs" demos: the retrieval is benchmarked. recall@5 is published for three configurations (vector only, hybrid, hybrid + reranking), and answer faithfulness is measured.
v1. Retrieval and generation are measured, and the eval questions have been reviewed by hand.
| Area | Status |
|---|---|
| Project setup: uv, ruff, mypy strict, pytest, CI | ✅ |
| Ingestion pipeline: parsing, section-aware chunking, batched embeddings | ✅ |
| Vector search on Postgres + pgvector, scores exposed | ✅ |
| Hybrid search (BM25 + vectors fused with RRF) | ✅ |
| Cross-encoder reranking, top-20 → top-5 | ✅ |
| Retrieval eval set, recall@k across configurations | ✅ |
| Chat with citations, honest refusals, faithfulness scored | ✅ |
| Multi-turn query rewriting | ✅ |
| LCEL rewrite and a measured framework comparison → v1 | ✅ |
Python 3.12 · uv · FastAPI · Pydantic v2 · Postgres + pgvector · SQLAlchemy / asyncpg · pytest · ruff · mypy strict · OpenRouter
uv sync
cp .env.example .env # add your OpenRouter key
docker compose up -d # Postgres + pgvector on port 5434
uv run pytest
uv run ruff check . && uv run mypyIndex a real corpus and probe it:
uv run uvicorn docpilot.main:app --reload --port 8001 # docs at /docs
uv run python scripts/ingest_corpus.py --limit 60 # FastAPI docs, ~$0.002
uv run python scripts/inspect_chunks.py # read what was indexed
uv run python scripts/probe_search.py # where retrieval fails
uv run python scripts/measure_recall.py # recall@k across configurations
uv run python scripts/measure_generation.py # faithfulness, relevance, refusal
uv run python scripts/compare_engines.py # hand-written vs LCEL
uv run python scripts/review_questions.py # check the eval questions by hand- Most RAG failures are retrieval failures. Fixed debugging order: recall@k first, generation prompt second.
- Hybrid always. Vector search wins on paraphrase, lexical search wins on exact identifiers (error codes, function names). Neither alone is enough.
- Small context beats large context. Top-5 reranked, not top-20.
- Scores are part of the product. Every hit carries its similarity, fusion and rerank scores.
- The query embedding model must match the index. Changing it means re-embedding everything.
| Endpoint | Auth | Description |
|---|---|---|
GET /health |
— | Liveness probe |
POST /v1/collections/{name}/documents |
X-API-Key |
Add or replace a document (idempotent) |
GET /v1/collections/{name}/search?q= |
X-API-Key |
Search, with scores |
GET /v1/collections/{name}/stats |
X-API-Key |
Documents and chunks indexed |
Search is exposed before any chat endpoint exists, on purpose: retrieval is where most RAG answers are won or lost, so it has to be inspectable on its own rather than only visible through a generated answer that hides which passages produced it.
The FastAPI documentation, 58 pages of real markdown:
| Metric | Value |
|---|---|
| Documents indexed | 58 |
| Chunks | 457 |
| Tokens embedded | ~82,500 |
| Embedding cost | $0.0016 |
| Wall time | 40 s |
| Re-ingest unchanged | 1.2 s, $0 |
Re-ingestion is idempotent by content hash, and the hash covers the pipeline version as well as the text. Hashing content alone would report "unchanged" after a parser fix and keep serving chunks built by the old parser — a fix that never reaches the index is not a fix.
Nothing in the stack raises an error for a badly parsed chunk: it is embedded, indexed and served like any other. Sampling the index caught two problems that cost tokens and diluted every embedding:
- documentation generators append anchor attributes to headings (
## Errors { #errors }), which ended up inside every embedded chunk; - pages whose
h1restates the page title produced breadcrumbs likeCustom Response > Custom Response > Available responses.
Fixing both cut the embedded corpus by 14 % — 95,900 to 82,500 tokens — for the same 457 chunks. Cheaper, and less noise competing with the actual content.
Fifty questions, each generated from a passage and kept only if it does not quote it — a question that copies its own passage measures string matching, not retrieval.
| Engine | recall@1 | recall@5 | recall@20 | MRR |
|---|---|---|---|---|
| lexical only | 28 % | 42 % | 44 % | 0.34 |
| vector only | 52 % | 88 % | 100 % | 0.68 |
| hybrid (RRF) | 68 % | 92 % | 100 % | 0.80 |
| hybrid + rerank | 76 % | 96 % | 100 % | 0.85 |
Comparing the confidence intervals of two configurations says none of these gains are real: on fifty questions the interval is roughly ±13 points and every pair overlaps.
That test is the wrong one. Each configuration answered the same fifty questions, so the measurements are paired, and comparing them as two independent samples throws that away. The questions both engines get right — most of them — carry no information about which is better; only the disagreements do. McNemar's exact test uses exactly those:
| Comparison (recall@1) | Independent intervals | Paired test |
|---|---|---|
| hybrid vs vector | "within the noise" | p = 0.008 (0 losses, 8 wins) |
| hybrid + rerank vs vector | "within the noise" | p = 0.002 (1 loss, 13 wins) |
Same data, opposite conclusion. Hybrid retrieval does not lose a single question that vector search alone answers, and wins eight it does not.
At recall@5 the difference is genuinely not distinguishable — but that is a ceiling effect, not a tie: vector search already reaches 88 %, and this dataset's questions were generated from the chunks, so every answer is findable by construction. recall@20 of 100 % is a property of the dataset, not an achievement of the retriever. recall@1 and MRR are the honest metrics here.
| Value | |
|---|---|
| Latency added per request | 206 ms for 20 pairs, on CPU |
| Throughput ceiling | ~5 requests/second per process |
The second number is the one that matters for deployment. Inference is serialised behind a lock — one thread at a time inside the model — so the reranker, not the database or the embedding call, becomes the bottleneck under load. Batching, or moving it behind its own service, is the next step before this is put in front of real traffic.
The top five reranked passages are numbered and given to the model, which must cite the passage behind every claim or refuse outright. Citations are then verified against the passages actually supplied — a regex and a comparison, not another model call: it is a property of the text, it is instant, and it cannot itself hallucinate.
Measured on 30 reviewed questions, plus 5 questions the corpus cannot answer:
| Metric | Result |
|---|---|
| Faithfulness | 96.6 % [82.8, 99.4] |
| Answer relevance | 100 % [88.3, 100] |
| Citations valid | 100 % [88.6, 100] |
| Correctly refused when the corpus is silent | 5 / 5 |
That last row matters more than it looks. A system that never says "I don't know" is a system that fabricates when the corpus is silent, and that failure is invisible in any accuracy score computed only on answerable questions.
The single unfaithful answer is the interesting one. Asked why code should be copied near the end of a Dockerfile, the model explained Docker layer caching — true, useful, and not in the passages. That is precisely what a strict faithfulness check is for: correct in the world is not the same as supported by the sources, and only one of those is verifiable by the reader.
"And inside a container?" is clear to a human reading the conversation and meaningless to a retriever: the words that would find the answer are in the previous turn. So the question is rewritten to stand alone for retrieval only — the model still answers the question the user actually asked, in their words.
| Passage found in the top 5 | |
|---|---|
| raw follow-up | 3 / 5 |
| rewritten | 5 / 5 |
Two questions fixed, none broken. Five cases cannot establish significance and the script says so — this demonstrates the mechanism, not a benchmark.
"And inside a container?"
-> "How do I run a FastAPI application in production inside a container?"
"Where do I put the token then?"
-> "Where do I put the OAuth2 token then?"
First turns are not rewritten: a question that already stands alone gains nothing from a round trip, and every rewrite is a chance to lose the user's own vocabulary. A rewrite that fails, comes back empty, or runs away into an answer is discarded and the raw question is searched instead — degraded retrieval beats no answer.
Before those numbers count, the judge is given answers known to be bad: one that invents an API parameter and a version number, and one that answers a different question. It rejects both, and names the invented claim. A judge that approves broken answers would report the same 96.6 % and mean nothing by it.
The same pipeline is available two ways — "engine": "native" or "engine": "lcel" on the chat
endpoint. Both get the same retrieved passages and the same system prompt, so any difference is the
framework's own handling of the call.
| faithful | relevant | citations | median | p95 | |
|---|---|---|---|---|---|
| native | 95 % | 100 % | 100 % | 1.79 s | 2.57 s |
| LCEL | 95 % | 100 % | 100 % | 1.88 s | 2.48 s |
20 of 20 answers were byte-identical. Not "statistically indistinguishable" — the same bytes. The paired test has nothing to work with because there is not a single disagreement.
So the choice is not about quality. It is about what the abstraction gives and costs:
What LCEL gives. astream, ainvoke, batch, with_retry and with_fallbacks come free on
any composed chain, and tracing is one environment variable. On a pipeline with branches or a
map-reduce step, that composition is worth real code.
What it costs. Eight extra packages, and the prompt is now assembled inside a library — answering "what exactly did we send?" means printing the chain rather than reading the call. That check is two lines and it is in the comparison script, because a composition layer that silently reshapes the prompt is the thing that makes debugging hard.
The verdict for this project: not worth adopting. Streaming, retries and fallbacks were already written, tested and understood before the framework was tried, so LCEL would replace working code with an equivalent that is harder to inspect. The rule the project follows — use a LangChain component only if you can explain what it does in HTTP — is what makes that a decision rather than a preference.
Both paths stay in the repository. A comparison that leaves no artefact is one the next person has to redo.
Ten probe queries against the indexed corpus. Identifier queries name a literal string a correct chunk must contain, which turns "the results look worse" into a number.
| Query type | Result |
|---|---|
| Exact identifiers | 3 of 5 found in the top 5 |
| Similarity, best case | 0.49 |
response_model_exclude_unset and HTTP_418_IM_A_TEAPOT are not retrieved at all: rare tokens
carry little semantic signal, and HTTP_418 sits almost on top of HTTP_409 in embedding space.
The paraphrase queries are not convincing either — "how can I send a file back to the person calling my API?" returns OpenAPI Webhooks, and "how do I stop the server blocking on a slow operation?" returns Run a Server Manually. Two of five are plainly wrong, and no similarity score exceeds 0.49.
So the honest summary is not "strong on meaning, weak on identifiers" — it is that vector search alone is weak here, and weakest on identifiers. That is the gap lexical search and reranking have to close, and the reason they get measured rather than assumed.
Published on v1: recall@5 / recall@20 across three retrieval configurations, faithfulness and answer relevance, reranker latency.