How It Works

This page walks through what Paper Trail actually does, end to end. You don't need to know anything going in.

The shape of it

flowchart LR
    U["👤 Reviewer"] --> W["Next.js frontend<br/>(Vercel)"]
    W --> A["FastAPI service<br/>(Azure)"]
    A --> S[("Supabase<br/>Postgres + pgvector + Storage")]
    A --> O["Official sites<br/>jozsefvaros.hu · rev8.hu · budapest.hu"]

Loading

The browser talks only to the FastAPI service. The service talks only to Supabase and the allowlisted official sites. That keeps all the keys server-side and the data in one place.

Step one: reading the strategy

Everything starts with the climate strategy PDF. The ingestion service pulls it apart with a rule-based parser that understands this document's own structure — strategic goals, thematic goals, measures, targets — or, when configured, an LLM extractor against any OpenAI-compatible endpoint for layouts the rules don't know. Each thing it finds becomes a candidate, and every candidate must carry its receipts: the page number and a verbatim excerpt, re-verified against the page text. (The LLM is asked for the page marker it was shown, not the number printed on the page, so the check looks at the right page.) No excerpt, no candidate. A malformed LLM answer is retried page by page rather than silently dropping a whole window of the document — and an ingest that extracts nothing says so in the job result instead of leaving an unexplained empty queue.

Candidates aren't commitments yet. They're a queue. A reviewer confirms or rejects each one — which is why the app is built like a triage inbox, with keyboard shortcuts and bulk actions, so clearing 49 candidates takes minutes rather than an afternoon. In the workspace these are the rows in the to confirm stage.

Step two: hunting for evidence

For each confirmed commitment, "Find evidence" kicks off a background job. Here's what it does:

1. Search. It builds a short query from the commitment's title and code and asks DuckDuckGo for results on each allowed domain, one site: query each, retrying transient failures before giving up on a domain. It also adds the known report PDFs from the official_sources registry — a database table, not code — because search engines surface those poorly.
2. Fetch. Each URL gets downloaded once — HTML through trafilatura, PDFs through pypdf — and the extracted text is cached forever. Nothing is ever fetched twice.
3. Chunk. Documents get split into 2400-character passages with a 400-character overlap, so a sentence that matters doesn't fall between two chunks. The whole document is processed; there's an emergency size limit, and hitting it produces a visible warning rather than a quiet cut.
4. Embed. Every chunk becomes a vector via a multilingual MiniLM model (~220 MB of open-source ONNX) by default, or via any OpenAI-compatible embeddings server such as BGE-M3 on your own box. Vectors are cached per page and model name so re-runs embed nothing again.
5. Rank. A hybrid score picks the top 20 chunks — dense cosine fused with stemmed lexical matching, a mix the benchmark measured beating either signal alone on this corpus. If a cross-encoder reranker is configured it reorders them, otherwise the fused order stands with a visible warning.
6. Dedupe. One candidate per source document — same URL, identical text, or a same-titled PDF at two URLs all collapse into one.
7. Extract. Organisations, places, dates, and money figures get pulled out — importantly, money and status are judged on the matched passage only, because a 100-page report contains every cue phrase somewhere.
8. Propose. Each surviving document becomes a link suggestion with a score and plain-language reasons ("names the same organisation", "same year: 2024"), stored as unreviewed.

The score is transparent arithmetic, not a black box: mostly the text similarity, plus small bonuses when the document shares the commitment's organisations, locations, or years. A suggested relationship label — direct_implementation, supporting, budget, indirect, or probably_unrelated — comes from simple thresholds over that score. It's a hint for the reviewer, never a verdict.

Step three: reviewing matches and the trail

Search results arrive as suggested matches (the to review stage). Each match shows the source, the passage and the suggested relationship, and is confirmed or rejected right in the detail panel or in bulk.

Confirmed commitments plus accepted links make up the trail — the public-facing record of what was promised and what evidence exists for it. Commitments with no accepted evidence show up too, as explicit gaps. That's the honest part: a gap on the record is the finding, not a failure of the tool.

The parts that keep it trustworthy

A few design choices are load-bearing rather than incidental:

• Human review is a hard wall. The review service is the only code path that can mark anything accepted. The pipeline literally cannot write a confirmed link.
• Caching is the cost model. Pages are fetched once and chunks are embedded once, so re-running a search costs a single query embedding. On a free-tier budget, that's what makes the whole thing feasible.
• Jobs survive restarts. Evidence searches take minutes, so they run as rows in a jobs table — any worker can claim one, double-submits collapse into the same job, and a worker that dies mid-run gets marked failed rather than silently lost.
• Everything swappable sits behind a small interface. The embedder, reranker and search provider are ABCs, and storage is two protocols (PolicyStore, EvidenceStore in repositories/ports.py) that both the SQLite and the Supabase repositories satisfy. Services only see the interface — tests run fully offline on stub providers, and production swaps in the real ones via environment variables alone.
• Writes can be gated. On a public deployment, setting PAPER_TRAIL_REVIEW_KEY makes every mutating endpoint require an X-Review-Key header; reviewers type it into a dialog in the top bar once and it stays in their browser. It's a shared secret, not accounts — unset means open review, which is what local dev uses.

What we deliberately didn't build

No auto-acceptance anywhere, no inferred status ("the deadline passed, so probably done"), no merged money figures (an estimate is not a budget is not spending), and no LLM summarising sources. The extraction rules are hand-written and boring on purpose — a wrong heuristic is visible and fixable, while a wrong generated sentence is just a confident lie.