Repository navigation
Models
Paper Trail uses a handful of machine-learning models, and this page goes through all of them: what each one does, why we picked it, where it runs, and how well the most important one, the extraction LLM, actually performs. It ends with the loop we use to make that model better. It's useful if you want to know how far to trust the machine's suggestions, or if you're a developer about to swap or retrain a model.
Before getting into any of it, it's worth repeating that none of these models makes a decision. Each one proposes, ranks or annotates something, and then a person decides (Accountability-Rules).
The models split into two groups, matching the two machine stages in How-It-Works. One group reads the strategy and the other searches for evidence:
flowchart LR
subgraph READING ["Reading a strategy"]
DOC["Docling or pypdf<br/>PDF to text"] --> LLM["Extraction LLM<br/>proposes commitments"]
end
subgraph SEARCHING ["Searching for evidence"]
EMB["Embedding model<br/>meaning as numbers"] --> RR["Reranker<br/>reads query + passage"]
RR --> NER["GLiNER + huSpaCy<br/>who, where, when"]
end
LLM -.->|"same model, offline"| NOTES["Plain-English notes"]
| Role | Production model | Runs on | What it does |
|---|---|---|---|
| Extraction LLM | unnobatroo/paper-trail-qwen3-4b-instruct-hu-awq |
GPU, Goethe | reads strategy pages and returns commitments with quotes |
| Embeddings | Qwen/Qwen3-Embedding-0.6B |
GPU, Goethe | turns passages and queries into vectors for similarity search |
| Reranker | paper-trail/bge-reranker-v2-m3-bf16 |
GPU, Goethe | re-scores the top 20 passages against the commitment |
| Entities | GLiNER urchade/gliner_multi-v2.1
|
CPU, worker | finds organisations, streets, squares, parks and dates in a passage |
| Hungarian language | huSpaCy hu_core_news_lg 3.8.1 |
CPU, worker | lemmas for keyword search ("fákat" becomes "fa"), plus organisations, places and dates |
| PDF parsing | Docling | CPU, worker | layout, tables and OCR, with pypdf as the fallback |
The extraction LLM has a side job too. It writes the optional plain-English notes on queue items (maintenance notes generate), and each note is labelled with the model that wrote it. There's also machine translation, but only in development: POST /api/translate calls the Hugging Face inference API (opus-mt-<language>-en) when HF_TOKEN is set. It's switched off in production, where reviewers have to read the source itself.
Some things deliberately aren't done by any model. Money, years, procurement ids and parcel numbers are all parsed with regular expressions (ml/entities.py), because money has to be exact and anyone has to be able to check how a number was read.
The extraction model is the project's own 4-bit build of Qwen/Qwen3-4B-Instruct-2507, quantized with AWQ and calibrated on Hungarian municipal text (more on that under Quantization). It's small enough for an 8 GB laptop GPU and good enough at following a JSON schema, which is all we need from it.
Every call to it runs at temperature 0 with a 2,048-token answer budget and forces a strict JSON schema with length limits on each field. Qwen3's "thinking" mode is switched off (enable_thinking: false), because otherwise the model burns its whole budget reasoning and never actually answers. What happens to its answers afterwards (the receipt check, code grounding, the fact-sheet filter and dedupe) is covered in How-It-Works.
You don't need a GPU for development, because EMBED_MODEL lets you choose a lighter embedder. If you leave it unset, you get sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 running locally on the CPU through fastembed, which is a one-time ~220 MB download. openai:<model> uses any OpenAI-compatible /v1/embeddings server at EMBED_BASE_URL, which is what production does. model2vec:<repo> loads a model2vec static model such as minishlab/potion-multilingual-128M, which is a lookup table instead of a transformer, so it needs no GPU and no server and embeds a couple hundred passages in milliseconds after a one-time ~510 MB download. On the evidence benchmark it finds fewer right passages in the top five than MiniLM (recall 0.27 against 0.34, Qwen3 on the GPU gets about 0.51) but more in the top twenty (0.62 against 0.55), and the top twenty is what hybrid search and the reranker work from, so it's a reasonable choice when there's no GPU box around. hashing is a word-hashing stub for tests that needs no download and gives meaningless results.
Whichever you pick, the database column holds at most 1024 dimensions and shorter vectors are padded with zeros. Vectors are stored per model name, so if you switch models, the next search simply re-embeds whatever it reads.
In production, every GPU model runs behind Goethe (Apache-2.0, built on SGLang). It's one OpenAI-compatible gateway for /v1/chat/completions, /v1/embeddings and /v1/rerank. The app only speaks those OpenAI-style endpoints, so in development any compatible server will do.
The reference server has an 8 GB GPU, and the LLM doesn't fit next to the two retrieval models. So the GPU runs in one of two phases at a time:
stateDiagram-v2
state "retrieval: embeddings + reranker, about 3.5 GB" as R
state "extraction: the 4B LLM, about 5 GB" as E
[*] --> R
R --> E: goethe-phase.sh extraction
E --> R: goethe-phase.sh retrieval
The retrieval phase loads the embedding model and the reranker, which is what evidence searches need. The extraction phase loads the LLM, for ingesting strategies and writing notes. A job that needs a model the current phase hasn't loaded fails with the model server's error, since the queue only retries timeouts and network errors. So switch first and then run the job again. The one exception is the reranker: if only that's missing, a search keeps the plain similarity order and tells you so. Switching phases is an operator task, covered in Deployment.
The Goethe settings that matter are all encoded in deploy/zakazserver/goethe-phase.sh, and each one is there for a reason. The embedding model and reranker load with --is-embedding and --disable-cuda-graph, which saves memory. The XLM-RoBERTa cross-encoder also needs --disable-radix-cache --chunked-prefill-size -1 --attention-backend triton, or SGLang trips an assertion about token positions. And Goethe sizes the LLM's memory for text in flight (the KV cache) from --ctx alone. With --ctx 4096 that came to about 1,965 tokens and cut JSON answers off halfway. --ctx 12288 --max-conc 4 gives about 6,100 tokens, which is enough for 3 to 4 pages at once. That's why production sets LLM_CONCURRENCY=4 and LLM_WINDOW=1.
To answer "is the model any good?" you need an answer key and a fair test, and we've got three pieces for that.
The first is the gold set. People drafted answer keys for real climate strategies, listing for each document and page the commitments on it with a title, a code and a verbatim quote. scripts/ml/check_gold.py then keeps only the rows whose quote really is on that page of the PDF (ignoring whitespace) and prints the rest for someone to fix. The current set, gold-v2, has 834 rows from 9 strategies: 7 Hungarian (Budapest districts and Pécs) and 2 English (Leicester, and Kensington and Chelsea).
The second is the split between train and holdout. Three of the documents are train, which means they're allowed for prompt work and fine-tuning. The other six are holdout, only ever used for testing, and both English ones are in that group. A model that has learned one city's layout looks brilliant on train and falls apart on holdout, so holdout is the number that counts. The gold files and PDFs live in data/eval/, which is gitignored.
The third is the scorer, scripts/ml/eval_extraction.py. It runs the extractor exactly the way ingestion does (receipts, filters, dedupe and all) over the gold pages and reports a few numbers per document. Recall is how many of the gold commitments a kept answer on the same page matched. Precision is how many of the kept answers match some gold row on their page. F1 balances the two, kept rate is how much of what the model returned survived the receipts, filters and dedupe, and seconds per page shows the speed at the given concurrency. Everything is averaged per document ("macro"), so one long document can't drown out the rest.
Here's the project's model on gold-v2, holdout documents only, macro-averaged:
| Version | Recall | Precision | F1 |
|---|---|---|---|
| after PR #24 | 0.871 | 0.935 | 0.902 |
| current main (PR #27: stricter form-row and dedupe matching) | 0.866 | 0.929 | 0.896 |
Put simply, on strategies it has never been tuned on, it finds a bit under 9 out of 10 of the promises a person marked, and more than 9 out of 10 of the things it keeps are real promises.
Those numbers come with some caveats. Part of the gold set was built from the model's own answers. After the first hand-drafted keys, we went through the answers the model gave that the keys had missed and added the real ones, which came to 159 of the 834 rows. That "pooling" is standard practice in search evaluation (TREC does it), and it makes precision fairer, but it flatters recall a little, because promises that neither the people nor the model spotted aren't in anybody's key. It also means precision is really a floor: the key can still be missing real promises, in which case a correct answer gets counted as junk. And nine documents is a small test, so treat a difference of a point or two as noise.
A couple of ideas lost on holdout and got removed. A second "critic" pass by the same model dropped holdout recall from 0.864 to 0.747 for a small gain in precision. Splitting pages into 2,000-character chunks bought 0.02 of recall but nearly doubled the candidates on dense documents.
eval_extraction.py --baseline <earlier report> compares a new run with an older one and exits with an error unless holdout macro recall goes up on the same holdout documents. A new model or prompt only ships if it passes.
Retrieval has its own benchmark as well. scripts/ml/benchmark_retrieval.py measures recall and MRR on labelled passages through the real Postgres hybrid index, with or without the reranker.
Every confirm and reject in the workspace is a label, and this loop turns those labels into a better model:
flowchart LR
REV["Reviewer decisions<br/>in the database"] --> DS["build_dataset.py<br/>one example per page"]
GOLD["checked gold keys<br/>train split only"] --> DS
DS --> HUB[("private dataset<br/>on Hugging Face")]
HUB --> LORA["lora_job.py<br/>Hugging Face Job, GPU"]
LORA --> AWQ["awq_job.py<br/>4-bit quantization"]
AWQ --> EVAL{"eval_extraction.py<br/>--baseline"}
EVAL -- "holdout recall up" --> SHIP["load into Goethe"]
EVAL -- "not better" --> STOP["keep the current model"]
It starts with scripts/ml/finetune/build_dataset.py, which builds training examples out of the database. It builds the prompt with the same function the server uses, so training and serving can't drift apart. A page with confirmed candidates turns into those items, with the reviewer's edited title or type. A page where everything was rejected turns into an empty answer, which teaches the model to say nothing. A page nobody reviewed gets skipped, because unknown isn't the same as empty. Documents passed with --holdout never get in, and --gold adds other cities' checked keys from the train split.
Those examples feed scripts/ml/finetune/lora_job.py, which runs a LoRA fine-tune of the base model as a Hugging Face Job. LoRA trains a small add-on to the model's weights instead of the whole thing. It runs 3 epochs at rank 16, with the loss computed on the JSON answer only, and over-long examples are dropped rather than cut short. The merged result goes to a private repository:
hf jobs uv run --flavor a10g-large --timeout 3h --secrets HF_TOKEN \
scripts/ml/finetune/lora_job.py Qwen/Qwen3-4B-Instruct-2507 DATASET_REPO OUT_REPOThen the result gets quantized (next section), and eval_extraction.py --baseline decides whether it ships.
AWQ ("activation-aware weight quantization") stores the weights in 4 bits instead of 16, which is how a 4-billion-parameter model fits on a small GPU. It uses sample text to work out which weights matter most, so the sample text needs to look like what the model will actually read. That's what scripts/ml/quantize/build_calibration.py is for: it picks 512 long passages from the municipality's own official sources and strategy.
scripts/ml/quantize/awq_job.py then runs llm-compressor (pinned in awq_job.py.lock) as a Hugging Face Job. It uses W4A16 asymmetric quantization, leaves lm_head in bf16 and calibrates on 256 samples of 512 tokens, after 512 samples of 1024 tokens ran out of memory on a 24 GB A10G. The output goes to a private repository, and making it public is a separate step that only happens after it wins the evaluation.
hf jobs uv run --flavor a10g-large --timeout 2h --secrets HF_TOKEN \
scripts/ml/quantize/awq_job.py BASE CALIBRATION_DATASET OUT_REPOTo load or switch models on the server, see Deployment. Every model setting is in Reference, and if you want to give the model better labels, Reviewing explains how.
Repository · GPL-3.0 · Nothing reaches the record without a human click · Something wrong here? Fix this wiki
Tutorial
How-to guides
Explanation
Reference