Skip to content

Concepts

jismailov edited this page Oct 31, 2026 · 2 revisions

The rest of this wiki leans on a small set of words, and this page explains each of them once. The terms build on each other, so it's worth reading top to bottom the first time. After that you can just jump back here whenever a word on another page doesn't ring a bell.

To keep things concrete, we'll follow one made-up but realistic promise from a Hungarian district's climate strategy the whole way through:

"Fák ültetése a kerületben: 2027-ig 500 új utcafát ültetünk." ("Planting trees in the district: by 2027 we'll plant 500 new street trees.")

Here's how the main pieces relate before we get into each one:

flowchart LR
    D["Strategy document"] -->|"split into"| P["Pages"]
    P -->|"LLM proposes"| C["Candidate<br/>+ receipt"]
    C -->|"person confirms"| M["Commitment"]
    M -->|"search official sources"| E["Evidence<br/>a passage on a web page"]
    M --- L(("Link"))
    E --- L
    L -->|"person accepts"| T["Trail"]
    M -.->|"nothing accepted"| G["Gap"]
Loading

From a PDF to a promise

It all starts with a strategy document, the PDF a municipality publishes to say what it's going to do about climate change, like Józsefváros's 2021 climate strategy. Paper Trail stores its title, publisher and official URL in the documents table, and then turns each page of it into plain text. Pages matter more than you'd think, because every promise stays tied to the page it came from. That way a reader can open the PDF at page 31 and see the sentence with their own eyes.

The reading itself is done by an LLM (large language model), a program trained on a huge amount of text that can read a page and answer questions about it. Paper Trail runs a small one, about 4 billion parameters, on its own hardware (Models has the details). It's good at spotting promises and bad at being trusted, which is why everything it says gets checked.

Whatever the LLM thinks is a promise becomes a candidate. A candidate is just a suggestion sitting in a queue until a person looks at it, for example a candidate titled "Fák ültetése a kerületben", type measure, from page 31.

Every candidate comes with a receipt, the exact quote it was taken from. The LLM has to quote the page word for word, and then Paper Trail searches the page text itself to make sure the quote is really there. If it isn't, the candidate gets thrown away before anyone sees it. In our example the receipt is "2027-ig 500 új utcafát ültetünk", found on page 31. We're this strict because LLMs sometimes make up text that sounds completely plausible. With a receipt, "the model says the strategy promises X" becomes "here's the sentence, go look", which is something you can actually check in ten seconds.

Once a person reads the candidate and agrees that yes, the municipality really promised this, it becomes a commitment, say commitment #12, "Fák ültetése a kerületben". Only a human click can create one. Commitments can also nest, so a measure can sit "under" the objective it belongs to.

Strategies tend to be written with four kinds of commitment, and Paper Trail uses the same four:

Kind Meaning Example
objective a goal, often with a code like M1 "M1 Greener streets"
measure a concrete action "M1.2 Planting trees in the district"
target a number to hit by a date "500 new street trees by 2027"
indicator how progress gets measured "number of trees planted per year"

Objectives, measures and targets get searched for evidence. Indicators get recorded but not searched, since they describe how you'd measure the work rather than the work itself.

That M1.2 in the table is a code, the label a strategy prints in front of a heading (M1.2, 1.01 and so on). Models love inventing codes, so Paper Trail only keeps one if the page actually prints it.

Looking for evidence

Once there's a commitment, the next question is where to look for proof. Paper Trail only trusts an official source, meaning a website or document a public body publishes itself: the district's site, the city's site, a utility's investment page, a budget decree. It won't fetch anything that isn't on its list of allowed sources, the source registry (the official_sources table). For Józsefváros, https://jozsefvaros.hu/ is the obvious first entry.

Which sources make sense depends on the asset owner (also called the jurisdiction), meaning whoever owns the thing the promise is about. A district doesn't own the water mains, so evidence about water mains lives on the water utility's site, not the district's. Each commitment gets an owner guessed from its wording, and the reviewer can change it. There are five: the district municipality, the city government, the water utility, district heating and the national government.

Finding the right page on those sites needs more than keyword matching, and that's where embeddings come in. An embedding is a list of numbers that represents what a text means, so texts with similar meanings end up with similar numbers. That's how a search for "Fák ültetése" can turn up "utcafa-telepítés befejeződött" ("street tree planting finished") even though they share no words. The embeddings are stored with pgvector, an extension for PostgreSQL that keeps these number lists and finds the closest ones fast, which lets Paper Trail search thousands of passages at once.

A passage (or chunk) is a slice of a fetched web page or PDF, about 2,400 characters long. Pages get cut up like this so the search can point at the one paragraph that matters instead of a whole 200-page report.

When a passage looks like it might show a commitment was (or wasn't) carried out, it becomes evidence. Picture a 2024 district news page saying "a Kálvária téren 120 fát ültettünk" ("we planted 120 trees on Kálvária Square"). Along with the passage, the evidence row records what it mentions: organisations, places, dates, money and contract numbers.

Evidence and commitments are connected by a link, a suggested match between one commitment and one piece of evidence. Each link has a score and some plain-language reasons, like "the text is a close match; same place: Kálvária tér". Every link starts out unreviewed and stays that way until a person accepts or rejects it.

A link also has a relationship, which says what kind of connection it is. The machine suggests one and the reviewer picks the final answer:

Relationship In the UI Means
direct_implementation evidence it happened the source documents work on this very commitment
supporting supporting evidence useful context, but doesn't prove it was done
budget budget information a cost, a budget line or reported spending
related_but_indirect related, but indirect related work with a loose connection
probably_unrelated probably unrelated nothing useful

From accepted direct-implementation evidence, Paper Trail reads a status, which is what the source says about the work: announced, planned, in preparation, in progress or completed. It never guesses status from a deadline passing. If the trail says "completed", that's because an accepted source said so.

Money gets the same careful treatment. Every figure is tagged with a budget state, the stage of the money cycle it comes from: estimated cost, appropriation, amended appropriation, contracted price, final accounts, or spending that's only reported. Different states are never added together, and Accountability-Rules explains why that matters so much.

What you end up with

All of this adds up to the trail (or paper trail) for a commitment: its accepted evidence, the status the sources report, the money on record and any audit findings.

Whatever the trail is missing gets listed as a gap, for example "No implementation evidence found yet.", "No confirmed budget found." or "No measurable target was stated." A gap isn't an error. Quite often it's the most useful thing the tool produces.

An audit finding is a warning the rules raise from accepted evidence, such as "Money was appropriated but no final accounts figure or completion certificate confirms it was spent."

Words you'll meet in the workspace

In the review workspace, each commitment sits in a stage that tells you what's left to do with it:

Stage Means
To confirm a candidate is waiting: is it a real promise?
To search confirmed, but no search has found anything to review yet
To review the search suggested matches and a person has to judge them
Done nothing left to do (evidence accepted, or it's an indicator)
Rejected a person said it isn't a commitment

Reading a PDF or searching for evidence takes a while, so each of those runs as a job in the background while you keep working, and the workspace shows its progress as it goes.

On a shared deployment, changes are protected by a review key, a shared password for the whole team. If one is set, every change (confirm, reject, starting a search) needs it, and you type it once per browser. It's a simple gate, not a user account system.

Reviewers who don't read Hungarian get help from notes. A note is an optional plain-English explanation of a queue item, written ahead of time by the project's own LLM, plus a suggestion (accept, reject or unsure). It's always labelled "AI" and it never decides anything.

Behind all the rules sits a language profile, the word lists and patterns for one language and country, stored in the database. It says which words mean "cancelled" or "amended", how money is written ("2,5 millió Ft"), what a procurement number looks like, which fact-sheet labels ("Időtáv", "Felelős") aren't promises, and it carries a glossary for the UI. Swapping the profile is how Paper Trail moves to another country. The one in use for Józsefváros is the hu profile.

Measuring the model

To know whether the extraction model is any good, you need an answer key, and that's the gold set: promises people have checked by hand. Each row says that in this PDF, on this page, there's this commitment, quoted like this. The current one has 834 rows from 9 strategies.

Scoring against it comes down to two numbers. Recall asks how many of the gold promises the model found, and precision asks how much of what the model returned was a real promise. A model that returns everything gets perfect recall and awful precision, which is why the two get combined into F1, one number that balances them.

Those numbers only mean something on a holdout, a set of documents that's never used to tune or train the model, only to test it. A model that has learned one city's layout scores great on its training documents and falls over on new ones, so holdout is the number we actually believe.

Now that the words are in place, How-It-Works shows them moving through the system, and Getting-Started lets you try it yourself.

Clone this wiki locally