Repository navigation
Project Layout
This page is a map of the repository: what each folder and package is for, and how the layers of the code depend on each other. It's meant for finding your way around before your first change. If you haven't got the app running yet, do Getting-Started first, because the map makes a lot more sense once you've clicked around.
The code is stacked in layers, and the arrows below point from a package to the packages it imports:
flowchart TD
UI["ui/<br/>NiceGUI workspace"] --> API["api/<br/>FastAPI routers, schemas"]
API --> SVC["services/<br/>ingestion, evidence, review, trail"]
SVC --> ML["ml/<br/>models, extraction, audit rules"]
SVC --> SRC["sources/<br/>fetch, parse, chunk, search"]
SVC --> REPO["repositories/<br/>Postgres store, index"]
API --> JOBS["jobs/<br/>job record, runners, queue"]
JOBS --> REPO
REPO --> SRC
ML --> DOM["domain/<br/>models and enums, no I/O"]
SRC --> DOM
REPO --> DOM
BOOT["bootstrap.py<br/>builds everything from settings"] -.-> SVC
There's one rule here: imports only go down. domain/ doesn't import anything of ours, nothing below api/ knows a web framework exists, and the workspace in ui/ never touches services or the database directly. Instead it calls the API's own router functions in-process, which is what keeps a rule like "a candidate needs a verified receipt" in exactly one place.
Holding the whole thing together is bootstrap.py. It's the one place that reads settings and wires the concrete pieces (which embedder, which parser, which search provider) into an AppState, and the API, the worker, the scripts and the tests all start from it.
To see the layers in action, here's what happens when a reviewer presses c on a candidate:
sequenceDiagram
actor Rev as Reviewer
participant Page as ui/pages.py
participant Act as ui/actions.py
participant Router as api/routers/candidates.py
participant Svc as services/review_service.py
participant Store as repositories/store.py
Rev->>Page: press c
Page->>Act: Workspace.confirm()
Act->>Act: check the review key
Act->>Router: accept() (a direct call, not HTTP)
Router->>Svc: accept_candidate()
Svc->>Svc: receipt verified?
Svc->>Store: create commitment, guess asset owner, mark candidate accepted
Store-->>Svc: commitment id
Svc-->>Router: commitment
Router-->>Act: commitment_id (or 404 / 422)
Act-->>Page: toast "Confirmed as a commitment."
The key handler in ui/pages.py calls Workspace.confirm(), and from there ui/actions.py checks the review key and calls the API's router function directly. accept() in api/routers/candidates.py is where errors turn into 404 or 422. The real work happens in accept_candidate() in services/review_service.py, which checks the receipt, creates the commitment, guesses the asset owner and marks the candidate accepted, and repositories/store.py runs the actual SQL.
Everything the app runs is under src/paper_trail/, one package per layer.
| Path | What's in it |
|---|---|
ui/ |
the review workspace. main.py is the ASGI app (the FastAPI app plus the NiceGUI pages in one process), pages.py is the workspace page itself (table, detail pane, dialogs, shortcuts), workspace.py holds the row model and all the UI wording with no NiceGUI in it, and actions.py is everything the UI reads and does, through the API's router functions |
api/ |
the REST API. app.py is the app factory with request logging, CORS and the routers, deps.py has the one AppState per process, the review-key check and the job handlers, schemas.py holds request bodies and joined response views, and routers/ has meta, documents, candidates, commitments, links, trail, jobs and translate |
services/ |
the pipeline's steps. ingestion_service.py goes from PDF to pages to candidates, evidence_service.py from a commitment to sources to ranked, audited links, and review_service.py is the only code that writes human decisions. metrics_service.py builds the trail (status, money, findings, gaps), gazetteer.py turns parcel numbers into street names, and translation.py is the optional dev-only machine translation |
ml/ |
everything shaped like a model or a rule. extraction.py is the LLM extractor (prompt, schema, receipts, filters, dedupe), client.py the OpenAI SDK client with one JSON-schema call helper, embeddings.py the local fastembed, model2vec, OpenAI-compatible and hashing embedders, and rerank.py the cross-encoder client for any /rerank server. ner.py (GLiNER) and nlp.py (huSpaCy) find entities and lemmas, entities.py parses money, years and status cues with regular expressions on purpose, matching.py computes the link score and relationship hint, accountability.py holds the audit rules, lang.py defines the language profile's shape (the data lives in the database), and notes.py writes the plain-English notes |
sources/ |
getting text from the outside world. fetch.py does allowlisted downloads with redirect and private-address checks and turns HTML or PDF into text, parsers.py wraps pypdf and Docling, pdf.py gives plain page-by-page PDF text shared with the scripts, chunking.py cuts structure-aware passages, web_search.py has the SearXNG, DuckDuckGo and fixture search providers plus the allowlist check, sitemaps.py does bounded sitemap and feed discovery, and document_store.py finds source PDFs by name in a plain folder |
repositories/ |
the database. store.py handles records (documents, candidates, commitments, evidence, links, money and so on), mapping.py turns rows into domain models, index.py handles pages, passages, their vectors and hybrid search, and db.py opens a plain one-off connection for maintenance and the worker's role check |
jobs/ |
slow work. __init__.py has the job record, local.py the in-process runner used in development (JOB_MODE=internal), and postgres.py the Postgres queue and worker loop used in production |
domain/ |
models.py and enums.py: plain data with no I/O |
settings.py, bootstrap.py
|
every setting (Reference), and the code that wires one AppState from them |
worker.py, maintenance.py
|
the production job worker, and the maintenance CLI (reset, rerun, source registry, profiles, notes) |
Outside src/, the other top-level folders support the app rather than being part of it.
| Path | What's in it |
|---|---|
db/ |
sql/ holds the numbered SQL migrations, which are never edited once released, and alembic/ with alembic.ini applies them (Database) |
deploy/ |
docker/api.Dockerfile builds both the api and worker images, and zakazserver/ has the Compose stack, Caddy, SearXNG, database init, backup and restore, and the GPU phase script (Deployment) |
scripts/ |
ops/ has the Postgres integration check and gazetteer seeding, and ml/ has the extraction evaluation, gold check and retrieval benchmark, plus finetune/ (dataset, LoRA job) and quantize/ (calibration, AWQ job) (Models) |
tests/ |
mirrors src/ (api, jobs, ml, repositories, scripts, services, sources, ui, plus test_settings.py), and fixtures/ holds synthetic data and the example language profiles hu.json and en.json
|
A few things you might expect in the repo deliberately live somewhere else. The data/ folder (source PDFs, evaluation sets, downloaded models and caches) stays on your machine and is gitignored. So are .env files, and only the .env.example files get committed. Deployment data like sources, owners' domains, streets and language profiles lives in the database, and the documentation is this wiki.
When you're ready to make a change, Contributing walks through the loop, and How-It-Works shows how all these pieces work together at runtime.
Repository · GPL-3.0 · Nothing reaches the record without a human click · Something wrong here? Fix this wiki
Tutorial
How-to guides
Explanation
Reference