Skip to content

Deployment

jismailov edited this page Oct 31, 2026 · 9 revisions

This guide is about running Paper Trail for real, on one self-hosted Linux server with one NVIDIA GPU. It assumes you're comfortable with Docker Compose and a shell. Everything it mentions is in the repository: deploy/zakazserver/ has the Compose file, Caddy, SearXNG, database init and the backup and GPU-phase scripts, and deploy/docker/api.Dockerfile builds both application images (the api and worker targets).

In the commands below, your-server stands for the machine, paper-trail.example.org for its public domain, and <...> for a value you choose. Whatever you do, never commit a filled-in .env.

What runs where

The whole stack is a handful of containers plus a model server on the host:

flowchart LR
    NET((Internet)) -->|"80, 443 tcp<br/>443 udp"| CADDY
    subgraph compose ["Docker Compose on your-server"]
        CADDY["caddy<br/>TLS, headers, rate limit"] --> API["api<br/>uvicorn, 1 worker:<br/>REST API + NiceGUI workspace"]
        API --> DB[("db<br/>Postgres 16 + pgvector")]
        WORKER["worker<br/>ingest + evidence jobs"] --> DB
        WORKER --> SX["searxng<br/>private search"]
        BACKUP["backup<br/>daily dumps"] --> DB
    end
    subgraph host ["On the host"]
        GOETHE["Goethe gateway :8443"]
        GOETHE --- RET["retrieval phase<br/>embeddings + reranker"]
        GOETHE --- EXT["extraction phase<br/>4B LLM"]
    end
    WORKER -->|"https, API key,<br/>pinned certificate"| GOETHE
    WORKER --> WEB["official websites<br/>allowlisted"]
Loading
Service Image Memory cap Role
caddy custom Caddy build with the rate-limit module 192 MB the only service with published ports
api paper-trail-api:local 768 MB API and workspace in one uvicorn process
worker paper-trail-worker:local 4 GB runs jobs; its image adds Docling, huSpaCy and GLiNER on CPU-only PyTorch
db pgvector/pgvector:pg16 2 GB the only data store
searxng SearXNG, pinned by digest 256 MB site: searches without a third-party API key
backup pgvector/pgvector:pg16 256 MB daily dump loop
migrate the api image, profile tools owner login, runs Alembic and maintenance commands when you ask

A few choices here are deliberate. There's only one uvicorn worker because the NiceGUI workspace keeps each browser tab's state inside the process, over a websocket, and several processes would split that state up. That's fine, because slow work already runs in the worker container. The worker itself has no review powers at all: it logs in as paper_trail_worker and gets an empty REVIEW_KEY (Database). And only Caddy can be reached from outside. The API, the database, search and the models all stay on the Compose network or the host.

1. Prepare the host

The host needs Docker with Compose v2, an NVIDIA driver and Goethe, which the next step sets up. The containers don't use the GPU directly, since all GPU work goes through Goethe, so you don't need the NVIDIA Container Toolkit.

2. Set up the model server (Goethe)

Every GPU model runs on the host behind Goethe, an OpenAI-compatible gateway. Models explains the models and why there are "phases". The containers reach the host under the name goethe (via extra_hosts: goethe:host-gateway) at https://goethe:8443/v1.

Start by installing Goethe from its release installer, then add the models with something like goethe install --model Qwen/Qwen3-Embedding-0.6B --yes. Models that aren't in a registry, like the bf16 reranker and the project's own LLM, go into ~/.goethe/models/<org>__<name>, and goethe scan --yes registers them.

Next, create an API key for the app:

goethe key --create paper-trail --rpm 0 --raw

The gateway should be reachable from the containers but not from your network. Bind it to the Docker bridge address, allowlist the Docker networks, and have the firewall only let those networks reach the gateway port.

Finally, it needs a usable certificate. The app verifies HTTPS to the model server against the certificate in GOETHE_CERT, which gets mounted into the containers as MODEL_CA_BUNDLE and only applies to model calls. That certificate has to name the hosts clients actually use (goethe and localhost). If the one Goethe generated has no subject alternative names, replace ~/.goethe/nginx/certs/server.crt and server.key with a self-signed pair that does.

3. Configure

cd deploy/zakazserver
cp .env.example .env && chmod 600 .env

Then fill in .env:

Variable What to put
DB_PASSWORD owner password (migrations, maintenance, backups)
API_DB_PASSWORD password of paper_trail_api
WORKER_DB_PASSWORD password of paper_trail_worker
REVIEW_KEY the shared review key, at least 32 characters (production refuses anything shorter)
SEARXNG_SECRET SearXNG's secret
ACME_EMAIL email for Let's Encrypt
DOMAIN the public domain, e.g. paper-trail.example.org
ALLOWED_DOMAINS normally empty, because sources belong in the registry (below)
BACKUP_DIR absolute host path for backups, e.g. /srv/paper-trail/backups
HUSPACY_MODEL_URL already set: huSpaCy 3.8.1 pinned to an immutable Hugging Face revision and SHA-256, checked during the build
GOETHE_API_KEY the key from step 2
GOETHE_CERT absolute path of Goethe's certificate on the host

Generate each secret separately, for example with openssl rand -hex 32 (hex avoids URL-quoting headaches). The app's own settings (models, search, parser and so on) are fixed in docker-compose.yml, and Reference lists all of them.

Warning

The database passwords are only read when the database volume is first created, so changing .env later does not change them. To rotate one, run docker compose exec db psql -U paper_trail -c "ALTER ROLE paper_trail_api PASSWORD '...'" (and the same for paper_trail_worker), then update .env.

4. First start

docker compose config --quiet                       # the file and .env are valid
docker compose build
docker compose up -d db                             # first start creates the api and worker logins
docker compose --profile tools run --rm migrate     # alembic upgrade head, as the owner
docker compose up -d api searxng backup worker caddy

The stack is up now, but it doesn't know anything about your municipality yet. That knowledge is deployment data, and it lives in the database rather than the repository. You load all of it through migrate, because that's the service with the owner login.

Start with the language profile. The image doesn't include the test fixtures, so mount the profile file in:

docker compose --profile tools run --rm \
  -v "$(pwd)/../../tests/fixtures/profiles:/profiles:ro" \
  migrate python -m paper_trail.maintenance profile import /profiles/hu.json

Then fill the source registry. The municipality's own sites get searched for every commitment, and you add --jurisdiction for sites that belong to an asset owner (Accountability-Rules):

docker compose --profile tools run --rm migrate python -m paper_trail.maintenance \
  source add https://jozsefvaros.hu/ --publisher "Józsefvárosi Önkormányzat"
docker compose --profile tools run --rm migrate python -m paper_trail.maintenance \
  source add https://www.vizmuvek.hu/hu/beruhazasok --publisher "Fővárosi Vízművek" \
  --jurisdiction public_utility_water

The gazetteer comes next. It pulls streets, squares and parks from OpenStreetMap, so this one needs internet access:

docker compose --profile tools run --rm migrate python scripts/ops/seed_gazetteer.py "VIII. kerület" --admin-level 9

Last are the strategy PDFs. The Ingest dialog lists files in /app/data/source_documents/ on the shared documents volume, so copy them there:

docker compose exec api mkdir -p /app/data/source_documents
docker compose cp strategy.pdf api:/app/data/source_documents/

Now open https://paper-trail.example.org, enter the review key and ingest your first strategy (Reviewing).

5. Switch GPU phases

On an 8 GB GPU the LLM can't sit next to the retrieval models, so the GPU runs one "phase" at a time and you switch between them on the host:

deploy/zakazserver/goethe-phase.sh extraction   # the LLM (~5 GB): for ingest and notes
deploy/zakazserver/goethe-phase.sh retrieval    # embeddings + reranker (~3.5 GB): for evidence searches
deploy/zakazserver/goethe-phase.sh status

The safe way to switch is to stop the worker first, so no job is halfway through when its model disappears:

docker compose stop worker
deploy/zakazserver/goethe-phase.sh extraction
docker compose start worker
# ... ingest, wait for it to finish ...
docker compose stop worker
deploy/zakazserver/goethe-phase.sh retrieval
docker compose start worker

Retrieval is the resting phase, since evidence searches are the everyday work. A job that needs a model that isn't loaded fails with the model server's error, so switch phases and start the job again. You can override the model ids each phase loads with EMBED_ID, RERANK_ID and LLM_ID.

PDF parsing doesn't take part in any of this. Docling runs on the worker's CPU, and the worker image ships CPU-only PyTorch to keep it a reasonable size (about 4 GB).

6. The edge: DNS, ports and TLS

To put the stack on a real domain, point an A record (and an AAAA record if you like) for the domain at the server's public address, and forward TCP 80, TCP 443 and UDP 443 (for HTTP/3) to the server. Check Docker's own firewall chains as well as your host firewall, because only Caddy should be reachable from outside.

Caddy (caddy/Caddyfile) takes care of the rest. It gets and renews certificates automatically and serves HTTP/1.1, HTTP/2 and HTTP/3. It adds security headers (HSTS, nosniff, DENY framing and a strict referrer policy) and hides its own Server header. For /api/*, /healthz and /readyz, it caps request bodies at 25 MB and rate-limits each client address to 120 requests a minute. Everything else is proxied to the workspace, websocket included, and the X-Review-Key header is scrubbed from its access logs. After changing the Caddyfile, validate it with:

docker compose run --rm --no-deps caddy caddy validate --config /etc/caddy/Caddyfile

No port forwarding? Use a Cloudflare Tunnel

If your domain's DNS is on Cloudflare but you can't (or don't want to) open ports on your router, a Cloudflare Tunnel gets you a real domain anyway, and that's how the Józsefváros deployment runs right now at https://paper-trail.jismailov.com. The server opens an outgoing connection to Cloudflare, Cloudflare handles TLS on its side, and visitors' requests come back down that connection to the app. expose.yml publishes the app on the server's localhost only, so the tunnel is the one way in, and Caddy stays off.

Setting it up takes one browser login and a few commands with the official cloudflared package:

cloudflared tunnel login                         # pick your domain in the browser
cloudflared tunnel create paper-trail            # writes ~/.cloudflared/<id>.json
cloudflared tunnel route dns paper-trail paper-trail.example.org

Then copy the credentials file to /etc/cloudflared/, write /etc/cloudflared/config.yml that sends your hostname to the app, and install it as a service that starts on boot:

tunnel: <id>
credentials-file: /etc/cloudflared/<id>.json
ingress:
  - hostname: paper-trail.example.org
    service: http://localhost:8000
  - service: http_status:404
sudo cloudflared service install
sudo systemctl restart cloudflared

The workspace's websocket goes through the tunnel without any extra settings. Like Funnel below, this route skips Caddy, so the same warning about headers and rate limits applies.

No domain at all? Use Tailscale Funnel

If you don't have a domain yet, you can still get the app online without Caddy. expose.yml publishes the app on the server's localhost only, and Tailscale Funnel puts HTTPS in front of it with a https://<machine>.<tailnet>.ts.net address:

cd deploy/zakazserver
export COMPOSE_FILE=docker-compose.yml:expose.yml   # keep this for every compose command
docker compose up -d api worker searxng backup       # caddy stays off
sudo tailscale funnel --bg 8000                      # first run prints a link to enable Funnel
sudo tailscale funnel --bg 8000 off                  # take it offline again

tailscale funnel status shows whether it's currently on and which address it's serving.

Warning

This route skips Caddy, which means no extra security headers and no /api rate limit. Writes still need the review key. Once the domain is ready, switch over by dropping expose.yml from COMPOSE_FILE and starting the caddy service.

7. Backups and restore

The backup service writes one timestamped folder a day into BACKUP_DIR. Each one holds a PostgreSQL custom-format dump, the roles (pg_dumpall --globals-only), an archive of the documents volume and SHA-256 checksums. It keeps seven days, and a run that's still going is named .pending-<stamp> until it finishes.

Warning

The role dump contains password hashes, so lock down the backup folder and encrypt any copy that leaves the host. A backup on the same disk isn't disaster recovery either, so copy finished snapshots off the host every day.

Before anything risky, like a migration, take a consistent snapshot and prove that it restores:

docker compose stop worker
docker compose run --rm backup once
deploy/zakazserver/restore-check.sh /srv/paper-trail/backups/<stamp>
docker compose start worker

restore-check.sh verifies the checksums and the archive, restores into a throwaway Postgres container with no network, counts commitments and links, and then removes the container and its volume. It never touches the live database.

8. Day to day

All maintenance runs through migrate:

docker compose --profile tools run --rm migrate python -m paper_trail.maintenance reset          # dry run
docker compose --profile tools run --rm migrate python -m paper_trail.maintenance reset --yes    # erase machine output
docker compose --profile tools run --rm migrate python -m paper_trail.maintenance rerun          # re-queue every evidence search
docker compose --profile tools run --rm migrate python -m paper_trail.maintenance source list
docker compose --profile tools run --rm migrate python -m paper_trail.maintenance notes generate # needs the extraction phase

Reviewing explains what reset keeps, and Reference has every command.

Upgrading follows the same pattern as the restore check: stop the worker, back up, migrate, then bring everything back.

git pull
docker compose build
docker compose stop worker
docker compose run --rm backup once
docker compose --profile tools run --rm migrate
docker compose up -d

To check health, use GET /healthz, which answers as long as the process is running. GET /readyz goes further and does a database write inside a rolled-back transaction, answering 503 if it can't. The api container's health check uses it, and Caddy waits for it.

Reviewers' per-browser storage (their saved review key and language choice) lives in /app/data/.nicegui on the documents volume, so it survives restarts. Its cookie is signed with a secret derived from REVIEW_KEY, so when you rotate the key every reviewer starts a fresh session and has to enter the new key once.

Before going public

Run through this list before you share the URL:

  • make check passes, and so does scripts/ops/check_postgres.py against a throwaway database (Database)
  • the extraction evaluation has been run for the model you're deploying (Models)
  • a real restore has been done with restore-check.sh
  • a change without the review key gets refused
  • the database, the API port, SearXNG and Goethe can't be reached from outside
  • TLS works on the public domain

Keep the last few dumps and source files around so you can roll back. Reference lists every setting the Compose file sets, and Models explains the models and phases in more depth.

Clone this wiki locally