Upgrading

Day-two operations for an OPNGMS deployment: moving between releases, the automatic schema migration, backups, the forward-only rollback caveat, MASTER_KEY rotation, and routine maintenance. For the first install see Installation; for every environment variable see Configuration.

The standard upgrade flow

Upgrading is always the same two commands — pull the new images, then up -d to recreate the changed containers. The only variable is which compose files you pass, which must match the TLS model you deployed with (see Installation). Always pass the same -f set you used to bring the stack up.

TLS model	Compose files	Upgrade command
1 — behind your proxy	base only	`docker compose -f docker-compose.prod.yml pull && docker compose -f docker-compose.prod.yml up -d`
2 — built-in nginx TLS	`+ docker-compose.tls.yml`	`docker compose -f docker-compose.prod.yml -f docker-compose.tls.yml pull && … up -d`
3a — Caddy (Let's Encrypt)	`+ docker-compose.caddy.yml`	`docker compose -f docker-compose.prod.yml -f docker-compose.caddy.yml pull && … up -d`
3b — Traefik (Let's Encrypt)	`+ docker-compose.traefik.yml`	`docker compose -f docker-compose.prod.yml -f docker-compose.traefik.yml pull && … up -d`
All-in-one (core + log lake)	`docker-compose.full.yml`	`docker compose -f docker-compose.full.yml pull && docker compose -f docker-compose.full.yml up -d`
Core + log-lake overlay	`+ docker-compose.logs.yml`	`docker compose -f docker-compose.prod.yml -f docker-compose.logs.yml pull && … up -d`

Worked example, Model 1:

cd /path/to/opngms          # the directory holding your .env and compose files

# 1. (recommended) back up first — see "Back up before you upgrade" below
# 2. pull the new images for the pinned OPNGMS_VERSION
docker compose -f docker-compose.prod.yml pull
# 3. recreate the changed containers; migrate runs first, then api/worker start
docker compose -f docker-compose.prod.yml up -d

up -d only recreates containers whose image or config changed. The db and redis containers keep running, so the upgrade is in-place; the opngms_pg Postgres volume is never touched by pull/up.

Note: Pin a release before pulling (next section). With OPNGMS_VERSION=latest, a fresh pull could fetch a newer release than the one you tested — fine for a deliberate upgrade, surprising if you only meant to restart.

Automatic database migration

You never run Alembic by hand. The compose stack includes a one-shot migrate service that runs alembic upgrade head on every up, and api/worker declare depends_on: migrate → condition: service_completed_successfully. So a new image that needs a schema change applies its migrations before the app starts; if the migration fails, migrate exits non-zero and the api/worker containers never start on the broken schema.

# docker-compose.prod.yml (excerpt)
migrate:
  image: ghcr.io/l0rdg3x/opngms-backend:${OPNGMS_VERSION:-latest}
  command: ["alembic", "upgrade", "head"]
  environment:
    ALEMBIC_DATABASE_URL: ${ADMIN_DATABASE_URL}   # owner role: DDL + creates opngms_app
  restart: "no"
api:
  depends_on:
    migrate:
      condition: service_completed_successfully

migrate connects as the owner role via ADMIN_DATABASE_URL (migrations create the RLS-exempt schema, the non-superuser opngms_app role, RLS policies and grants), not as the RLS-restricted app role.

Watch the migration

After up -d, follow the one-shot migrate container to confirm the schema upgrade applied cleanly:

# Stream migrate output; it exits 0 on success and the container stops
docker compose -f docker-compose.prod.yml logs -f migrate

# Or wait on its exit code explicitly
docker compose -f docker-compose.prod.yml up migrate
echo "migrate exit code: $?"     # 0 = head applied

A healthy run ends with Alembic logging each Running upgrade <from> -> <to> line and the container exiting. If it exits non-zero, the app will not start — inspect the logs (the usual causes are a mismatched DB password pair or the db container not yet healthy; see Installation and Troubleshooting).

Once migrate is done, confirm the app came up:

docker compose -f docker-compose.prod.yml ps          # api should report healthy
docker compose -f docker-compose.prod.yml logs -f api

Version pinning and moving between releases

Images are published only from semver git tags vX.Y.Z — never from a mid-development main commit — so every published tag is a deliberate, complete release. OPNGMS_VERSION in .env selects which tag the compose files pull:

Pin a release for reproducibility — set OPNGMS_VERSION=0.1.0 so every host runs an identical, known image regardless of when it pulls. Recommended for production.
Track the newest release — leave OPNGMS_VERSION=latest (the default); latest always resolves to the most recently tagged release.

To move between releases, edit one line in .env, then pull and recreate:

# .env
OPNGMS_VERSION=0.2.0      # was 0.1.0

docker compose -f docker-compose.prod.yml pull
docker compose -f docker-compose.prod.yml up -d
docker compose -f docker-compose.prod.yml logs -f migrate   # watch the schema upgrade

The same OPNGMS_VERSION value drives the backend, frontend, and migrate images, so all three move together — the migrations that ship in the new backend image are exactly the ones migrate applies.

Note: Upgrade one minor/patch step at a time in production and read the release notes for each vX.Y.Z you cross, so a migration in an intermediate release is never skipped.

Back up before you upgrade

Migrations are forward-only (next section), so a backup taken immediately before the upgrade is your rollback. Back up four things:

What	Why
Postgres database	All app state: tenants, devices, encrypted credentials, config snapshots, reports, audit log.
`.env`	Your secrets — `MASTER_KEY`, `SESSION_SECRET`, DB passwords, `OPNGMS_VERSION`. Without `MASTER_KEY` the encrypted columns in a restored DB cannot be decrypted.
`./certs` (Models 2 / TLS)	Your own `fullchain.pem` + `privkey.pem` if you supply the certificate. (Caddy/Traefik manage their own ACME data in named volumes.)
Log-lake CA / certs (Log-Lake)	The syslog receiver cert files in the `opngms_syslog_certs` volume. The CA itself lives inside the database (encrypted with `MASTER_KEY`), so it is already covered by the DB dump + `.env`.

Dump the database

Run pg_dump inside the running db container as the owner, redirecting to a file on the host:

# Plain-SQL dump of the whole database (owner role; reads POSTGRES_USER/DB from .env)
docker compose -f docker-compose.prod.yml exec -T db \
  pg_dump -U "$POSTGRES_USER" -d "$POSTGRES_DB" \
  > "opngms-$(date +%F).sql"

# Or a compressed custom-format dump (smaller, restorable selectively with pg_restore)
docker compose -f docker-compose.prod.yml exec -T db \
  pg_dump -U "$POSTGRES_USER" -d "$POSTGRES_DB" -Fc \
  > "opngms-$(date +%F).dump"

Note: $POSTGRES_USER / $POSTGRES_DB here are the host shell's view of your .env; if your shell does not export them, substitute the literal values (e.g. -U opngms -d opngms). Store the dump alongside a copy of .env from the same moment — they are a matched pair.

Restore (into a freshly initialised, empty db volume — see the rollback section):

# plain SQL
cat opngms-2026-06-13.sql | docker compose -f docker-compose.prod.yml exec -T db \
  psql -U "$POSTGRES_USER" -d "$POSTGRES_DB"

# custom format
cat opngms-2026-06-13.dump | docker compose -f docker-compose.prod.yml exec -T db \
  pg_restore -U "$POSTGRES_USER" -d "$POSTGRES_DB" --clean --if-exists

Rollback: forward-only migrations

OPNGMS upgrades are forward-only. The migrate service runs alembic upgrade head and nothing else — a deploy never runs a down-migration. Re-pinning OPNGMS_VERSION to an older tag and pulling rolls the images back, but it does not roll the schema back: the database stays at the newer head, and an older backend may not run against it.

The supported rollback path is therefore restore from the pre-upgrade backup, not "downgrade the image":

# 1. Stop the stack
docker compose -f docker-compose.prod.yml down

# 2. Re-pin the previous release in .env
#    OPNGMS_VERSION=0.1.0

# 3. Reset the database to a clean, empty volume so the old schema can be restored
docker volume rm opngms_opngms_pg        # name is <project>_opngms_pg; check `docker volume ls`

# 4. Bring up just the db, restore the matching dump, then start the rest
docker compose -f docker-compose.prod.yml up -d db
#    …wait for db healthy, then restore as shown in "Back up before you upgrade"…
docker compose -f docker-compose.prod.yml up -d

Note: Restore the dump that was taken on the same release you are rolling back to, together with that release's .env (the MASTER_KEY must match the encrypted data in the dump). This is why a fresh backup before every upgrade is non-negotiable. The Alembic migration files do define downgrade() steps, but they are not invoked by any deploy path and are not a supported production rollback mechanism.

Rotating `MASTER_KEY`

MASTER_KEY is the Fernet key that encrypts every secret at rest in the database. Five kinds of data are encrypted with it:

Data	Where
Device API key + secret	`devices`
Config snapshots (the captured `config.xml`)	`config_snapshots`
MFA TOTP secrets	`user_mfa`
SMTP relay password	`smtp_settings`
Internal syslog CA private key	`syslog_ca`

Why a rotation can be zero-downtime. The crypto layer uses Fernet's MultiFernet: it encrypts with the primary MASTER_KEY and decrypts with the primary or any key listed in MASTER_KEY_OLD_KEYS. So you can introduce a new primary key while the old one still decrypts existing data, re-encrypt everything in the background, and only then retire the old key — the app keeps serving throughout. See Security for the at-rest threat model.

Procedure

Generate a new key:

python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"

Demote the current key, promote the new one — in .env, move the old MASTER_KEY value into MASTER_KEY_OLD_KEYS (comma-separated; it is decrypt-only) and set MASTER_KEY to the new key:
```
MASTER_KEY=<new-key>
MASTER_KEY_OLD_KEYS=<previous-key>     # retained so existing ciphertext still decrypts
```
Apply the new env and re-encrypt every stored secret. Recreate the containers so they read the new keys, then run the bundled rekey script as the owner role (it must bypass RLS to see every tenant's rows):
```
docker compose -f docker-compose.prod.yml up -d        # api/worker pick up the new keys
docker compose -f docker-compose.prod.yml exec api \
  python -m app.scripts.rekey_secrets
# prints: re-keyed <N> encrypted records
```
The script re-encrypts every *_enc column — devices (API key + secret), config_snapshots, user_mfa (TOTP secret), smtp_settings (password), and syslog_ca (CA key) — under the new primary key, decrypting each with whichever key fits. (A test in the suite enumerates the model's encrypted columns and fails if any is left uncovered, so the script can't silently miss one.) It refuses to run unless ADMIN_DATABASE_URL is set — running as the RLS-restricted app role would silently match zero rows and "re-key 0", which would then make every secret undecryptable once the old key is retired.
Retire the old key. Only after the rekey script reports success, remove the previous key from MASTER_KEY_OLD_KEYS and recreate the containers once more:
```
MASTER_KEY_OLD_KEYS=
```
```
docker compose -f docker-compose.prod.yml up -d
```

Note: Keep the retired key in MASTER_KEY_OLD_KEYS until the rekey script has completed successfully. If you retire it too early, any ciphertext that was not yet re-encrypted becomes permanently unreadable. The script covers all five encrypted column families above, so no manual re-entry is needed — but it is still wise to confirm the rotation worked before removing the old key: log in with MFA, send a test report email (Admin → SMTP delivery), and open a device's config map. If all three succeed under the new key alone, the rotation is complete.

Routine operations

View logs

docker compose -f docker-compose.prod.yml logs -f api        # request handling, auth
docker compose -f docker-compose.prod.yml logs -f worker     # polling, report delivery
docker compose -f docker-compose.prod.yml logs --tail=200 frontend
docker compose -f docker-compose.prod.yml ps                 # status + health of every service

Set TZ in .env (e.g. TZ=Europe/Rome) to read container logs in local time; stored data and report-schedule hours stay in UTC (see Configuration).

Restart a single service

docker compose -f docker-compose.prod.yml restart worker     # e.g. after a transient polling issue
docker compose -f docker-compose.prod.yml restart api

Prune old images

After several upgrades, superseded image layers accumulate. Reclaim space safely (this only removes images no running container references):

docker image prune -f                  # dangling layers only
docker image prune -a -f               # every image not used by a running container — be sure first

Note: Do not run docker volume prune blindly — the opngms_pg (database), Caddy/Traefik ACME, and opngms_syslog_certs volumes hold state you cannot regenerate. Remove volumes only with the explicit docker volume rm <name> flow shown in the rollback section.

For diagnosing a failed upgrade see Troubleshooting; for the log-lake bring-up and its CA see Log-Lake; for security hardening and the at-rest encryption model see Security.

OPNGMS Wiki

Home Screenshots

Deploy & operate

Understand & extend

Navigation Menu

Upgrading

Upgrading

Contents

The standard upgrade flow

Automatic database migration

Watch the migration

Version pinning and moving between releases

Back up before you upgrade

Dump the database

Rollback: forward-only migrations

Rotating `MASTER_KEY`

Procedure

Routine operations

View logs

Restart a single service

Prune old images

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

OPNGMS Wiki

Clone this wiki locally

Upgrading

Upgrading

Contents

The standard upgrade flow

Automatic database migration

Watch the migration

Version pinning and moving between releases

Back up before you upgrade

Dump the database

Rollback: forward-only migrations

Rotating MASTER_KEY

Procedure

Routine operations

View logs

Restart a single service

Prune old images

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

OPNGMS Wiki

Clone this wiki locally

Rotating `MASTER_KEY`