Troubleshooting

Troubleshooting

Symptom → cause → fix tables for the most common OPNGMS failures, grouped by area, with the diagnostic commands to confirm each one. For the happy-path install see Installation; for every environment variable in detail see Configuration; for hardening notes see Security.

---

Contents

• Startup & containers
• Login & sessions
• TLS models
• Devices & connector
• Reports & email
• Config editor & catalogs
• Log lake
• Collecting diagnostics

Note: Every docker compose command below assumes you pass the same -f file set you started the stack with. The examples use -f docker-compose.prod.yml; append your TLS overlay (-f docker-compose.tls.yml, -f docker-compose.caddy.yml, -f docker-compose.traefik.yml) or the log-lake overlay (-f docker-compose.logs.yml) wherever your bring-up used it, or substitute -f docker-compose.full.yml if you run the all-in-one file.

---

Startup & containers

The deploy is six core services: db (TimescaleDB), redis, a one-shot migrate, api (uvicorn), worker (arq), and frontend (nginx). api and worker both wait on migrate completing successfully and on db/redis being healthy, so a failure low in that chain blocks everything above it.

Symptom	Cause	Fix
api exits immediately with Refusing to start: unedited .env.example placeholder(s) in …	The fail-closed change-me guard found a secret still holding the .env.example placeholder. The guard checks DATABASE_URL, ADMIN_DATABASE_URL, SESSION_SECRET, MASTER_KEY, and APP_ROLE_PASSWORD.	Set real values for the named variables in .env (the error message lists exactly which ones). Generate fresh SESSION_SECRET/MASTER_KEY per Installation; then up -d again.
migrate exits non-zero, api/worker never start	Wrong password pair, or db not yet healthy when migrate ran. migrate connects as the owner via ALEMBIC_DATABASE_URL (= ADMIN_DATABASE_URL) and creates the opngms_app role from APP_ROLE_PASSWORD.	Read logs migrate. If it is an auth error, confirm the two password pairs match: DATABASE_URL password = APP_ROLE_PASSWORD, and ADMIN_DATABASE_URL password = POSTGRES_PASSWORD. Fix .env, then re-run up -d (migrate re-runs on every start).
migrate fails with password authentication failed even though .env looks right	POSTGRES_PASSWORD is only applied by the db container on first boot. Editing it later does not change the role inside an existing data volume.	Either set the password to what the volume already has, or wipe and recreate the DB volume (destroys data): docker compose … down -v then up -d.
db never reports healthy	The pg_isready healthcheck (5 s interval, 20 retries) is failing — bad volume permissions, corrupted volume, or an OOM-killed container.	logs db for the Postgres startup error. If the volume is corrupt and disposable, down -v then up -d.
api stuck starting / never healthy	The /healthz healthcheck (10 s interval, 12 retries) can't reach uvicorn — usually because migrate has not completed or the DB connection is failing at runtime.	docker compose ps to see which dependency is unhealthy; then logs migrate and logs api.
Error response from daemon: … no matching manifest / wrong-arch image	Pull hit a registry/arch problem. Images are multi-arch (linux/amd64, linux/arm64) and published only from semver tags.	Re-run … pull. Pin a known-good release with OPNGMS_VERSION=0.1.0 in .env rather than latest. To run unreleased code, build locally (docker build … ./backend) and up -d without pull.
docker compose errors on the !override YAML tag	Docker Compose older than v2.24.4. The TLS overlay files use ports: !override.	Upgrade Docker Compose. Confirm with docker compose version.

Read each service's logs (a non-zero migrate is the most common blocker):

docker compose -f docker-compose.prod.yml ps          # which service is unhealthy / exited
docker compose -f docker-compose.prod.yml logs migrate # one-shot; should exit 0
docker compose -f docker-compose.prod.yml logs api
docker compose -f docker-compose.prod.yml logs worker
docker compose -f docker-compose.prod.yml logs db
docker compose -f docker-compose.prod.yml logs redis

Hit the API healthcheck the same way the container does:

docker compose -f docker-compose.prod.yml exec api \
  python -c "import urllib.request; print(urllib.request.urlopen('http://localhost:8000/healthz').status)"
# expect: 200

---

Login & sessions

OPNGMS uses server-side sessions carried in a Secure, HttpOnly, SameSite=Lax cookie (opngms_session) plus a non-HttpOnly CSRF cookie (opngms_csrf). The Secure flag is always set — there is no plain-HTTP fallback — so the SPA must be reached over HTTPS or the browser silently discards the cookie.

Symptom	Cause	Fix
Login succeeds (curl/network tab shows 200) but the browser bounces straight back to the sign-in page	The SPA is served over plain HTTP on a real domain, so the browser drops the Secure session cookie.	Put TLS in front (see TLS models). On Model 1, the upstream proxy must forward X-Forwarded-Proto: https — uvicorn runs with --proxy-headers --forwarded-allow-ips * to honour it, but it only sees https if the proxy sends it.
Logged out unexpectedly after a short time	Sessions are TTL-bound (full session default 12 h). An MFA-pending session lives only ~5 min and an MFA-setup session 1 h by design.	Expected for the pending/setup states — complete the second factor promptly. For full sessions, this is the configured session_ttl_hours.
429 Too many attempts with a Retry-After header at login	Sliding-window rate limiter tripped: 5 failed attempts per 15-minute window, keyed on `email	client-IP(andmfa
Last superadmin locked out of MFA (401 Invalid code, no authenticator, recovery codes gone)	TOTP enrolled, device/codes lost — no in-app path to clear it for yourself.	Break-glass CLI clears the user's TOTP + recovery codes (audited as mfa.cli_reset): docker compose -f docker-compose.prod.yml exec api python -m app.cli mfa-reset --email admin@example.com. They can re-enrol on next login.
POST /api/login/mfa returns 400 No MFA challenge	The session is not in the mfa_pending state (e.g. the MFA step was called without first completing POST /api/login, or the pending session expired).	Restart the flow: POST /api/login with password first, then POST /api/login/mfa with the code, within the pending window.
POST /api/setup returns 409 Conflict — Setup already completed: at least one user already exists.	The one-time bootstrap endpoint refuses once any user exists.	This is expected after first run. To add more users, sign in and create them in the admin console — /api/setup is bootstrap-only.
/api/setup (or creating a user) rejects the email address	Email goes through RFC-compliant validation (email-validator). Reserved/undeliverable-looking TLDs such as .local, .internal, .test are rejected.	Use a real, deliverable email domain.
/api/setup rejects the password	Password policy enforces a minimum length (12 characters).	Choose a longer password.

Verify the API is reachable and the bootstrap endpoint behaves as expected (run from the host, against your real URL):

# Create the first superadmin (only works while zero users exist)
curl -i -X POST https://<your-domain>/api/setup \
  -H 'Content-Type: application/json' \
  -d '{"email":"admin@example.com","name":"Admin","password":"<strong-password>"}'
# 201 = created; 409 = a user already exists

---

TLS models

HTTPS is mandatory in production because of the Secure cookie. The overlay files are mutually exclusive — apply exactly one. See Installation for the full model descriptions.

Symptom	Cause	Fix
docker compose errors on !override when adding a TLS overlay	Compose < v2.24.4 (the tls/caddy/traefik overlays use the !override merge tag).	Upgrade Docker Compose; check docker compose version.
Model 2: browser shows a TLS / self-signed warning	nginx generated a self-signed certificate because no real cert was found. The frontend entrypoint self-signs when ${CERT_DIR}/fullchain.pem + privkey.pem are missing so the container still boots.	Place a real fullchain.pem + privkey.pem in ${CERT_DIR} (default ./certs), set SERVER_NAME, and restart the frontend container.
Models 3a/3b: Let's Encrypt never issues a certificate	The HTTP-01 challenge can't complete: the DOMAIN A/AAAA record doesn't point at this host, or ports 80/443 aren't reachable from the public internet.	Confirm DNS resolves to this host and both ports are open inbound. Watch the issuer's logs while it retries.
Login still fails on Model 1 even though TLS is "in front"	The upstream proxy terminates TLS but doesn't tell the app — so the app issues cookies for a scheme the browser sees as mismatched, or the proxy itself talks plain HTTP to the app without the header.	The proxy must add X-Forwarded-Proto: https. nginx preserves it end-to-end; for cloud LBs/ingress, enable proxy-protocol/forwarded headers.

Check DNS and port reachability for automatic TLS, and inspect what the frontend is actually serving:

# DNS points at this host?
dig +short A your.domain
dig +short AAAA your.domain

# 80/443 reachable from outside (run from a different network)
nc -vz your.domain 80
nc -vz your.domain 443

# What cert is the frontend presenting? (self-signed vs real)
echo | openssl s_client -connect your.domain:443 -servername your.domain 2>/dev/null \
  | openssl x509 -noout -issuer -subject -dates

# Confirm the proxy is forwarding the scheme header into the app
docker compose -f docker-compose.prod.yml logs frontend | grep -i x-forwarded

---

Devices & connector

The worker polls each managed OPNsense box over HTTPS using an API key/secret stored encrypted at rest (Fernet, keyed by MASTER_KEY). Telemetry is refreshed by the enqueue_device_polls cron (default every 60 s) which fans out poll_device jobs.

Symptom	Cause	Fix
Device reachability/test-connection fails	Wrong API key/secret (surfaces as an auth error), wrong address, host unreachable (DNS/TCP/timeout), or a TLS verification/pinning failure.	Check each in turn: re-enter the API key & secret from the OPNsense box; confirm the address/port; ensure the host is reachable from the server (not your laptop). If using a self-signed box cert, set the TLS fingerprint pin or disable CA verification for that device.
Test passes but telemetry never updates; device shows unverified	A later poll hit a connector error (auth/reachability) on one of the identity/system/interface/gateway/VPN reads — the poll falls back and marks the device unverified without writing metrics.	Read logs worker for the per-device error. Re-run the reachability test; fix creds/network/TLS; status recovers on the next successful poll.
Edition/version shows blank or doesn't resolve	The connector reads core/firmware/status to parse edition (community/business/devel) and version. If the box is unreachable, or returns an unparseable version, resolution falls back.	Confirm the box is reachable and the firmware status endpoint returns data (the reachability test reports the detected firmware version). An unparseable version is treated as the newest capability profile, so most features still work.
Config apply/push stays scheduled and never runs	The per-device advisory lock was unavailable (another op in flight) or the job was orphaned (worker restart mid-apply).	The sweeper (sweep_orphaned_actions, every ~5 min, with a ~5 min grace) re-enqueues genuinely stuck scheduled rows up to a retry cap, then marks them failed and raises an action_orphaned alert. Wait a cycle, or check logs worker.
Config apply lands in conflict	The staleness guard re-read the device and the canonical config hash no longer matches the baseline_hash captured when the change was drafted — the box changed underneath you.	Re-draft the change against the current config and re-apply.
Config apply lands in failed	The connector raised an error during the live push, or the change kind was unknown.	logs worker for the apply error. Fix the underlying cause and re-apply. Note that live mutations only happen when live-push is enabled; otherwise applies run dry.
A previously-applied change was reverted / can't be reverted	Revert is available for changes in applied/failed state whose kind has an inverse builder — all live-applied kinds (firewall_alias, opnsense_setting, firewall_rule, monit_test, ids_policy, catalog_setting) except ids_rulesets (its apply is additive). A set/delete revert needs the pre-apply snapshot.	If revert is unavailable, the kind has no inverse builder (ids_rulesets) or the snapshot is missing — re-apply the desired state forward instead.

# Watch the polling / apply / sweeper activity live
docker compose -f docker-compose.prod.yml logs -f worker

# Confirm the server itself can reach an OPNsense box (creds + TLS), from inside the api container
docker compose -f docker-compose.prod.yml exec api \
  python -c "import socket; socket.create_connection(('<box-address>', 443), 5); print('tcp ok')"

---

Reports & email

Scheduled reports are generated and emailed by the worker (enqueue_due_reports → deliver_scheduled_report → send_report_email_job, which retries up to 12 times at ~10 min spacing). SMTP is configured entirely in-app (Admin → SMTP delivery) — there are no SMTP variables in .env; the password is encrypted at rest with MASTER_KEY. See Reporting for the schedule model.

Symptom	Cause	Fix
Scheduled reports never arrive	SMTP delivery is not enabled, or the relay settings are wrong.	Admin → SMTP delivery: enable delivery and use Send a test email to confirm the relay. Until enabled, send jobs error with smtp not configured.
Reports still don't fire even though SMTP works	The schedule is disabled, has no recipients, or (device-scoped) its device was deleted.	Confirm the schedule is enabled and has at least one recipient. A schedule with zero recipients returns no-recipients and never sends; a device-scoped schedule whose device was removed auto-disables.
Send a test email fails	The relay rejected the connection or auth (DNS/host unreachable, bad port/security mode, bad username/password).	Re-check host, port, security (STARTTLS / implicit TLS / none), and credentials. The error detail is returned redacted (no credentials) — inspect logs api/logs worker for context.
Test email works but scheduled delivery exhausts retries	Transient relay errors at send time, or a per-message size/recipient rejection by the relay.	logs worker around the schedule's UTC hour. Note schedule hours are UTC by design — changing TZ shifts only log timestamps, not when reports fire.

# Watch report generation + email send + retries
docker compose -f docker-compose.prod.yml logs -f worker | grep -iE "report|smtp|email"

---

Config editor & catalogs

The version-aware config editor renders forms from a per-(edition, version) catalog that the backend fetches from published release assets, verifies by SHA-256 against the signed manifest.json, and caches. A failed integrity check fails closed. See Configuration-Editor for the editor workflow and Configuration for the catalog settings.

Symptom	Cause	Fix
Editor shows an empty state / No catalog available for this device version (404)	No catalog could be resolved for the device's edition+version: auto-fetch disabled with an empty cache, the host is offline, the device's version isn't published, or a SHA-256 mismatch caused the fetched catalog to be rejected and not cached.	Confirm the device's edition/version is resolved (see Devices) and that a matching catalog is published. If a publish was tampered/partial, the SHA-256 check rejects it — re-publish and let it re-fetch.
A catalog updates upstream but the editor still shows the old shape	The previous catalog is cached; a SHA-256 mismatch on the new fetch keeps the old cached copy in place.	Re-run the catalog publish action so manifest.json + the catalog asset are consistent, then the next fetch validates and replaces the cache.
Editor loads the form but live values are blank and editing is disabled	The catalog resolved, but the live read from the device failed (box unreachable/unreadable), so the editor returns reachable: false with empty values.	Fix device reachability (see Devices); the live values populate once the box responds.
Business-edition device shows no catalog	Business editions are mapped to a Community base version via business-base.json; if the mapping or the resolved Community catalog is missing, resolution returns nothing.	Ensure the published catalog set includes the Business→Community base mapping and the resolved Community version.

# Ask the API what catalog (if any) resolves for a device — 200 with body or 404
curl -i https://<your-domain>/api/tenants/<tenant-id>/devices/<device-id>/catalog \
  -b 'opngms_session=<your-session-cookie>'

# Catalog fetch / SHA-256 verification lives in the API logs
docker compose -f docker-compose.prod.yml logs api | grep -iE "catalog|sha256|manifest"

---

Log lake

The optional log lake (overlay docker-compose.logs.yml, or bundled in docker-compose.full.yml) adds three services: opensearch, a one-shot syslog-bootstrap, and syslog-ng. Devices ship logs over mTLS to port 6514; syslog-ng writes daily opngms-logs-YYYY.MM.DD indices to OpenSearch over the internal network. See Log-Lake for the full bring-up.

Symptom	Cause	Fix
syslog-bootstrap doesn't complete; syslog-ng won't start	The one-shot ensures the CA, writes CA.pem/server.pem/server.key to the certs volume, and applies the OpenSearch index template + ISM retention policy. syslog-ng waits for it to exit 0.	logs syslog-bootstrap. It connects as the DB owner (ADMIN_DATABASE_URL) and reaches OpenSearch at OPENSEARCH_URL — both must be set. Re-run: up -d (or restart syslog-bootstrap); add --force to overwrite existing cert files.
A device isn't shipping logs	Port 6514 isn't reachable from the device to SYSLOG_RECEIVER_HOST, or the device's mTLS client cert is missing/expired/revoked or signed by the wrong CA.	Confirm 6514 is open from the device's network to the receiver's public name/IP. Confirm the device has a current client cert (its Subject carries the device + tenant IDs); re-provision if expired/revoked. Check logs syslog-ng for TLS handshake rejections.
Logs page is empty (no error)	No logs shipped yet for this tenant, the query's time window excludes them, or your tenant doesn't match the logs' tenant_id (queries are tenant-scoped).	Widen the time range; confirm at least one device is provisioned and shipping; verify in OpenSearch directly (below).
Logs page errors / OpenSearch not reachable	The opensearch container isn't running, or OPENSEARCH_URL is wrong. OpenSearch is internal-only plain HTTP (security disabled, not host-published).	docker compose ps; logs opensearch. Query cluster health from inside the network (below).
Index template / retention not applied	syslog-bootstrap didn't apply them (failed or first-run race).	Re-run syslog-bootstrap with --force and verify the template + ISM policy exist (below).

# Is syslog-ng listening on 6514?
docker compose -f docker-compose.logs.yml exec syslog-ng ss -tuln | grep 6514

# OpenSearch reachable from inside the compose network + cluster health
docker compose -f docker-compose.logs.yml exec api \
  python -c "import urllib.request,os; print(urllib.request.urlopen(os.environ['OPENSEARCH_URL']+'/_cluster/health').read().decode())"

# List the daily log indices and confirm docs are arriving
docker compose -f docker-compose.logs.yml exec api \
  python -c "import urllib.request,os; print(urllib.request.urlopen(os.environ['OPENSEARCH_URL']+'/_cat/indices/opngms-logs-*?v').read().decode())"

# Verify the index template + ISM retention policy are applied
docker compose -f docker-compose.logs.yml exec api python - <<'PY'
import urllib.request, os
base = os.environ["OPENSEARCH_URL"]
for path in ("/_index_template/opngms-logs", "/_plugins/_ism/policies/opngms-logs-retention"):
    try:
        print(path, "->", urllib.request.urlopen(base + path).status)
    except Exception as e:
        print(path, "->", e)
PY

# TLS handshake / shipping errors
docker compose -f docker-compose.logs.yml logs syslog-ng | tail -100

Note: The multi-node overlay (docker-compose.logs.multinode.yml) runs three OpenSearch nodes (opensearch-n1/n2/n3) and applies a 2-shard / 1-replica template instead of 1-shard / 0-replica, so it tolerates one node failing. Service names differ — adjust exec/logs targets accordingly.

---

Collecting diagnostics

When opening an issue or escalating, gather this first. Substitute your real -f file set.

# 1. Versions
docker version
docker compose version          # must be >= v2.24.4 for the TLS overlays

# 2. Service state — what's up, exited, or unhealthy
docker compose -f docker-compose.prod.yml ps

# 3. Recent logs from every core service (timestamped, last 200 lines each)
for svc in migrate api worker db redis frontend; do
  echo "===== $svc ====="
  docker compose -f docker-compose.prod.yml logs --no-color --timestamps --tail 200 "$svc"
done

# 4. API healthcheck, the way the container probes it
docker compose -f docker-compose.prod.yml exec api \
  python -c "import urllib.request; print(urllib.request.urlopen('http://localhost:8000/healthz').status)"

# 5. Effective config sanity — confirm no change-me placeholders survived
#    (greps the merged compose env without printing secret values)
docker compose -f docker-compose.prod.yml exec api \
  python -c "import os; print({k:('SET' if v else 'EMPTY') for k,v in os.environ.items() if k in {'DATABASE_URL','ADMIN_DATABASE_URL','SESSION_SECRET','MASTER_KEY','APP_ROLE_PASSWORD'}})"

Note: Never paste raw .env contents or logs that include secrets into an issue. The app deliberately keeps secrets out of error messages (SMTP/connector errors are redacted), and the snippet above reports only whether a secret is set, not its value.

---

Related pages: Installation · Configuration · Upgrading · Security · Configuration-Editor · Log-Lake · Reporting · Architecture · Development · Home