Troubleshooting

Troubleshooting

Failure modes visible in the code, with the text you will actually see.

The container refuses to start

RuntimeError: Refusing to start: DISABLE_AUTH=true with a production-like
configuration (APP_ENV=production, secrets_configured=True).

DISABLE_AUTH is a development-only escape hatch and the startup guard turns it into a loud deploy failure rather than a silent fleet-wide security hole. A deployment counts as production-like when APP_ENV names prod or staging or when REPORTMATE_PASSPHRASE / API_INTERNAL_SECRET is set. Set APP_ENV=development to use it locally, or unset DISABLE_AUTH.

Migrations did not run

[STARTUP] Migrations failed: ...

Startup logs this and continues. The app will serve reads against whatever schema exists, so a healthy container does not mean a migrated schema — check for this line after every deploy. /api/v1/health/ready reports whether the database is reachable, not whether it is at head.

Everything returns 401

{"error":"Unauthorized","detail":"Authentication required. Supply X-API-Key (per-client), X-Client-Passphrase (clients), X-Internal-Secret (internal). See /api/v1/auth/config.","status_code":401}

No credential was presented. When OIDC is on, the list starts with Authorization: Bearer (OIDC sign-in). Only /api/v1/health, /api/v1/health/live, /api/v1/health/ready, and /api/v1/auth/config are anonymous; the last tells a client which credentials the deployment accepts.

{"error":"Internal server error","detail":"..."}

with [ERR] REPORTMATE_PASSPHRASE not configured but client attempted passphrase auth in the log means the caller sent a passphrase and the server has none configured. This is a 500 because it is a server misconfiguration, not a bad credential.

A valid credential returns 403

{"error":"Forbidden","detail":"This credential lacks the required 'admin' scope.","status_code":403}

The scope is derived from the request, not the route: any path containing /admin needs admin, POST needs ingest, PUT/PATCH/DELETE need admin, everything else needs read. Mint a key with the scope you need — key scopes cannot be edited, only replaced.

On the settings endpoints you may instead see:

{"error":"Forbidden","detail":"This endpoint requires internal service authentication","status_code":403}

PUT /api/v1/settings and GET /api/v1/settings/inventory/discover require the authentication method to be internal_secret. A fleet passphrase holds the admin scope and is still refused here on purpose — the fleet credential must not be able to rewrite org settings or enumerate inventory values.

Devices are being throttled

{"error":"Too many requests","detail":"Rate limit exceeded: 3000 per minute per address","status_code":429}

per address means the address bucket, and behind a proxy that almost always means X-Forwarded-For is not reaching the app — every device then shares one bucket keyed on the ingress address. Fix the proxy first; raising RATE_LIMIT_IP_PER_MINUTE only moves the cliff.

per device means one machine is genuinely check-in looping. Throttled ingest POSTs are recorded as throttle / rate_limited in GET /api/v1/events/failures, so you can see which serial.

Note that the limiter counts in process memory per replica, so the effective fleet-wide ceiling is the configured value times the replica count.

Bulk endpoints time out or return empty

The database is likely fine and the pool is not. Endpoints run in Starlette's threadpool and block waiting for a connection; with the pool ceiling below the threadpool size, a burst of slow bulk requests holds every connection and queues everything else past the dashboard's own abort. Raise DB_POOL_MAX toward the threadpool size (default 40) rather than assuming a slow query.

A query that genuinely runs long is cut off by the per-connection statement_timeout of 120 seconds; the socket timeout is 130 so the server, not the socket, is the arbiter.

A device is installed but never appears

This is what GET /api/v1/events/failures exists for. Query it by serial:

curl -H "X-API-Key: $KEY" \
  "https://api.example.com/api/v1/events/failures?serial=WS-0001&outcome=all&hours=168"

Common reasons and what each means:

Reason	What happened
missing_credentials	The agent has no passphrase or API key configured.
invalid_passphrase / invalid_api_key	The agent has the wrong one, or you retired a passphrase entry too early.
sentinel_serial	The machine's BIOS reports a placeholder serial such as To Be Filled By O.E.M.
serial_equals_hostname	The agent sent its hostname where the hardware serial belongs.
upload_aborted / empty_body	The upload died in flight. If the outcome is retried, the client's retry landed and nothing is wrong.
malformed_json	Read the transport note in detail — a received well below declared is a truncated upload, not a serializer bug.
rate_limited	See above.
usage_write_failed	The check-in succeeded but its usage rows were lost.
usage_client_too_old	The check-in was stored, but the agent build is older than the usage floor, so its usage rows were dropped. Update the agent.

Also check outcome. The default view is rejected — turned away with nothing since. retried rows are dropped uploads whose retry landed and are not a problem; accepted rows are check-ins that got in but carried a client defect worth watching.

A passphrase rotation seems stuck

[ROTATION] Deprecated passphrase (position 2 of 2) accepted from ... (IP: ...).
This client has not picked up the current passphrase; do not retire that
entry until these stop.

REPORTMATE_PASSPHRASE is a comma-separated list; the first entry is current and later ones are deprecated-but-accepted. Watching this warning fall silent is how you know the fleet has finished rolling over and the old entry is safe to remove.

Dashboard install counts do not match /installs/full

The dashboard sums the precomputed cimian_errors / cimian_warnings / munki_errors / munki_warnings columns on installs; /installs/full reads the JSONB. Ingest keeps them in step, and the clear-errors admin endpoint recomputes them in the same statement it rewrites the JSONB. If they have drifted, or the classifier has changed since the rows were stored, run POST /api/v1/admin/installs/reclassify — not an edit to the JSONB.

Two other gaps look like this. A device an inventory reports as in storage is left out of the dashboard's counts but still appears in /installs/full. And a module row whose device row is gone appears in neither, because every fleet query joins back to devices; GET /api/v1/admin/orphans reports those and what they would have added to the tiles.

A device shows an error that cleared hours ago

Check installs.collectedAt on /api/v1/installs/full. The installs module only refreshes after a managed-software session that did work, so it can be older than the device's lastSeen. A run's events are replaced by the next run's, so the error goes when the next installs module arrives.

A run that failed only because the device was offline does not show as an error at all: transient network failures are suppressed at ingest and kept under transientProblems. See Ingest-Pipeline.

Fresh data does not show up immediately

Read responses are cached in process for 15–120 seconds depending on the endpoint, and the ingest path deliberately does not invalidate them. With several replicas, two consecutive requests can be served results a TTL apart. Admin, settings, and inventory-state writes do invalidate, so those are visible on the next read.

A 500 with a reference id

{"error":"Internal server error","detail":"Internal server error. Reference: 9f2c41ab7d03","reference":"9f2c41ab7d03","status_code":500}

Server-error detail is masked because it can carry SQL, connection strings, or stack fragments. The reference matches the X-Request-ID response header and the server log line — grep the container logs for it. If the 500 was on the ingest path, it is also recorded in ingest_failures as server_error / internal_error with the same reference in its detail.

Real-time events do not arrive

{"url":null,"accessToken":null,"error":"EVENTS_CONNECTION not configured"}

or "error": "WebPubSub SDK not installed". Both are returned with HTTP 200 so the dashboard falls back to polling instead of erroring. Set EVENTS_CONNECTION if you want the live stream.

The image crash-loops with ModuleNotFoundError

The Dockerfile copies an explicit file list. A new top-level module that is imported but not copied builds green and fails at startup. tests/test_dockerfile.py catches this — if it is failing, add the COPY line.