ops: two misfiring alerts, a blind spot on visual capture, a 44% false-positive gate, and three hardening gaps (from live edge-nl-01 inspection)

[JSONbored]

Summary

Production observability findings from live inspection of edge-nl-01 on 2026-07-27. Two alerts are firing that should not be, one useful signal has no instrumentation at all, and three small hardening gaps. Most of these are config changes on the server, not repo code — noted per item.

1. LoopoverBackupMissing has been firing for 8 days and is wrong (config)

Firing continuously since 2026-07-19T07:44:18Z with target="sqlite".

Postgres backups are healthy: manifest and postgres/ + qdrant/ directories written 2026-07-27 06:15, 6.2 GB retained, BACKUP_RETAIN=7. The sqlite/ directory was last written 2026-07-15 23:34 — it is the retired target from the Postgres cutover.

The alert expression is loopover_backup_files{target=~"postgres|sqlite"} == 0, so the dead sqlite target keeps it permanently red. Eight days of continuous warning is exactly the fatigue that would mask a genuine backup failure.

• Scope the alert to active backup targets only (Prometheus/alertmanager rules on edge-nl-01).
• Stop the backup exporter from reporting a metric for a retired target, or explicitly mark it retired.

2. LoopoverRequestLatencySLOBreach is unactionable (code + config)

Fired ~17:23–20:15 UTC on 2026-07-27, reporting p95 = 9.75 s against a 1 s SLO.

But the histogram has no route label — sum by (le, route) (rate(loopover_http_request_duration_seconds_bucket[5m])) returns a single unlabelled series — and total traffic is ~0.6 requests/minute. A p95 over a 5-minute window is therefore computed from roughly three requests. The alert cannot be attributed to a route and is not statistically meaningful.

• Add a bounded route (or path-group) label to loopover_http_request_duration_seconds. Cardinality is already handled carefully elsewhere in src/selfhost/metrics.ts — follow that discipline, use the route pattern, never the raw path.
• Gate the alert on a minimum request rate so a handful of slow background requests cannot trip an SLO page.

3. No metric on visual-capture failure (code)

Covered in detail in #9464, restated here because it is the observability half: src/review/visual/shot.ts:313-373 logs render_screenshot_error with no counter, and src/selfhost/health.ts has readiness probes for Redis, Qdrant, the GitHub App and codex but none for BROWSER_WS_ENDPOINT. A browserless outage is invisible in Prometheus while it silently affects gate outcomes.

4. secret_leak gate is 44% false-positive on metagraphed (config)

The engine's own anomaly detector said so at 2026-07-27T20:03:05Z:

gate false-positive spike: secret_leak blocked 9 PR(s), 4 merged anyway (44% false-positive, 0 overridden) — the gate is holding mergeable PRs. Keep it advisory / loosen it.

Detector: src/review/ops-wire.ts:161-169. Gate policy lives in the private server-side config (/opt/loopover/loopover-config/jsonbored__metagraphed/.loopover.yml), not the public repo.

• Investigate secret_leak precision on metagraphed and either loosen the rule or move it to advisory, per the detector's own recommendation.

5. Three small hardening items (code)

Empty secret file silently degrades to "unset" — src/selfhost/load-file-secrets.ts:26-45. A missing or unreadable <NAME>_FILE correctly throws (#6284), but a zero-byte or whitespace-only file sets env[NAME] = "", which every downstream nonBlank() treats as unconfigured; preflight.ts:173 deliberately skips absent values. So a truncated GITHUB_WEBHOOK_SECRET file boots an instance that rejects every webhook. This is directly adjacent to the known secret-rotation footgun on this host.

• Treat an empty post-trim secret file as the same fatal error as a missing one.

Crash handler can stall exit behind telemetry flush — src/selfhost/process-lifecycle.ts:81-92 awaits flush() with no deadline before exit(1), wired to flushPostHog (src/server.ts:334). A wedged PostHog egress delays or prevents the very restart the module exists to guarantee.

• Promise.race([flush(), sleep(3000)]).

Half-written blobs are served forever — src/selfhost/blob-store.ts:48-52 writes with writeFile directly. Keys are input-hash-addressed (loopover/shots/<hash>.png), so a truncated PNG from a mid-write kill becomes a permanent, "valid" cache hit. The same codebase already does tmp+rename correctly at src/selfhost/private-config.ts:466-469.

• Write to ${target}.tmp-<uuid> then rename.

6. Host pressure watches CPU only (code)

src/selfhost/host-pressure.ts:12-22 uses loadavg / cpus().length, consumed at src/selfhost/maintenance-admission.ts:269-282. On this GPU box the realistic killer is memory — Ollama at 9.86 GiB and browserless at 1.45 GiB — and nothing sheds or even observes available memory; the OOM killer decides. It also mis-normalises if the container is ever given a cpuset or CPU quota, since it reads host loadavg over container cores.

• Add a memory dimension to the pressure signal.

(The livelock question is otherwise well handled — #9233's 1.5 s pressure-signal memo plus the 4 h trickle and 10 min drain escapes bound starvation correctly.)

[linear-code]

JSO-6551