ops(config): a permanently-firing backup alert, an ungated latency SLO, and a 44% false-positive secret_leak gate on edge-nl-01

[JSONbored]

Parent: #9487

Summary

The server-side config half of the 2026-07-27 edge-nl-01 inspection. None of these are repo code — they live in Prometheus/alertmanager rules and the private per-repo .loopover.yml on the host — which is why they are split out from the code items (#9544, #9545) rather than bundled with them.

1. LoopoverBackupMissing has been firing for 8 days and is wrong

Firing continuously since 2026-07-19T07:44:18Z with target="sqlite".

Postgres backups are healthy: manifest plus postgres/ and qdrant/ directories written 2026-07-27 06:15, 6.2 GB retained, BACKUP_RETAIN=7. The sqlite/ directory was last written 2026-07-15 23:34 — it is the retired target from the Postgres cutover.

The expression is loopover_backup_files{target=~"postgres|sqlite"} == 0, so the dead sqlite target keeps it permanently red. Eight days of continuous warning is exactly the fatigue that would mask a genuine backup failure — the alert is now worse than no alert.

• Scope the alert to active backup targets only.
• Stop the backup exporter reporting a metric for a retired target, or mark it explicitly retired.

2. LoopoverRequestLatencySLOBreach needs a minimum-rate gate

The code half — the missing route label — is #9545. The remaining half is config: at ~0.6 requests/minute, a 5-minute p95 is computed from roughly three requests, so a handful of slow background requests trips an SLO page with no statistical meaning.

• Gate the alert expression on a minimum request rate.

3. secret_leak gate is 44% false-positive on metagraphed

The engine's own anomaly detector said so at 2026-07-27T20:03:05Z:

gate false-positive spike: secret_leak blocked 9 PR(s), 4 merged anyway (44% false-positive, 0 overridden) — the gate is holding mergeable PRs. Keep it advisory / loosen it.

Detector: src/review/ops-wire.ts:161-169. Gate policy lives in the private server-side config (/opt/loopover/loopover-config/jsonbored__metagraphed/.loopover.yml), not the public repo.

• Investigate secret_leak precision on metagraphed and either loosen the rule or move it to advisory, per the detector's own recommendation.

Note

Each item is an operator action on edge-nl-01. Recorded as an issue so they are tracked rather than living only in an audit write-up, and kept separate from the code work so neither blocks the other.

[linear-code]

JSO-6596

[JSONbored]

All three items done on edge-nl-01, verified against live data. promtool check rules passes (41 rules), Prometheus reloaded, and both alerts are now inactive with 0 firing — the 8-day continuous warning has stopped.

alerts.yml backed up to alerts.yml.bak-9546 first.

1. LoopoverBackupMissing — fixed

Live values confirmed the diagnosis exactly: postgres 7, qdrant 7, sqlite 0 — the retired target from the Postgres cutover holding the alert red.

I deliberately did not hardcode the expression to postgres. A self-host install legitimately runs SQLite, and pinning it would silently stop alerting on a genuine SQLite backup failure there — trading one blind spot for another. Instead the alert now requires the target to have actually produced a backup:

loopover_backup_files{target=~"postgres|sqlite"} == 0
and
max_over_time(loopover_backup_files{target=~"postgres|sqlite"}[30d]) > 0

Verified with live 30-day maxima, which is what makes this provably correct rather than merely quiet:

target	30d max	result
postgres	9	0 == 0 AND 9 > 0 → still alerts on a real failure
sqlite	0	0 == 0 AND 0 > 0 → can never fire again

The full expression currently returns an empty vector.

2. LoopoverRequestLatencySLOBreach — fixed (config half)

Added a minimum-rate gate: and sum(rate(loopover_http_request_duration_seconds_count[5m])) > 0.05.

Live rate is 0.0316 req/s (~1.9 req/min) — below the 0.05 floor (3 req/min), so the alert is correctly silent at this traffic and re-arms automatically above it. At the old rate a 5-minute p95 was computed from roughly three requests, so a single slow background call paged with no statistical meaning.

The code half — the missing bounded route label that made a fired alert unattributable — shipped separately in #9545.

3. secret_leak 44% false-positive — investigated; no loosening warranted, the premise is wrong

This one I would have gotten wrong by following the recommendation. The detector said "keep it advisory / loosen it", and loosening a leaked-credential gate on a heuristic FP rate, without first checking what actually matched, would have been reckless. It turns out nothing was wrong with the secret patterns at all.

What the data shows. Pulling the real review_audit rows for metagraphed: 7 distinct PRs (#8106, #8095, #8005, #7586, #5218, #4886, #4582), and all 7 subsequently merged — a 100% false-positive rate on that sample, worse than the reported 44%. But reading the actual PR comment shows the finding was never a secret match:

Patch-less file(s) could not be fully scanned for secrets (2): GitHub omitted inline diff for: generated/graphql/types.ts, package-lock.json. Fetched content exceeded the 512000-char scan cap…

That is secret_scan_incomplete — absence of evidence, not evidence of a leak — and #9082 already made it a neutral hold, not a close (consistent with decision='hold' on every row but the oldest). So the gate was already behaving as "advisory"; moving it there was a no-op request.

And the underlying cause is already fixed and deployed. The 512KB cap was raised to 4MB by #8605 (16dc507a9, 2026-07-24 22:54 PDT = 2026-07-25 05:54 UTC). Every blocked PR above predates that — the newest, #8106, was blocked at 2026-07-25 03:49 UTC, about two hours before the fix landed. Confirmed the running container carries it:

SECRET_SCAN_PATCH_FALLBACK_MAX_CHARS = 4e6

So the detector's alert on 2026-07-27 was summarising a window of pre-fix history. No config change made. Changing the secret_leak rule would have loosened a working credential gate to "fix" a scan-cap problem that no longer exists.

Closing — all three items resolved.