fix(selfhost): close the dead-letter alerting blind spot and require alerting setup

[JSONbored]

Summary

Closes #2533.

• Two of the three dead-letter alert conditions leave a real gap: the rate alert (increase(gittensory_jobs_dead_total[15m]) > 0) only covers a ~15-20 minute window around a job's transition into dead, and the backlog alert only trips above 50. A single dead job — the common case, e.g. one job hitting a since-fixed bug during a deploy — sits invisible in the queue table forever once that window closes, with no automatic way for an operator to find out.
• Alertmanager ships intentionally silent by default (no receiver configured, so docker compose --profile observability up -d always comes up green) — correct, since a working default notification path can't be baked into a committed file without a secret. But the "you must configure this yourself" step was easy to miss.

What changed

• Added GittensoryDeadLetterBacklogPersisting to prometheus/rules/alerts.yml: a level check with no count floor, firing on ANY non-zero dead-letter count sustained for an hour. Complements the existing rate/threshold pair rather than replacing them.
• Strengthened alertmanager/alertmanager.yml's own top-of-file documentation to frame receiver setup as required, not optional, for an unattended deployment, and cross-referenced the fastest verified path (a Discord webhook — already scaffolded as a commented block in the same file) plus the existing Grafana "Active Alerts" panel as a pull-based fallback visible without any extra setup.
• Added a new "Alerting" section to the self-hosting Operations doc making the same required-step framing explicit where an operator actually reads onboarding docs.

Validation

• promtool check rules (prom/prometheus:v3.12.0 image, matching the shipped compose version): 21 rules found, up from 20.
• amtool check-config (prom/alertmanager:v0.33.0 image, matching the shipped compose version): SUCCESS.
• Full local gate: npm run test:ci green, npm audit --audit-level=moderate clean, git diff --check clean.
• No src/** files touched, so no Codecov patch-coverage obligation for this change (config + docs only).

Safety

• No secrets or credentials introduced or exposed.
• No behavior change to existing alert thresholds — purely additive.

[loopover-orb]

Warning

🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨

⏸️ Gittensory review result - manual review recommended

_{Review updated: 2026-07-02 09:06:34 UTC}

3 files · 1 AI reviewer · 1 blocker · readiness 80/100 · CI green · blocked

⏸️ Suggested Action - Manual Review

• Maintainer test expectations unmet — Add or update tests, or attach passing validation output that satisfies the maintainer's test expectations.

Review summary
This change closes a real observability gap by adding a sustained level alert for any non-empty dead-letter queue, complementing the existing transition-rate and high-backlog alerts. The Prometheus rule is syntactically consistent with the surrounding alert file and uses an exported gauge already present in the self-host queue path. The docs and Alertmanager comments correctly make the silent default explicit, but there are a couple of wording/consistency cleanups worth tightening.

Nits — 6 non-blocking

• nit: prometheus/rules/alerts.yml:99 says the same job(s) have sat for over an hour, but the expression only proves the queue count stayed non-zero for an hour, not that the identical records persisted the whole time.
• nit: alertmanager/alertmanager.yml:13 calls it the Grafana "Active Alerts" panel while apps/gittensory-ui/src/routes/docs.self-hosting-operations.tsx:121 calls it the "Alerts" row, so use one operator-facing name consistently.
• prometheus/rules/alerts.yml:99: change the description to "The dead-letter queue has been non-empty for over an hour" so the alert matches what `gittensory_queue_dead > 0 for: 1h` actually proves.
• alertmanager/alertmanager.yml:13 and apps/gittensory-ui/src/routes/docs.self-hosting-operations.tsx:121: align the Grafana panel/row name to the exact dashboard label operators will see.
• PR author also opened the linked issue — Link an issue that was opened by a different contributor, or provide a rationale for why this self-authored issue represents genuine discovery work.
• Code changes lack test evidence — Add focused regression tests or explain why existing coverage is sufficient.

Concerns raised — review before merging

• Maintainer test expectations unmet — Add or update tests, or attach passing validation output that satisfies the maintainer's test expectations.

Signal	Result	Evidence
Code review	❌ 1 blocker	1 reviewer
Linked issue	✅ Linked	#2533
Related work	⚠️ 3 scoped overlaps	Top overlaps are listed below; lower-confidence bulk is hidden.
Change scope	❌ 8/20	High review scope from cached public metadata (size label size:S; 1 linked issue).
Validation posture	✅ 25/25	PR body includes validation/test evidence.
Contributor workload	✅ 10/10	Author activity: 65 registered-repo PR(s), 55 merged, 548 issue(s).
Contributor context	✅ Confirmed Gittensor contributor	JSONbored; Gittensor profile; 65 PR(s), 548 issue(s).
Gate result	❌ Blocking	Repo-configured hard blocker found.

Review context

• Author: JSONbored
• Role context: owner (maintainer lane)
• Public audience mode: oss maintainer
• Lane context: Repository registration is not available in the local Gittensory cache.
• Public profile languages: not available
• Official Gittensor activity: 65 PR(s), 548 issue(s).
• Related work: Items reference the same linked issue fix(selfhost): wire alerting to reach an operator by default + close dead-letter blind spot #2533. (issue #2533)
• Related work: Open PR work references issue fix(selfhost): wire alerting to reach an operator by default + close dead-letter blind spot #2533. (issue #2533)
• Related work: Items reference the same linked issue fix(selfhost): add a healthcheck to the ollama service #2504. (issue #2504, PR #2548)
• Additional title-only matches omitted; title-only overlap does not block.

Contributor next steps

• Treat this as maintainer-lane context rather than normal contributor-lane activity.
• Review top overlaps.
• Add a concise scope and risk note.
• No action.
• Check active issues and PRs before submitting.

Signal definitions

• Related work = same linked issue, overlapping active PRs, or title/path similarity.
• Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
• Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
• Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
• Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

• Re-run Gittensory review

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	gittensory-ui	1700f03	Commit Preview URL Branch Preview URL	Jul 02 2026, 09:04 AM