fix(observability): correct GittensoryHighJobFailureRatio alert formula

[JSONbored]

Summary

• GittensoryHighJobFailureRatio divided failed-job rate by gittensory_jobs_processed_total alone. That counter only increments on success, so the expr computed failed:success instead of a true failure percentage — a genuine 50% failure rate read as 100%, and the alert both fired far earlier than its stated 10% threshold and misreported the value in its page text.
• Fixed the denominator to sum both counters (failed + processed), matching the Grafana "Job Failure Rate" panel's existing formula for the equivalent _persisted_total metrics.
• Added a regression test that pins the corrected PromQL shape by parsing the real prometheus/rules/alerts.yml.

Found via a fresh performance/scalability/accuracy hardening audit of the self-host ORB stack. Tracked under #1667.

Scope

• prometheus/rules/alerts.yml — corrected expr + updated comment/description text
• test/unit/alerts-job-failure-ratio-formula.test.ts — new regression test

Validation

• npm run typecheck
• npm run selfhost:validate-observability (dashboards + alert rules valid)
• npx vitest run test/unit/alerts-job-failure-ratio-formula.test.ts test/unit/validate-observability-configs-script.test.ts test/unit/docs-selfhost-troubleshooting-metric-names.test.ts test/unit/selfhost-grafana-dashboard.test.ts — all green
• git diff --check clean

Safety

• Config/test-only change; no src/** lines touched, no secrets, no behavior change beyond the alert's own threshold math.

Closes #3892

[superagent-security]

Superagent didn't find any vulnerabilities or security issues in this PR.

[loopover-orb]

Tip

🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩

✅ Gittensory review result - approve/merge recommended

_{Review updated: 2026-07-07 06:16:04 UTC}

2 files · 1 AI reviewer · no blockers · readiness 100/100 · CI green · clean

✅ Suggested Action - Approve/Merge

• safe to merge

Review summary
This fixes a genuine math bug in the GittensoryHighJobFailureRatio alert: since gittensory_jobs_processed_total only increments on success, dividing failed-rate by processed-rate alone computed failed:success instead of failed:total, so a 50% failure rate could read as 100% and the alert fired earlier than its documented 10% threshold. The corrected expr sums failed+processed in the denominator, which now correctly bounds the ratio in [0,1] and matches the Grafana panel's existing formula. The new regression test parses the real alerts.yml and pins the exact corrected PromQL shape via regex, and both assertions match the actual diff content, so the test is real (not fabricated) coverage of this fix. Change is config/test-only, closes the linked issue #3892, and CI is green.

Nits — 5 non-blocking

• prometheus/rules/alerts.yml: the outer `(ratio > 0) > 0.10` boolean-as-number pattern is preexisting and a bit hard to read at a glance; a follow-up could restructure as two clearer conditions, though not required here since the pattern itself wasn't changed.
• test/unit/alerts-job-failure-ratio-formula.test.ts: the whitespace-flattening regex match is a bit brittle to incidental formatting changes in the YAML; a PromQL-aware comparison (e.g. normalizing and comparing token lists) would be more robust, though acceptable given the existing lightweight-validation convention in this repo (scripts/validate-observability-configs.mjs).
• Consider also asserting in the test that the description/summary annotation text was updated to say "attempted" rather than "processed", to guard against the wording regressing back to the old (misleading) phrasing.
• If not already covered elsewhere, add a similar regression test for the equivalent `_persisted_total`-based Grafana panel formula mentioned in the description, to prevent the same processed-only-on-success pitfall recurring there.
• PR author also opened the linked issue — Link an issue that was opened by a different contributor, or provide a rationale for why this self-authored issue represents genuine discovery work.

Signal	Result	Evidence
Code review	✅ No blockers	1 reviewer
Linked issue	✅ Linked	#3892
Related work	✅ No active overlap found	No same-issue or scoped active PR overlap found.
Change scope	✅ 20/20	Low review scope from cached public metadata (1 linked issue).
Validation posture	✅ 25/25	PR body includes validation/test evidence.
Contributor workload	✅ 10/10	Author activity: 51 registered-repo PR(s), 43 merged, 343 issue(s).
Contributor context	✅ Confirmed Gittensor contributor	JSONbored; Gittensor profile; 51 PR(s), 343 issue(s).
Gate result	✅ Passing	No configured blocker found.

Review context

• Author: JSONbored
• Role context: owner (maintainer lane)
• Public audience mode: oss maintainer
• Lane context: Repository is configured for direct PR review.
• Public profile languages: not available
• Official Gittensor activity: 51 PR(s), 343 issue(s).
• PR-specific overlap: none found.

Contributor next steps

• Treat this as maintainer-lane context rather than normal contributor-lane activity.

Signal definitions

• Related work = same linked issue, overlapping active PRs, or title/path similarity.
• Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
• Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
• Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
• Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

• Re-run Gittensory review

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.57%. Comparing base (a8a2287) to head (3fa72c8).
⚠️ Report is 3 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3908   +/-   ##
=======================================
  Coverage   93.57%   93.57%           
=======================================
  Files         357      357           
  Lines       34309    34309           
  Branches    12557    12557           
=======================================
  Hits        32105    32105           
  Misses       1580     1580           
  Partials      624      624           

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[loopover-orb]

Gittensory approves — the gate is satisfied and CI is green.