Skip to content

feat(observability): surface ops-anomaly signals as metrics/alerts/panels#4043

Merged
JSONbored merged 1 commit into
mainfrom
feat/ops-anomaly-metric-and-alert-v2
Jul 7, 2026
Merged

feat(observability): surface ops-anomaly signals as metrics/alerts/panels#4043
JSONbored merged 1 commit into
mainfrom
feat/ops-anomaly-metric-and-alert-v2

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

Closes #4007.

  • runOpsAlerts (src/review/ops-wire.ts) already scans gittensory's own outcome data hourly and emits a structured ops_anomaly log on a review burst or review failure burst (the fix(review): stop the medium-effort retry storm burning tokens on zero output #3747 incident shape: a stuck-CI finalize loop or retry storm). Nothing surfaced this on /metrics, so an operator had to grep Workers Logs instead of alerting on it.
  • Adds gittensory_ops_anomaly_total (counter, labeled by kind + repo; repo label redacted on the cloud worker the same way gittensory_gate_decisions_total/gittensory_reviews_published_total already are).
  • Adds a GittensoryOpsAnomalyDetected Prometheus alert (prometheus/rules/alerts.yml), matching the existing GittensoryDeadLetterJobsGrowing absolute-increase style.
  • Adds a Grafana panel pair to the self-host observability dashboard.

Test plan

  • npm run typecheck — clean
  • npx vitest run test/unit/ops-wire.test.ts test/unit/selfhost-metrics.test.ts — 65/65 passed, including new tests for the counter incrementing on both anomaly kinds and the repo-label redaction behavior
  • npm run selfhost:validate-observability — dashboards and alert rules valid

…+ panel

runOpsAlerts (src/review/ops-wire.ts) already scanned gittensory's own outcome data
hourly and emitted a structured ops_anomaly log on a review burst or review failure
burst (the #3747 incident shape: a stuck-CI finalize loop or retry storm). Nothing
surfaced this on /metrics, so an operator had to grep Workers Logs instead of alerting.

Adds gittensory_ops_anomaly_total (counter, labeled by kind + repo, repo redacted on
the cloud worker same as gittensory_gate_decisions_total), a GittensoryOpsAnomalyDetected
Prometheus alert, and a Grafana panel pair on the self-host observability stack.
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 7, 2026
@loopover-orb

loopover-orb Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Warning

🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨

⏸️ Gittensory review result - manual review recommended

Review updated: 2026-07-07 13:11:59 UTC

6 files · no blockers · readiness 100/100 · CI green · clean

⏸️ Suggested Action - Manual Review

  • AI review already in progress for this PR head: Another Gittensory pass is already running the AI review for this exact PR head. This pass is skipping to avoid a duplicate LLM call.

Review summary
AI review is already running for this PR head in another Gittensory pass. Gittensory is holding this PR for manual review until that pass completes.

Nits — 1 non-blocking
  • AI review already in progress for this PR head — The gate is held for a human reviewer rather than passed automatically; it re-evaluates once the in-flight review completes or on the next update.
Signal Result Evidence
Code review ✅ No blockers No AI review summary
Linked issue ✅ Linked #4007
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (1 linked issue).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 51 registered-repo PR(s), 43 merged, 570 issue(s).
Contributor context ✅ Confirmed Gittensor contributor JSONbored; Gittensor profile; 51 PR(s), 570 issue(s).
Gate result ⚠️ Not blocking Advisory; not blocking this PR.
Review context
  • Author: JSONbored
  • Role context: owner (maintainer lane)
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: Python, TypeScript, JavaScript, Ruby, Go, Kotlin, MDX, Shell
  • Official Gittensor activity: 51 PR(s), 570 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Treat this as maintainer-lane context rather than normal contributor-lane activity.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

  • Re-run Gittensory review

@codecov

codecov Bot commented Jul 7, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.58%. Comparing base (c899c3c) to head (99f5911).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4043   +/-   ##
=======================================
  Coverage   93.58%   93.58%           
=======================================
  Files         382      382           
  Lines       35534    35538    +4     
  Branches    13028    13030    +2     
=======================================
+ Hits        33253    33257    +4     
  Misses       1618     1618           
  Partials      663      663           
Files with missing lines Coverage Δ
src/review/ops-wire.ts 96.49% <100.00%> (+0.26%) ⬆️
src/selfhost/metrics.ts 100.00% <ø> (ø)
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@loopover-orb loopover-orb Bot added the manual-review Gittensor contributor context label Jul 7, 2026
@JSONbored
JSONbored merged commit 01ccdd0 into main Jul 7, 2026
11 checks passed
@JSONbored
JSONbored deleted the feat/ops-anomaly-metric-and-alert-v2 branch July 7, 2026 13:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. manual-review Gittensor contributor context

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(observability): surface ops-anomaly signals as metrics/alerts/panels

1 participant