feat(observability): surface ops-anomaly signals as metrics/alerts/panels#4043
Conversation
…+ panel runOpsAlerts (src/review/ops-wire.ts) already scanned gittensory's own outcome data hourly and emitted a structured ops_anomaly log on a review burst or review failure burst (the #3747 incident shape: a stuck-CI finalize loop or retry storm). Nothing surfaced this on /metrics, so an operator had to grep Workers Logs instead of alerting. Adds gittensory_ops_anomaly_total (counter, labeled by kind + repo, repo redacted on the cloud worker same as gittensory_gate_decisions_total), a GittensoryOpsAnomalyDetected Prometheus alert, and a Grafana panel pair on the self-host observability stack.
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
|
Warning 🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨 ⏸️ Gittensory review result - manual review recommendedReview updated: 2026-07-07 13:11:59 UTC
⏸️ Suggested Action - Manual Review
Review summary Nits — 1 non-blocking
Review context
Contributor next steps
Signal definitions
🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed 💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →. Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4043 +/- ##
=======================================
Coverage 93.58% 93.58%
=======================================
Files 382 382
Lines 35534 35538 +4
Branches 13028 13030 +2
=======================================
+ Hits 33253 33257 +4
Misses 1618 1618
Partials 663 663
🚀 New features to boost your workflow:
|
Summary
Closes #4007.
runOpsAlerts(src/review/ops-wire.ts) already scans gittensory's own outcome data hourly and emits a structuredops_anomalylog on a review burst or review failure burst (the fix(review): stop the medium-effort retry storm burning tokens on zero output #3747 incident shape: a stuck-CI finalize loop or retry storm). Nothing surfaced this on/metrics, so an operator had to grep Workers Logs instead of alerting on it.gittensory_ops_anomaly_total(counter, labeled bykind+repo; repo label redacted on the cloud worker the same waygittensory_gate_decisions_total/gittensory_reviews_published_totalalready are).GittensoryOpsAnomalyDetectedPrometheus alert (prometheus/rules/alerts.yml), matching the existingGittensoryDeadLetterJobsGrowingabsolute-increase style.Test plan
npm run typecheck— cleannpx vitest run test/unit/ops-wire.test.ts test/unit/selfhost-metrics.test.ts— 65/65 passed, including new tests for the counter incrementing on both anomaly kinds and the repo-label redaction behaviornpm run selfhost:validate-observability— dashboards and alert rules valid