💡 Health-Check Issue Lifecycle Automation for Alert Noise Reduction

[github-actions[bot]]

Summary

Add lifecycle management to the health-check automation: auto-close resolved health-check issues when the underlying failure no longer reproduces, and aggregate recurring failures into a single rolling tracker issue instead of creating a new issue per day. This reduces the 15+ open daily health-check issues (30% of the issue tracker) that currently obscure real enhancement and bug signals.

Market Signal

• The Arize 2026 AI observability guide identifies "alert fatigue" as a primary failure mode in autonomous monitoring systems — the volume of alerts degrades signal-to-noise to the point where real problems are missed
• Every major monitoring platform (PagerDuty, Datadog, OpsGenie) implements alert lifecycle management: auto-resolve when the condition clears, deduplicate recurring alerts into a single tracker, and escalate only on new or worsening conditions
• The industry pattern is detect → aggregate → resolve, not detect → accumulate → manually close
• Gartner's 40%-cancellation-by-2027 prediction for agentic AI projects cites monitoring operational overhead as a contributing factor — alert noise from the monitoring itself is part of the problem

User Signal

• 15 of 50 open issues (30%) are health-check / automated-report issues — daily workflow failure reports and PR review failure reports that accumulate without resolution
• Multiple daily reports reference the same underlying workflow (e.g., consecutive "workflow failures detected" issues for lint.yml on June 24, 25, 26, 27, 28, 29, 30, July 1, 2, 3)
• The fleet-tracker label shows persistent single-workflow failures generating repeated daily issues ([Fleet Monitor] petry-projects/.github-private — .github/workflows/lint.yml #918 for lint.yml, [Fleet Monitor] petry-projects/.github-private — .github/workflows/ci.yml #832 for ci.yml)
• The signal-to-noise degradation makes it harder to identify genuine bugs (pr-quality ruleset fails to create (HTTP 422): unsupported automatic_copilot_code_review_enabled param #1027, dev-lead emits empty, model-less token records that pollute the "no price" count in the cost report #1009) and enhancements (Populate canary-rings.json with the 6 #482 reusables (registry is dev-lead-only → evaluate-all covers nothing else) #1036, Autonomous fleet-wide soak-and-promote workflow for all first-party reusables #993) in the issue tracker
• Several health-check issues may already be resolved but were never closed (the underlying workflow recovered)

Technical Opportunity

The health-check scripts already generate structured data about which workflows failed:

1. Deduplication: Before creating a new issue, search for an existing open issue with the same failure signature (workflow name + failure type). If found, add a comment to the existing issue with the new failure data instead of creating a duplicate. This turns 10 daily issues for the same lint.yml failure into 1 issue with 10 comments.

2. Auto-resolution: On each health-check run, check whether previously-reported failures have resolved (the workflow's most recent run succeeded). If so, auto-close the issue with a "✅ Resolved — $WORKFLOW_NAME has been passing since $TIMESTAMP" comment.

3. Escalation on worsening: If a deduplicated tracker's failure count exceeds a threshold (e.g., 7 consecutive days), add a label (needs-human-review) to escalate persistent failures that auto-remediation hasn't fixed.

The FLEET_GATE_WORKFLOWS list in fleet_report.sh already distinguishes intentional gate failures from health failures — lifecycle logic respects this distinction.

Assessment

Dimension	Score	Rationale
Feasibility	high	Structured data exists. Dedup = search + comment. Auto-close = query recent runs + close. Both use standard gh CLI.
Impact	med	Reduces issue tracker noise by ~30%. Makes real issues more visible. Reduces manual triage burden.
Urgency	med	The noise level is already high (15+ open health issues) and grows daily. Not blocking, but worsening over time.

Adversarial Review

Strongest objection: Daily health-check issues serve as an audit trail. Auto-closing loses that history, and deduplication makes it harder to see the timeline of a recurring failure.

Rebuttal: Auto-closed issues are not deleted — they remain searchable and preserve the full timeline. The aggregated tracker issue provides a better audit trail: it shows the full lifecycle (first detected → ongoing → resolved) in a single thread instead of scattered across 10+ separate issues. And the signal-to-noise improvement (reducing 15+ noise issues to a handful of active trackers) makes real issues more visible — which is the primary goal of the issue tracker.

Suggested Next Step

Add a dedup-and-lifecycle step to the health-check issue creation scripts:

1. Before creating a new issue, search for an existing open issue with the same failure signature (workflow name + failure type) — if found, comment instead of creating new
2. Add a resolution-check that queries recent workflow runs and auto-closes if the failure has cleared
3. Gate behind DRY_RUN for initial testing to verify the dedup and auto-close logic before fleet-wide deployment