[agent-job-health] Agent Job Health Monitor — 2026-10-19: fleet rate 14.4% (run-weighted), no novel action-worthy clusters

[github-actions[bot]]

Summary

• Window evaluated: 2026-10-18T23:20Z → 2026-10-19T23:21Z (UTC)
• total_runs: 201 | distinct_workflows: 63 | agent_failures: 29
• Run-weighted fleet failure rate: 29/201 = 14.43%
• Workflow-weighted: mean 20.65%, median 0.0% (19 of 63 workflows had any agent-job failure; 44 had none)
• Tracked vs. novel: 0 confirmed-novel, action-worthy clusters (see caveat below). 15 of 29 failing runs match chronic, previously-tracked signatures; 5 are explained by the system's own obsolete_pr_runs classification (not a bug); 6 are isolated single-workflow incidents below the issue-creation bar; 1 (ai-moderator) is excluded entirely (confirmed no jobs ever ran, status unknown, not counted as an agent-job failure).
• Trend: up notably vs. 2026-10-18 (4.23-6.5% range, ~5% point estimate) and 2026-10-17 (5.27%). The jump looks large on the surface, but it is driven entirely by clusters that are either already chronic/tracked or explained by a known non-bug classification (see Novel Failure Clusters), not a new fleet-wide regression.

Warning

Run-weighted rate roughly tripled vs. the last two recorded days. Flagged as WARNING per the day-over-day threshold, even though root-cause analysis below does not point to a new, unaddressed bug — see Recommendations.

Failure Rate by Step

Failing step (agent job)	Runs	Share of agent_failures
Execute GitHub Copilot CLI (driver_exit)	12	41.4%
Checkout PR branch (driver_exit)	5	17.2%
(no distinct step signature — generic "Handle agent failure" only)	6	20.7%
Execute Pi CLI	2	6.9%
Install Pi CLI	1	3.4%
Execute Claude Code CLI (cascaded into "Parse agent logs for step summary" in the same run)	1	3.4%
Warm up model	1	3.4%
Assert sub-agent model and wire API evidence	1	3.4%

Note: one daily-security-observability run (39281292500) failed at "Execute GitHub Copilot CLI" and then also failed its "Redact secrets in logs" cleanup step as a cascading consequence — counted once under the primary step above, not twice.

Tracked Failures

Cluster	Failing runs	Distinct workflows	Prior tracking (per project trend history — not verified live this run, see caveat)
Execute GitHub Copilot CLI driver_exit	12	4 (daily-security-observability, pr-code-quality-reviewer, contribution-check, copilot-opt)	Chronic signature appearing in nearly every prior daily report since 2026-08-23; previously associated with #54186 / #55413 / #56694
Execute/Install Pi CLI failure	3	2 (issue-monster, smoke-pi-routed)	Chronic Pi CLI flakiness, previously associated with #65909

Caveat: GitHub issue search (search_issues, list_issues without a narrow label filter) was blocked by an integrity policy in this sandbox for every query this run — consistent with nearly every prior entry in this workflow's own trend history (cache-memory agent-job-health/history.jsonl, 2026-10-12 through 2026-10-18). list_pull_requests and label-scoped list_issues calls worked normally, so this is scoped to general issue-content reads, not a full outage. As a result, the issue numbers above are carried from project history, not re-confirmed live — treat them as "likely already tracked" rather than verified.

Novel Failure Clusters

None met the bar for a new tracking issue. For completeness, here is every cluster that is not a confirmed chronic/tracked signature:

1. "Checkout PR branch" failing simultaneously across 5 reviewer workflows (design-decision-gate, mattpocock-skills-reviewer, ponytail-reviewer, pr-code-quality-reviewer, test-quality-sentinel; engines codex/copilot/pi) — all 5 runs fired at the exact same second (2026-10-19T17:08:13Z) against the same branch, chore/remove-dead-functions-20261019-99aeb36d1a1341a6 (an ephemeral bot-created PR branch). Representative run: https://github.com/github/gh-aw/actions/runs/39291542240 . Checkout failed in ~1 second — consistent with the PR/branch having been closed, superseded, or force-pushed by the time the reviewer fleet's checkout step ran. This matches the audit system's own obsolete_pr_runs classification (10 total in the window) — it is a known, named failure mode of PR-triggered fan-out reviewers racing ephemeral bot branches, not a new infrastructure bug. Blast radius: PR-review fleet only, not engine- or fleet-wide. No issue recommended — recommend instead (see below) that the fleet-rate calculation exclude obsolete_pr_runs from the denominator/numerator to stop them inflating the agent-failure rate.
2. Six isolated single-run, single-workflow failures with no distinct step signature beyond the generic "Handle agent failure" conclusion-job signal: Daily Project Performance Summary Generator (pi), Smoke Temporary ID (pi), Daily Spec Coverage Review (copilot), Smoke Pi (pi), Smoke Copilot (copilot), Daily Go Test Stubs — Aider (aider). Each affects only one workflow with one occurrence — below the 2-workflow / 10%-of-runs bar for action. No pattern connects them (different engines, different workflows, no shared timestamp). No issue recommended.
3. ai-moderator run https://github.com/github/gh-aw/actions/runs/39313096545 produced zero GitHub Actions jobs (confirmed_no_jobs) — status genuinely unknown (could be a cancelled/superseded dispatch, not necessarily an agent-job failure). Excluded from agent_failures. Flagging for awareness only.

Schedule Heartbeat

Checked all 123 schedule-triggered workflows (daily or sub-daily cadence) that did not appear in this window's 201-run sample, via the GitHub Actions API (most-recent-run-of-any-kind + get_workflow state). No novel blind spots. 3 large gaps found, all fully explained by a disabled state (consistent with every prior entry in project history):

Workflow	.lock.yml	Last observed run	Expected cadence	Gap	State
daily-geo-optimizer	.github/workflows/daily-geo-optimizer.lock.yml	2026-08-18T16:31:58Z	daily	62.3d	disabled_manually
daily-hippo-learn	.github/workflows/daily-hippo-learn.lock.yml	2026-07-27T07:24:10Z	daily around 7:00	84.7d	disabled_manually
slide-deck-maintainer	.github/workflows/slide-deck-maintainer.lock.yml	2026-09-23T16:24:44Z	daily around 16:00 on weekdays	26.3d	disabled_manually

All other 120 checked workflows (including weekly ones like chaos-pr-bundle-fuzzer, 4.9d gap vs a 15d threshold) last ran within their expected cadence. None had zero runs ever.

View per-workflow breakdown (workflows with at least 1 agent-job failure this window)

Workflow	Runs	Agent failures	Failure rate
PR Code Quality Reviewer	15	9	60.0%
Contribution Check	2	2	100%
Issue Monster	14	2	14.3%
Copilot Opt	1	1	100%
Daily BYOK Ollama Test	1	1	100%
Daily Go Test Stubs — Aider	1	1	100%
Daily Project Performance Summary Generator (Using MCP Scripts)	1	1	100%
Daily Security Observability Report	1	1	100%
Daily Spec Coverage Review	1	1	100%
Design Decision Gate 🏗️	14	1	7.1%
Matt Pocock Skills Reviewer	16	1	6.3%
Ponytail Reviewer	15	1	6.7%
Smoke Copilot	2	1	50.0%
Smoke Copilot Sub Agents	1	1	100%
Smoke Pi	1	1	100%
Smoke Pi Routed	2	1	50.0%
Smoke Temporary ID	1	1	100%
Test Quality Sentinel	15	1	6.7%
Workflow Portfolio Analyst	1	1	100%
(44 other workflows ran with 0 agent-job failures)	—	0	0%

Recommendations

1. No new issue opened this run. Every failing cluster is either already chronic/tracked per project history (Copilot CLI driver-exit, Pi CLI), explained by an existing non-bug classification (obsolete-PR checkout races), or an isolated single-occurrence incident below the action bar.
2. Consider excluding obsolete_pr_runs from the agent-job denominator/numerator in this report's own fleet-rate math — 5 of today's 29 "failures" (17%) are PR-reviewer-fleet runs racing a since-superseded bot branch, which inflates the headline rate without reflecting a real reliability problem. This would have dropped today's run-weighted rate to roughly 24/196 = 12.2%.
3. PR Code Quality Reviewer (60% failure rate, 9/15 runs) is the single largest contributor to today's elevated rate, almost entirely via the chronic Copilot CLI driver-exit pattern. If [aw-failures] [P0] Cross-engine agent-CLI segfault (exit 139) — confirmed on Auto-Triage/Pi CLI, now spread to 8 workflows acros #54186/[aw-failures] [aw] Copilot-BYOK auth rejected by api-proxy — Codex 404 (Ponytail Reviewer, Issue Monster) + Claude 401 (Smoke Claude on Copilo #55413/[aw] Avenger failed #56694 are indeed still open and unresolved, this workflow is the highest-value target for a fix (or for forcing a review of whether auto model routing on copilot needs a more defensive retry around that CLI's exit behavior).
4. Re-enable or formally retire daily-geo-optimizer, daily-hippo-learn, and slide-deck-maintainer — all three have been sitting disabled_manually for 26-85 days per this and every recent report. If they are intentionally retired, consider removing their schedule trigger (or the workflow file) so they stop appearing in this audit's candidate list every day.
5. Restore GitHub issue-search access for this workflow's identity. The integrity-policy block on search_issues/unfiltered list_issues has now recurred in essentially every run since 2026-10-12, making Tracked vs. Novel classification structurally unverifiable without relying on this workflow's own memory file. Since label-scoped list_issues calls (e.g. labels:[bug]) partially succeed, consider either (a) granting this workflow's identity read access to issue content at "approved" integrity, or (b) standardizing on a label (e.g. automated) for all bot-filed tracking issues so future runs can reliably query by label instead of free-text/semantic search.

Generated by 🩺 Agent Job Health Monitor · claude · agent · 643.6 AIC · ⌖ 0.899 AIC · ⊞ 6.9K · ◷

• expires on Oct 22, 2026, 3:46 PM UTC-08:00