[agent-job-health] Agent Job Health Monitor — 2026-08-12: 19.2% run-weighted / 0% median-workflow failure rate

[github-actions[bot]]

Summary

Window: 2026-08-11 23:00 UTC to 2026-08-12 23:15 UTC (last 24h ending at workflow start)
Total agent-job runs: 344 across ~164 distinct workflows (engine mix: copilot 162, pi 77, claude 52, codex 20, aider 1, crush 2, goose 2, opencode 2)
Agent-job failures: 66 (52 driver_exit engine-process crashes, 14 agent_logic failures) - excludes 1 non-agentic CI job (Super Linter Report) and 2 intentionally-failing smoke-test runs, out of scope for this monitor.

Metric	Value
Run-weighted failure rate	19.2% (66 / 344)
Workflow-weighted median rate	0% - over half of all distinct workflows had zero failures
Workflow-weighted mean rate	~23% (approx.)
Tracked (existing open issue)	54 / 66 (82%)
Novel (no matching open issue found)	12 / 66 (18%)
Trend vs prior day	No prior data - this is the first recorded baseline for this monitor

Reading the two rates together: the 19.2% run-weighted number is not representative of typical fleet health. It is pulled up by (a) a cluster of high-frequency PR-gate reviewers with real-but-moderate per-workflow failure rates (9-33%), and (b) roughly three dozen low-frequency Daily-X baseline/experimental workflows that each ran once in the window and happened to fail, contributing 100%% each to the mean but with negligible run volume. The median workflow failure rate of 0%% is the more trustworthy fleet-health signal: most of the ~164 active workflows had a clean day.

Failure Rate by Step

Per-CI-step names required the audit MCP tool, which was unavailable in this run (see limitation note below), so the breakdown uses the coarser failure_kind classification captured for every run:

Failure kind	Count	Share of failures	Notes
driver_exit (engine/CLI process crash or unexpected exit, during the Run agent step)	52	79%%	Dominant failure mode; includes the time-correlated cluster below
agent_logic (agent completed but logic/tool/validation failure)	14	21%%	Spread across Issue Monster, Code Scanning Fixer, and several one-off workflows

The agenticworkflows audit tool was not reachable this run, so exact failing-step names and stack excerpts could not be pulled for individual runs; the classification above is the finest grain available from the logs tool summary data.

Tracked Failures

54 of 66 failures already have an open tracking issue. Highlights:

Workflow(s)	Failures this window	Tracking issue(s)
PR Sous Chef	4	#52333, #52345, #52353, #52359 (+ recurring #52183)
Test Quality Sentinel	5	#52364, #52365, #52367, #52372
Matt Pocock Skills Reviewer	3	#52363, #52370, #52374
PR Code Quality Reviewer	2	#52362, #52366, #52373
Ponytail Reviewer	6	#52361, #52369
Impeccable Skills Reviewer	3	#52360, #52368
Design Decision Gate	3	#52371
Issue Monster	1	#52379 (0 failures for Contribution Check this window - 6/6 runs succeeded)
AI Moderator	1	#52354, #52385
Avenger	1	#51984 (P0 Bun runtime segfault crashing Claude Code CLI - open since 08-11)
Agent Performance Analyzer - Meta-Orchestrator	1	#52296 (root cause #51789, Copilot CLI harness segfault, open since 08-10)
Daily Max Ai Credits Test	1	#52278 (root cause #51589, Copilot billing 403, 5+ consecutive daily failures)
Step Name Alignment	1	root cause #51547 (P0 sandbox blocks its own input file - 100%% failure, open since 08-09)
Daily Ambient Context Optimizer, Daily Regulatory Report Generator, Daily Caveman Optimizer, Code Scanning Fixer, aw Failure Investigator (6h), Daily Code Debt Cleanup - Aider, Daily Code Metrics and Trend Tracking Agent, Linter Miner, GEO Optimizer Daily Audit, Daily CLI Performance Agent, Daily Agent of the Day Blog Writer, Daily Fact, CI Optimization Coach, CLI Consistency Checker, Blog Auditor, Daily Choice Type Test, GPL Dependency Cleaner (gpclean)	1 each	Individually auto-filed same day (#52221 to #52358 range)

No open issue search matched a Copilot CLI segfault report beyond #51789/#51984 (both still open, both implicated again today), and PR Sous Chef / Issue Monster remain chronic as flagged - both have multiple open issues spanning several days.

Novel Failure Clusters

Cluster A - Time-correlated PR-gate driver_exit spike, 21:30-21:54 UTC (already tracked, root cause still open)
6 distinct PR-review-gate workflows (Design Decision Gate, Ponytail Reviewer, Matt Pocock Skills Reviewer, Test Quality Sentinel, Impeccable Skills Reviewer, PR Code Quality Reviewer) failed with driver_exit within the same ~24-minute window, across 3 different engines (claude, copilot, pi) and different, unrelated PRs (#52325, #52210, and others) - ruling out a single-PR-content trigger in favor of a shared infrastructure or engine-provider blip. This is already captured by rollup issue #52079 "[aw] Failure cascade detected", which lists 10 of the individually-filed issues but has not yet identified a root cause. Blast radius: engine-wide, 6+ workflows - would have met the issue-creation bar, but is already tracked, so no new issue was filed.

Cluster B - Untracked one-off Daily workflow failures (12 failures, no matching open issue)
12 distinct low-frequency workflows each ran exactly once in the window and failed, with no corresponding open issue found: arXiv Paper Researcher: GitHub Agentic Workflows, Daily Elixir Credo Snippet Audit, Daily AgentRx Trace Optimizer, Daily AstroStyleLite Markdown Spellcheck, Daily Compiler Quality Check, Daily action/setup/* Security Audit, Go Logger Enhancement, Daily Regression Audit, Daily Documentation Healer, Daily AW Cross-Repo Compile Check, daily-experiment-report, Daily Container Image Security Scan.

• Count: 12 runs / 344 (3.5%% of runs), 12/~164 distinct workflows (7%%)
• No shared error signature was confirmed between them (the audit tool needed to pull per-run error text was unavailable this run), so these are reported as individually-novel, low-severity items rather than a single cluster with one root cause.
• Blast radius: single-workflow each. Below both issue-creation thresholds (needs 2+ workflows sharing a cause, or 10%%+ of runs) individually and collectively, since no shared cause was established - so no issue filed for these either.

Per-Workflow Breakdown

Runs and failures by workflow (partial list - top contributors by run volume)

Workflow	Runs	Failures	Failure rate
PR Sous Chef	46	4	8.7%%
Test Quality Sentinel	19	5	26.3%%
Design Decision Gate	18	3	16.7%%
Ponytail Reviewer	18	6	33.3%%
Impeccable Skills Reviewer	18	3	16.7%%
PR Code Quality Reviewer	18	2	11.1%%
Matt Pocock Skills Reviewer	17	3	17.6%%
AI Moderator	13	1	7.7%%
Contribution Check	6	0	0%%
Auto-Triage Issues	5	0	0%%
Code Scanning Fixer	4	3	75%%
aw Failure Investigator (6h)	4	1	25%%
PR Triage Agent	4	0	0%%
Issue Monster	2	1	50%%
Avenger	2	1	50%%
Daily Credit Limit Test	2	0	0%%
~150 other workflows (mostly 1 run/day)	~165	~34	varies (0%% or 100%% per workflow)

Distinct-workflow total (~164) is an approximation: the 24h dataset spans 250+ uniquely-named workflows with heavy long-tail sparsity (most run once/day), and exact deduplication of the full list was done by manual inspection rather than a database query.

Recommendations

1. Cluster A (cascade, [aw] Failure cascade detected #52079): No new issue needed, but the finding that the 21:30-21:54 UTC spike crossed 3 engines and multiple unrelated PRs (not just one PR's content) is new information not yet in that issue. Worth a comment there redirecting the root-cause search from PR content toward shared infra/provider (e.g. a transient GitHub Actions runner or engine-provider blip in that window).
2. Code Scanning Fixer (75%% failure, 3/4 runs) and Ponytail Reviewer (33%%) are the two established (non-singleton) workflows furthest above the fleet median this window - worth a closer look even though each already has an open tracking issue, since their rates are notably higher than peer PR-gate workflows running the same class of check.
3. Cluster B (12 untracked singleton failures): individually too small to act on, but if the daily-cron baseline workflows keep failing 1-for-1 on their only run each day, consider whether the auto-filer that creates aw-X-failed issues for the high-frequency workflows should also cover these lower-frequency ones - right now they appear to fall through that net.
4. Known chronic offenders remain chronic: PR Sous Chef (4 fresh issues + a recurring report, still ~9%% failure rate) and Issue Monster stay tracked; Contribution Check was healthy this window (0/6 failures) despite its chronic reputation - worth confirming whether its prior fix is holding.

Limitation: the agenticworkflows audit tool was unreachable this run (registration failure, not a timeout), so this report relies on the logs tool run-level summary data rather than per-run stack traces/step names. Distinct-workflow counts and the workflow-weighted mean are therefore best-effort approximations rather than exact.

References:

• §31643436363 - Ponytail Reviewer, driver_exit, part of Cluster A
• §31643652320 - Design Decision Gate, driver_exit, part of Cluster A
• §31606461662 - Super Linter Report, excluded from denominator (non-agentic)

Generated by 🩺 Agent Job Health Monitor · agent · 346.3 AIC · ⌖ 39.5 AIC · ⊞ 6.9K · ◷

• expires on Aug 15, 2026, 3:57 PM UTC-08:00