[agent-job-health] Agent Job Health Monitor - 2026-08-23

[github-actions[bot]]

Summary

Over the 24 hours ending 2026-08-23 23:00 UTC, the fleet ran 358 runs across 111 distinct workflows. The agent job itself (not activation/detection/safe-outputs) genuinely failed in 33 runs (one additional run had a skipped agent job and was excluded as never-executed).

• Run-weighted fleet failure rate: 33 / 358 = 9.2%
• Workflow-weighted median rate: 0% (85 of 111 workflows had zero agent failures)
• Workflow-weighted mean rate: 16.8% (skewed up by ~17 workflows that ran once and failed once)

Note

This is the first recorded data point in the trend history (/tmp/gh-aw/cache-memory/agent-job-health/history.jsonl was empty before this run) — no day-over-day comparison is possible yet. All three failure clusters below map cleanly onto already-open P0 issues, and no untracked cluster meets the bar for a new issue. This looks like the residue of 3 known, already-diagnosed engine outages spread across a wide set of workflows — not a novel fleet-wide regression.

Failure Rate by Step

Failing step (normalized signature)	Runs	Distinct workflows	Engine
Execute GitHub Copilot CLI	15	12	Copilot
Execute Codex CLI	9	8	Codex
Execute Claude Code CLI	6	3	Claude
Execute Pi CLI	1	1	Pi
Parse agent logs for step summary	1	1	Copilot
Enforce critical vulnerability and license gates	1	1	n/a (custom gate step)

The last row (Daily Container Image Security Scan) is very likely the workflow doing its job — blocking on a real vulnerability/license finding — not an engine or harness bug. Recommend a human spot-check rather than treating it as fleet health noise.

Tracked Failures

Every one of the 33 failures maps to an already-open issue, either by workflow name or by matching one of three known root-cause clusters:

Cluster / signature	Runs	Tracking issue
Cross-engine CLI crash (Copilot CLI failures, 12 workflows)	15	#54186 — "Cross-engine agent-CLI segfault (exit 139)... spread to 8 workflows"
Codex OpenAI invalid_project 401 auth failure	9	#54242 — "Codex OpenAI invalid_project 401 auth failure — unfixed 3+ days"
Claude Code CLI startup crashes (Anthropic API egress blackout)	5	#54462 — "Claude Code CLI startup crashes — Anthropic API egress blackout"
Design Decision Gate hangs mid-tool-use (Claude CLI)	1	separate [P1] issue specific to Design Decision Gate
Per-workflow individual failures (Code Scanning Fixer, Issue Arborist, Test Quality Sentinel, Linter Miner, Daily Cache Strategy Analyzer, Daily Max Ai Credits Test, Impeccable/Matt Pocock Skills Reviewers, Copilot PR Prompt Pattern Analysis, Dead Code Removal Agent, AI Moderator, Duplicate Code Detector, Daily Team Evolution Insights, [aw] Failure Investigator (6h), Ponytail Reviewer)	remainder	each already has an open [aw] <workflow> failed issue

None of these needed a new issue — all satisfy "open issue already exists" via workflow name and/or error signature.

Novel Failure Clusters

None. Every failing run in this window is accounted for by the tracked clusters above. No new issue was created.

Schedule Heartbeat

Config verified via the status tool; no workflow was found in disabled_manually / disabled_inactivity state (checked via the Actions API workflow list).

Full 192-workflow × multi-day cadence sweep was not exhaustively completed — repeated Actions-API pagination for schedule-run history hit transient network errors partway through a multi-day fetch. Effort was bounded to: (a) confirming no workflow is disabled, and (b) spot-checking all 5 weekly-* cadence workflows plus known chronic offenders (PR Sous Chef, Issue Monster, Contribution Check — all ran normally in-window, not blind spots).

Workflow	Cadence	Last run found	Status
Weekly Editors Health Check	weekly	2026-08-22 (1 day ago)	OK
Weekly Issue Summary	weekly	2026-08-17 (6 days ago, failed)	OK — within 15-day threshold
Weekly Safe Outputs Spec Review	weekly	2026-08-17 (6 days ago)	OK
Weekly Blog Post Writer	weekly	2026-08-17 (6 days ago)	OK
Weekly Network Domains Audit	weekly	no runs found; not present in the registered Actions workflow list at all	⚠️ Blind spot

Warning

weekly-network-domains-audit.md and its compiled .lock.yml both exist in the repo, but the workflow does not appear in the GitHub Actions workflow registry and has zero recorded runs — it appears to have never been activated/dispatched. This isn't a disabled_manually/disabled_inactivity case (those show up in the registry); it looks like the workflow was added but never picked up. Recommend a maintainer check whether it needs a manual trigger or a push to register it.

Per-workflow failure breakdown

25 workflows with at least one agent-job failure (33 failures total)

Workflow	Runs	Failures	Rate
Code Scanning Fixer	3	3	100%
Issue Arborist	3	2	66.7%
Daily Go Test Parallelizer	8	2	25%
Design Decision Gate	22	4	18.2%
[aw] Failure Investigator (6h)	3	1	33.3%
Test Quality Sentinel	23	1	4.3%
Ponytail Reviewer	24	1	4.2%
PR Code Quality Reviewer	24	1	4.2%
Matt Pocock Skills Reviewer	24	1	4.2%
Impeccable Skills Reviewer	24	1	4.2%
Smoke Codex	1	1	100%
Linter Miner	1	1	100%
Duplicate Code Detector	1	1	100%
Dev	1	1	100%
Dead Code Removal Agent	1	1	100%
Daily Team Evolution Insights	1	1	100%
Daily Syntax Error Quality Check	1	1	100%
Daily Max Ai Credits Test	1	1	100%
Daily Evals Feature Report	1	1	100%
Daily Documentation Diagram	1	1	100%
Daily Container Image Security Scan	1	1	100%
Daily Cache Strategy Analyzer	1	1	100%
Copilot PR Prompt Pattern Analysis	1	1	100%
Avenger	1	1	100%
Artifacts Usage Report	1	1	100%
AI Moderator	1	1	100%

Note: most 100%-rate rows are single-run daily workflows that happened to run once in this window and hit one of the three tracked engine outages above — not independent chronic failures.

Recommendations

1. No new issue needed this cycle — the three tracked root causes ([aw-failures] [P0] Cross-engine agent-CLI segfault (exit 139) — confirmed on Auto-Triage/Pi CLI, now spread to 8 workflows acros #54186 Copilot/cross-engine segfault, [aw-failures] [P0] Codex OpenAI invalid_project 401 auth failure — unfixed 3+ days, Daily Cache Strategy Analyzer failing every #54242 Codex 401 auth, [aw-failures] [P0] Claude Code CLI startup crashes — Anthropic API egress blackout #54462 Claude/Anthropic egress) already explain 30 of 33 failures across 23 workflows. Prioritizing a fix for any one of them will visibly move the fleet rate.
2. Follow up on weekly-network-domains-audit — confirm whether it needs manual registration/dispatch.
3. Treat the Daily Container Image Security Scan "failure" as a security finding requiring triage, not a monitor bug.
4. Next run of this monitor will have a real day-over-day comparison point now that history.jsonl has its first entry.

References:

• §32624719359 — Code Scanning Fixer, Copilot CLI failure (cluster [aw-failures] [P0] Cross-engine agent-CLI segfault (exit 139) — confirmed on Auto-Triage/Pi CLI, now spread to 8 workflows acros #54186)
• §32658283193 — Daily Cache Strategy Analyzer, Codex CLI failure (cluster [aw-failures] [P0] Codex OpenAI invalid_project 401 auth failure — unfixed 3+ days, Daily Cache Strategy Analyzer failing every #54242)
• §32643440704 — Design Decision Gate, Claude CLI failure (cluster [aw-failures] [P0] Claude Code CLI startup crashes — Anthropic API egress blackout #54462)

Generated by 🩺 Agent Job Health Monitor · agent · 383.9 AIC · ⌖ 8.16 AIC · ⊞ 9K · ◷

• expires on Aug 26, 2026, 3:48 PM UTC-08:00