[agent-job-health] Agent Job Health: fleet failure rate 5.42% (11/203 runs), 24h ending 2026-10-01 23:21 UTC

[github-actions[bot]]

Summary

• Window evaluated: last 24 full hours ending 2026-10-01 ~23:21 UTC
• total_runs: 203 · distinct_workflows: 63 · agent_failures: 11
• Run-weighted fleet failure rate: 11 / 203 = 5.42%
• Workflow-weighted rate: median 0.0%, mean 17.46% (across 63 distinct workflows)
• Tracked vs. novel: 5 tracked (45.5%) / 6 novel (54.5%)
• Trend vs. last recorded entry (2026-09-23): rate fell from 17.97% → 5.42% (−12.6 points) and workflow-weighted mean fell from 34.5% → 17.46% — a >10-point swing, but in the improving direction, not a regression.

Warning

One novel failure cluster ("Redact secrets in logs" step failing after a successful CLI run) affects 3 distinct workflows across 2 engines — see below.

Sample-size note: 11 failures out of 203 runs is a small absolute count. The 11 failures are spread across 11 different workflows (no workflow failed more than once), so the workflow-weighted mean (17.46%) is pulled up entirely by one-off, low-frequency workflows each failing their single run in the window — not by a chronic repeat offender. None of the three workflows explicitly named as chronic offenders in this monitor's charter (PR Sous Chef, Issue Monster, Contribution Check) failed this window (34/18/2 runs, 100% success each).

Failure Rate by Step

Failing step	Runs	Share of agent_failures
Execute GitHub Copilot CLI	5	45.5%
Redact secrets in logs (post-successful CLI exec)	3	27.3%
Start MCP Gateway	1	9.1%
Install Codex CLI (stuck/timeout)	1	9.1%
Warm up model (BYOK Ollama)	1	9.1%

Tracked Failures

Workflow	Failing runs	Existing coverage
Daily Reliability Review, Daily SPDD Spec Planner, Daily Security Observability Report, PR Triage Agent, Code Scanning Fixer	5 (all Execute GitHub Copilot CLI, engine=copilot)	Matches a well-established recurring cross-engine agent-CLI crash pattern previously filed as #54186 (closed 2026-08-27, not_planned/expired) and #55413 (closed 2026-08-31, not_planned/expired), both auto-filed by the [aw] Failure Investigator (6h) workflow. That investigator workflow ran 3× successfully in this window (§36873382883, §36889117810, §36935844892) and, per its established pattern, is the appropriate owner for re-filing/updating a tracker for today's recurrence. GitHub search in this session was integrity-filtered for most issue queries, so I could not confirm today's specific issue number — treating as tracked by pattern-match rather than opening a duplicate. Recommend a maintainer confirm whether a fresh open issue exists for this window's instance; if not, let the Failure Investigator's next run pick it up rather than hand-filing one.

Novel Failure Clusters

1. "Redact secrets in logs" step fails after the agent CLI itself already succeeded — issue filed, see below

• Count: 3 runs, 3 distinct workflows
• Workflows / engines: Deep Report (claude), Daily Agent of the Day Blog Writer (copilot), Daily Ambient Context Optimizer (copilot)
• Representative run: §36907433969 (Deep Report)
• Evidence: job_details[].steps[] shows Execute Claude Code CLI / Execute GitHub Copilot CLI = success, but the later Redact secrets in logs step = failure, on the same job.
• Suspected root cause / blast radius: spans both the Claude and Copilot engines, so this looks like a bug in the shared post-execution redaction script itself rather than an engine-specific issue — fleet-wide blast radius (any workflow reaching this step is exposed), not confined to one workflow or engine.

2. "Start MCP Gateway" step failure — single occurrence, no issue filed (below threshold)

• Count: 1 run — Daily Fact (engine=codex)
• Representative run: §36876021277
• Root cause: not isolated within budget; gateway process failed to start for this run only.

3. "Install Codex CLI" step stuck/timeout — single occurrence, no issue filed (below threshold)

• Count: 1 run — Daily Cache Strategy Analyzer (engine=codex)
• Representative run: §36908325221
• Evidence: step log shows Install Codex CLI stuck at status: in_progress with all subsequent steps pending when the job was terminated — looks like an install/network hang rather than a crash.

4. "Warm up model" step failure — single occurrence, no issue filed (below threshold)

• Count: 1 run — Daily BYOK Ollama Test
• Representative run: §36935063590
• Root cause: this is a BYOK/Ollama smoke-test workflow; model warm-up failures here are plausibly infra/model-availability flakiness specific to that experimental engine path rather than a fleet signal.

Schedule Heartbeat

240 schedule-triggered workflows were identified from .github/workflows/*.md frontmatter (excluding shared/ includes and workflow_dispatch-only workflows). A stratified sample of 51/240 (≈21%), covering every cadence class and specifically including audit-workflows (the workflow originally affected by #53252), was checked against the GitHub Actions API for most-recent-run-of-any-kind.

No blind spots detected in the 51 checked — every sampled workflow's most recent run was well inside 2×cadence+1 day (most ran within the last 24h). This was a partial sweep (189/240 workflows not checked) due to the scale of the fleet; it should not be read as a clean bill of health for the full 240.

Aside (out of scope for this phase but worth flagging): several of the checked workflows' most recent run had failed outright — audit-workflows, daily-code-metrics, daily-squid-image-scan, workflow-normalizer, daily-agentrx-trace-optimizer, windows-grower, daily-spec-coverage-kiro, ubuntu-image-analyzer, pr-triage-agent, code-scanning-fixer, deep-report, daily-schema-audit-cursor, example-workflow-analyzer, weekly-network-domains-audit, archivx-agentic-workflows-analyzer. These are visible failures (they fired and errored), not silent gaps, so they don't count as blind spots here — several already appear in this report's failure clusters above.

View per-workflow breakdown

Workflow	Runs	Agent failures	Failure rate
PR Sous Chef	34	0	0%
Issue Monster	18	0	0%
Design Decision Gate 🏗️	13	0	0%
Impeccable Skills Reviewer	13	0	0%
Matt Pocock Skills Reviewer	13	0	0%
PR Code Quality Reviewer	13	0	0%
Ponytail Reviewer	13	0	0%
Test Quality Sentinel	13	0	0%
AI Moderator	11	0	0%
Daily Go Test Parallelizer	5	0	0%
[aw] Failure Investigator (6h)	3	0	0%
Auto-Triage Issues	2	0	0%
Contribution Check	2	0	0%
Code Scanning Fixer	1	1	100%
Daily Agent of the Day Blog Writer	1	1	100%
Daily Ambient Context Optimizer	1	1	100%
Daily BYOK Ollama Test	1	1	100%
Daily Cache Strategy Analyzer	1	1	100%
Daily Fact	1	1	100%
Daily Reliability Review	1	1	100%
Daily SPDD Spec Planner	1	1	100%
Daily Security Observability Report	1	1	100%
Deep Report	1	1	100%
PR Triage Agent	1	1	100%
(49 more workflows, 1 run each, 0 failures)	49	0	0%

Recommendations

1. File/confirm tracking for the Copilot-CLI cluster (5 runs, tied to a recurring cross-engine crash pattern): ask a maintainer to check whether [aw] Failure Investigator (6h) already opened a fresh issue for this window's recurrence before filing a new one — don't duplicate [aw-failures] [P0] Cross-engine agent-CLI segfault (exit 139) — confirmed on Auto-Triage/Pi CLI, now spread to 8 workflows acros #54186/[aw-failures] [aw] Copilot-BYOK auth rejected by api-proxy — Codex 404 (Ponytail Reviewer, Issue Monster) + Claude 401 (Smoke Claude on Copilo #55413's lineage.
2. Investigate the "Redact secrets in logs" cluster (novel issue filed from this report): since the step fails after a successful model turn and spans 2 engines, start by diffing what differs between Deep Report/Daily Agent of the Day Blog Writer/Daily Ambient Context Optimizer's log volume or content vs. the many same-day runs where this step succeeded — likely a size/pattern edge case in the redaction script, not an engine bug.
3. Daily Cache Strategy Analyzer: look at whether Install Codex CLI has a missing timeout/retry — a stuck in_progress step burns the full job timeout with zero useful signal.
4. Schedule heartbeat: extend the Phase 6 sweep to the full 240 schedule-triggered workflows on a future run (only 51 were checked this time) to get a complete blind-spot picture, and consider wiring the per-step failure-rate breakdown in this report into that same sweep so visibly-failing scheduled workflows aren't checked twice by two different monitors.

References:

• §36907433969 — Deep Report, Redact secrets in logs failure (novel cluster representative)
• §36937679913 — Daily Reliability Review, Execute GitHub Copilot CLI failure (tracked cluster representative)
• §36908325221 — Daily Cache Strategy Analyzer, Install Codex CLI stuck/timeout

Generated by 🩺 Agent Job Health Monitor · claude · agent · 961.7 AIC · ⌖ 6.92 AIC · ⊞ 6.8K · ◷

• expires on Oct 4, 2026, 3:52 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Agent Job Health Monitor.

A newer discussion is available at Discussion #65133.