[agent-job-health] Agent Job Health Monitor - 2026-10-12

[github-actions[bot]]

Agent Job Health Monitor — 2026-10-12

Note

No fleet-wide regression. Failure rate is driven by a long tail of one-off, isolated failures rather than a broad or chronic pattern — but this run's log collection and issue-tracking checks were both partial; see caveats below before treating any number here as definitive.

Summary

• Window evaluated: intended last 24 full hours ending at workflow start (UTC). Actual coverage achieved: ~8.5 hours, 2026-10-12T14:53:31Z → 2026-10-12T23:22:22Z. Of the 313 pre-downloaded run directories under /tmp/gh-aw/aw-mcp/logs/, only 231 had complete aw_info.json + run_summary.json pairs; the other 82 were incomplete (partial download) and excluded. This is not a full 24h sample — treat all rates below as a snapshot of a late-evening ~8.5h slice, not the full day.
• total_runs = 231, distinct_workflows = 61, agent_failures = 14
• Run-weighted fleet failure rate: 14 / 231 = 6.06%
• Workflow-weighted rate: median 0.0%, mean 13.93% — the gap between median (most workflows: zero failures) and mean (pulled up by several 1-run/1-failure workflows at 100%) confirms the aggregate is not evidence of a broad regression; it's a handful of isolated single-run failures plus two slightly-elevated repeat offenders.
• Tracked vs. novel split: could not be verified this run — see "Tool Access Limitation" below.
• Trend vs. previous entries (/tmp/gh-aw/cache-memory/agent-job-health/history.jsonl): the most recent prior entry (2026-10-09) measured a run-weighted rate of 3.04% (with an upper bound of 18.25% due to 40 indeterminate runs that day) and a workflow-weighted mean of 13.62%. Today's 6.06% run-weighted rate is +3.0pp, and the mean (13.93%) is essentially flat (+0.3pp) — not a >10pp regression signal. Both the 2026-10-09 and today's entries share the same partial-window caveat (~7.8h and ~8.5h respectively), so day-over-day comparisons here are approximate.

Tool Access Limitation (read this before the rest)

GitHub issue search and direct issue reads (search_issues, list_issues, issue_read) were blocked by an integrity policy for the entire duration of this run (every call returned [Filtered] ... integrity policy or an empty result with a redaction notice). This means Phase 3 (Tracked vs. Novel classification) could not be performed live — I have no way to confirm which of today's failure clusters already have an open tracking issue.

To avoid creating a duplicate issue, no new issue was opened this run. Instead, the clusters below are cross-referenced against the tracked_issue values recorded in prior runs of this same history file (best-effort, may be stale — those issues could since have been closed, or new ones opened that I can't see).

Failure Rate by Step

Failing step	Runs	Share of agent_failures
Execute Pi CLI	4	28.6%
Execute GitHub Copilot CLI	4	28.6%
Install Pi CLI	2	14.3%
Start MCP Gateway	2	14.3%
Execute Claude Code CLI	1	7.1%
Warm up model (Ollama)	1	7.1%

Tracked Failures (best-effort — see limitation above)

Workflow(s)	Failing runs	Prior/likely tracking issue
AI Moderator, Issue Monster, PR Sous Chef, Smoke Pi Routed (Pi CLI install/execute, 6 runs)	6	Pi-engine CLI failures were tracked under #65909 on 2026-10-06 (HTTP 429 rate-limit theme); unverified whether still open or whether today's failures are the same root cause
Copilot Opt, Daily Reliability Review, Daily SPDD Spec Planner, Daily Security Observability Report (Copilot CLI, 4 runs)	4	Copilot CLI step failures have recurred in nearly every prior window (tracked under #54186 → #55413 historically); current status unverified

Novel Failure Clusters

Classification as "novel" vs. "tracked" is unverified this run (see limitation above). Reporting as clusters requiring human triage:

1. Start MCP Gateway failure (codex engine) — 2 runs, 2 distinct workflows (Agentic Workflow Audit Agent, Daily Cache Strategy Analyzer). Representative run: 38309913711 (Agentic Workflow Audit Agent, 2026-10-12T21:20:08Z). This exact step/engine combination does not appear in any prior entry of the trend history file, so it is plausibly genuinely new — but this could not be confirmed against open issues. Suspected root cause: codex engine's MCP gateway startup; blast radius appears codex-engine-specific rather than fleet-wide (only 2 of the ~organization's codex-engine runs failed this way in the window). Recommended action: a maintainer should check these two runs' "Start MCP Gateway" step logs directly and search open issues for existing tracking before filing anything new.
2. Execute Claude Code CLI — 1 run, Workflow Portfolio Analyst (38275493621). Single isolated occurrence, below the 2-workflow / 10% threshold for action.
3. Warm up model (Ollama) — 1 run, Daily BYOK Ollama Test (38316937103), log shows an orphaned ollama process being terminated post-job — consistent with the BYOK/local-model warm-up step being inherently flaky. Single isolated occurrence.

No cluster in this window independently meets the "2+ distinct workflows or ≥10% of runs" issue-creation bar with confirmed novel status, so no issue was opened. The Pi CLI and Start MCP Gateway clusters are the closest candidates and are flagged above for manual follow-up once issue search access is restored.

Schedule Heartbeat

245 workflows in this repo declare a schedule: trigger (241 active, 4 disabled at the gh-aw config level). Checking all 241 active workflows individually against the GitHub Actions API for staleness was not exhaustive given the repo's scale (338 workflows total, 40,000+ historical Actions runs) — this would require up to 241 individual API calls, which was out of budget for this run. The check below covers: (a) the 4 workflows gh-aw reports as disabled, confirmed against the live GitHub Actions API, and (b) the 61 distinct workflows that actually appeared in this run's ~8.5h sample (all healthy/recently active by definition).

Workflow	.lock.yml	Last observed state	Expected cadence	Root cause
GEO Optimizer Daily Audit	daily-geo-optimizer.lock.yml	disabled_manually	daily	Explicitly disabled — not a silent misfire
Daily Hippo Learn	daily-hippo-learn.lock.yml	disabled_manually	daily (~7:00)	Explicitly disabled — not a silent misfire
Slide Deck Maintainer	slide-deck-maintainer.lock.yml	disabled_manually	daily (~16:00 weekdays)	Explicitly disabled — not a silent misfire
Smoke CI	smoke-ci.lock.yml	disabled_manually	every 2 days	Explicitly disabled — not a silent misfire

All 4 are confirmed disabled_manually via the live Actions API — this matches gh-aw's own "disabled" status for these workflows, so it is a deliberate/known state rather than an undiagnosed blind spot. No action needed unless a maintainer intends to re-enable them (gh workflow enable <workflow>).

No additional blind spots were found among the 237 remaining active scheduled workflows that this run was able to check (cross-referenced against the ~8.5h run sample plus the 4 confirmed-disabled ones above), but this is not a full 241-workflow sweep — a dedicated, budgeted pass (e.g., a stratified sample of daily/sub-daily non-smoke workflows, as done in the 2026-10-09 entry) is recommended to restore full Phase 6 coverage next run.

View per-workflow breakdown (top 30 by failure/run count)

Workflow	Runs	Failures	Rate
PR Sous Chef	31	2	6.45%
Smoke Pi Routed	9	2	22.22%
Issue Monster	12	1	8.33%
AI Moderator	8	1	12.5%
Agentic Workflow Audit Agent	1	1	100%
Copilot Opt	1	1	100%
Daily BYOK Ollama Test	1	1	100%
Daily Cache Strategy Analyzer	1	1	100%
Daily Reliability Review	1	1	100%
Daily SPDD Spec Planner	1	1	100%
Daily Security Observability Report	1	1	100%
Workflow Portfolio Analyst	1	1	100%
Auto-Triage Issues	15	0	0%
Design Decision Gate	13	0	0%
Matt Pocock Skills Reviewer	13	0	0%
PR Code Quality Reviewer	13	0	0%
Ponytail Reviewer	13	0	0%
Test Quality Sentinel	13	0	0%
Smoke Claude Routed	10	0	0%
Smoke Copilot Routed	10	0	0%
Daily Trajectory Grader Implementer	9	0	0%
Smoke Copilot SDK Routed	9	0	0%
Daily Go Test Parallelizer	4	0	0%
Avenger	2	0	0%
Contribution Check	2	0	0%
Agent Job Health Monitor	1	0	0% (in progress — this run)
Agent Persona Explorer	1	0	0%
Agentic Workflow AIC Usage Optimizer	1	0	0%
Breaking Change Checker	1	0	0%

All other workflows observed in the window (33 more) had 0 failures across 1 run each.

Recommendations

1. Restore issue-search access before the next run. This run's biggest gap wasn't the failure rate (which looks healthy) — it was being unable to check search_issues/list_issues/issue_read at all, which blocked Phase 3 entirely and forced a no-issue-creation default out of caution. Investigate the "integrity policy" block on github/gh-aw issue reads for this workflow's token/permissions.
2. Manually triage the "Start MCP Gateway" (codex) cluster — Agentic Workflow Audit Agent (38309913711) and Daily Cache Strategy Analyzer — to confirm whether it's tracked or genuinely new; it's the only cluster this run that doesn't have a clear historical precedent in the trend file.
3. Confirm Pi CLI install/execute failures (6 runs, 4 workflows: AI Moderator, Issue Monster, PR Sous Chef, Smoke Pi Routed) are still covered by [aw] Issue Monster hit engine rate limit (HTTP 429) #65909, or reopen/file fresh if that issue has since closed — this is the single largest cluster this window and spans the most distinct workflows.
4. No action needed on the 4 disabled scheduled workflows (GEO Optimizer, Daily Hippo Learn, Slide Deck Maintainer, Smoke CI) — confirmed intentionally disabled, not silently failing.
5. Budget a full 241-workflow schedule sweep on a future run (or a rotating stratified sample) since this run's Phase 6 coverage was bounded by the scale of the fleet.

Generated by 🩺 Agent Job Health Monitor · claude · agent · 757 AIC · ⊞ 6.8K · ◷

• expires on Oct 15, 2026, 3:45 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Agent Job Health Monitor.

A newer discussion is available at Discussion #68553.