[agent-job-health] Agent Job Health Monitor - 2026-08-18: 3.33% fleet failure rate, 3 untracked schedule blind spots

[github-actions[bot]]

Summary

Window: 2026-08-17T22:46:22Z to 2026-08-18T22:46:22Z (24h, agent-job scope only)

• Total runs: 300 across 80 distinct workflows
• Agent-job failures: 10
• Fleet failure rate (run-weighted): 3.33% (10/300)
• Workflow-weighted rate: mean 11.25%, median 0% - the mean is pulled entirely by a handful of low-volume (1-2 run) workflows; the typical workflow in the fleet had zero agent-job failures. Failures are concentrated, not systemic.
• Tracked vs novel: 8 tracked runs / 2 novel runs
• Trend: no prior history found in history.jsonl - this is the baseline entry, so no day-over-day comparison is available yet.

Note

Fleet-wide agent-job health is stable this window (3.33% run-weighted failure rate, concentrated in a handful of chronic-offender workflows already under active investigation). No regression signal - this is the first recorded baseline. The one actionable finding is in Schedule Heartbeat below: 3 scheduled workflows have gone silent well past their expected cadence.

Failure Rate by Step

Failing Step	Runs	Distinct Workflows	Engine(s)
Execute GitHub Copilot CLI	6	6	copilot
Start MCP Gateway	2	1 (Scout)	claude
Execute Claude Code CLI (30m timeout)	1	1 (Agentic Workflow Audit Agent)	claude
Check KVM availability for docker-sbx	1	1 (Daily Fact)	codex

Tracked Failures

Workflow	Runs	Tracking Issue	Basis
Scout	2	#53191	MCP Gateway digest-pin - confirmed match by the 2026-08-18 19:00 UTC Failure Investigator report (#53804)
Architecture Guardian	1	#52253	Copilot proxy (port 10002) connection refusals - confirmed match by #53804
Linter Miner	1	#52253	Same signature - confirmed match by #53804
CLI Consistency Checker	1	#52253 (likely)	Identical Execute GitHub Copilot CLI signature; not independently re-verified this window
CI Optimization Coach	1	#52253 (likely)	Identical signature; not independently re-verified this window
Daily Malicious Code Scan Agent	1	#52253 (likely)	Identical signature; not independently re-verified this window
Daily BYOK Ollama Test	1	#52253 (likely)	Identical signature; not independently re-verified this window

No duplicate issue opened for this cluster - recommend the 4 unconfirmed rows above be attached to #52253 once re-verified (see Recommendations).

Novel Failure Clusters

Two single-occurrence, single-workflow failures - below the >=2-workflow / >=10%-of-runs threshold for opening a new issue. Each already has a per-run auto-filed failure issue; neither has a root-cause tracking issue yet:

Workflow	Runs	Step	Signature	Auto-Issue	Representative Run
Daily Fact	1	Check KVM availability for docker-sbx	Sandbox/hypervisor precheck failed before the agent CLI ran	#53712	§32147275242
Agentic Workflow Audit Agent	1	Execute Claude Code CLI	Agent job exceeded the 30-minute timeout (guardrail, not a crash)	#53835	§32186551970

Schedule Heartbeat

Warning

3 of 187 schedule-triggered workflows have gone silent well beyond 2x their expected cadence, with no open issue tracking the gap. All three last runs completed successfully - they simply stopped being triggered, which is the defining "silent blind spot" pattern this check targets.

Workflow	Lock File	Last Run	Last Status	Expected Cadence	Gap
craft	.github/workflows/craft.lock.yml	2026-07-31T05:28:34Z	completed	weekly (Mon 09:00)	~18.7d (threshold 15d)
daily-hippo-learn	.github/workflows/daily-hippo-learn.lock.yml	2026-07-27T07:24:10Z	completed	daily (~07:00 UTC)	~22.6d (threshold 3d)
smoke-ci	.github/workflows/smoke-ci.lock.yml	2026-07-18T08:57:59Z	completed	daily	~31.6d (threshold 3d)

smoke-ci also has a pull_request trigger, so it may still be running on PR events even though its schedule: trigger looks dormant. An issue has been opened for this finding (see below) since it was untracked.

Full per-workflow breakdown (80 workflows, 300 runs)

Workflow	Runs	Failures	Rate
Scout	2	2	100%
CLI Consistency Checker	1	1	100%
CI Optimization Coach	1	1	100%
Daily Malicious Code Scan Agent	1	1	100%
Daily Fact	1	1	100%
Architecture Guardian	1	1	100%
Linter Miner	1	1	100%
Agentic Workflow Audit Agent	1	1	100%
Daily BYOK Ollama Test	1	1	100%
PR Sous Chef	40	0	0%
Test Quality Sentinel	24	0	0%
Ponytail Reviewer	22	0	0%
Matt Pocock Skills Reviewer	21	0	0%
Design Decision Gate	21	0	0%
Impeccable Skills Reviewer	21	0	0%
PR Code Quality Reviewer	21	0	0%
Issue Monster	12	0	0%
Squad Implement Worker	12	0	0%
Squad	8	0	0%
Daily Go Test Parallelizer	6	0	0%
AI Moderator	6	0	0%
Auto-Triage Issues	5	0	0%
PR Description Updater	5	0	0%
Avenger	4	0	0%
Contribution Check	3	0	0%
Code Scanning Fixer	2	0	0%
[aw] Failure Investigator (6h)	2	0	0%
PR Triage Agent	2	0	0%
Deep Report	2	0	0%
(51 additional workflows, 1 run each, 0 failures)	51	0	0%

Note: PR Sous Chef, Issue Monster, and Contribution Check - the fleet's known chronic offenders - had zero agent-job failures this window. Their visible failure volume (many open "[aw] Failed jobs: ..." issues, e.g. the PR Sous Chef safe_outputs-job JSON-parse crash tracked in #53804) is entirely in the safe_outputs job, which is out of scope for this report.

Recommendations

1. Re-verify whether CLI Consistency Checker, CI Optimization Coach, Daily Malicious Code Scan Agent, and Daily BYOK Ollama Test share the exact root cause tracked in [aw-failures] [P1] Copilot engine proxy (port 10002) refuses connections despite passing key validation — Daily Assign Issue To [Content truncated due to length] #52253 (Copilot proxy port 10002 refusals); if confirmed, fold them into that issue explicitly so future reports classify them as tracked with confidence.
2. Watch Daily Fact's docker-sbx KVM precheck failure (§32147275242) for recurrence - single occurrence today, no action needed yet.
3. Consider raising timeout-minutes for Agentic Workflow Audit Agent above the current 30 minutes if this becomes recurring (see [aw] Agentic Workflow Audit Agent timed out #53835).
4. Investigate and restore the 3 schedule blind spots below - all stopped triggering after a clean run, suggesting a scheduling/registration issue rather than a code failure.

References:

• §32147275242 - Daily Fact KVM precheck failure
• §32186551970 - Agentic Workflow Audit Agent timeout
• §32152602055 - Scout MCP Gateway failure

Generated by 🩺 Agent Job Health Monitor · agent · 441.2 AIC · ⌖ 10.4 AIC · ⊞ 8.1K · ◷

• expires on Aug 21, 2026, 3:51 PM UTC-08:00