[agent-job-health] Agent Job Health Monitor - 2026-08-28: fleet improved, 1 novel Windows Runner failure, 3 schedule blind spots

[github-actions[bot]]

Summary

Note

Fleet health improved vs. the 2026-08-23 baseline: run-weighted agent-job failure rate dropped from 9.22% → 7.41%, and workflow-weighted mean failure rate dropped from 16.8% → 8.84%. No >10-point regression in either direction.

• Window: last 24h ending 2026-08-28T23:00Z
• Total runs: 351 across 166 distinct workflows
• Agent-job failures: 26 (run-weighted rate 7.41%; workflow-weighted median 0.0%, mean 8.84%)
• Novel (untracked) failures: 2 clusters, 4 runs, 2 workflows — none meets the ≥2-workflow / ≥10%-of-runs bar for a standalone issue
• Schedule blind spots: 3 untracked (see below) → 1 issue filed covering all three

Warning

Windows Runner Integration Test fails 100% of its runs (3/3) at the Setup Scripts step, and is not covered by any open issue — flagged as Novel below.

Failure Rate by Step

Failing step (agent job)	Runs	Distinct workflows	Status
Execute Claude Code CLI	8	2 (Avenger, Deep Report)	Tracked
Execute GitHub Copilot CLI	8	6 (Code Scanning Fixer, PureLock, Dead Code Removal Agent, Daily BYOK Ollama Test, Agent Performance Analyzer, PR Triage Agent)	Tracked
Execute Codex CLI	2	2 (Duplicate Code Detector, Daily Max Ai Credits Test)	Tracked
Setup Scripts (pre-flight)	3	1 (Windows Runner Integration Test)	Novel
Install and build docs / Build documentation	2	2 (Multi-Device Docs Tester, Visual Regression Checker)	Mixed (see below)
Execute Pi CLI	1	1 (Package Specification Enforcer)	Tracked
Check KVM availability for docker-sbx	1	1 (Daily Fact)	Tracked
Build and install gh-aw CLI from source	1	1 (Smoke Copilot)	Tracked

Tracked Failures (existing open issues)

Workflow	Runs	Failing step	Issue
Avenger	7	Execute Claude Code CLI	#56737, #56728, #56694, #56647, #56600
Code Scanning Fixer	3	Execute GitHub Copilot CLI	#56705, #56582, #56656
PR Triage Agent, PureLock, Dead Code Removal Agent, Daily BYOK Ollama Test, Agent Performance Analyzer	5	Execute GitHub Copilot CLI	#55413 (cross-workflow Copilot-BYOK cluster), plus #56645, #56664, #56738, #56651
Duplicate Code Detector	1	Execute Codex CLI	#56734
Daily Max Ai Credits Test	1	Execute Codex CLI	#54242 (chronic Codex CLI cluster)
Deep Report	1	Execute Claude Code CLI	#56644
Package Specification Enforcer	1	Execute Pi CLI	#56629
Daily Fact	1	Check KVM availability for docker-sbx	#56665
Visual Regression Checker	1	Build documentation	#56627
Smoke Copilot	1	Build and install gh-aw CLI from source	#56479 / #56477 (issue groups)

22 of 26 agent-job failures are already covered by an existing "[aw] X failed" auto-filed issue or a broader tracked cluster.

Novel Failure Clusters

2 novel clusters, 4 runs, 2 workflows — neither meets the issue-creation threshold alone

1. Windows Runner Integration Test — Setup Scripts step, 3/3 runs failed (100%)

• Distinct workflows affected: 1
• Engine: n/a (fails before agent CLI invocation)
• Representative run: §33140830599
• Suspected root cause: pre-flight Setup Scripts step failing consistently on the Windows runner, before any Copilot/Claude/Codex CLI step executes — likely an environment/toolchain provisioning issue specific to the Windows runner image, not an engine-side failure.
• Blast radius: single workflow, but 100% failure rate — every scheduled run of this workflow is currently non-functional.

2. Multi-Device Docs Tester — Install and build docs step, 1/1 run failed

• Distinct workflows affected: 1
• Representative run: §33154183421
• Suspected root cause: docs build/install step failure, isolated to this workflow's toolchain (separate signature from Visual Regression Checker's Build documentation failure, despite similar naming).
• Blast radius: single workflow, single run — below the ≥2-workflow / ≥10%-of-runs bar for a standalone issue.

Neither cluster independently triggers a new issue per the issue-creation criteria. Recommend filing if Windows Runner Integration Test's 100%-failure streak continues past today.

Schedule Heartbeat

3 schedule-triggered workflows have gone silent well past their expected cadence, with no existing tracking issue:

Workflow	Expected cadence	Last observed run	Gap
Workflow Craft Agent	weekly (Monday 09:00)	~2026-07-31 (28 days ago)	~4x cadence
Feature Grower	daily on weekdays	No run observed (API returns total_count:1, empty workflow_runs[])	Indeterminate — likely stuck
Windows Grower	daily	No run observed (same API anomaly)	Indeterminate — likely stuck

3 other workflows (Daily Hippo Learn, Smoke CI, GEO Optimizer Daily Audit) also show no recent runs but are disabled_manually — explained gaps, not blind spots.

One issue is being filed to cover all three genuine blind spots.

Per-Workflow Breakdown

166 distinct workflows ran in the window; 16 had at least one agent-job failure

Workflow	Runs	Failures	Failure rate
Avenger	7	7	100%
Windows Runner Integration Test	3	3	100%
Code Scanning Fixer	3	3	100%
PR Triage Agent	3	1	33%
Deep Report	3	1	33%
Visual Regression Checker, Smoke Copilot, PureLock, Package Specification Enforcer, Multi-Device Docs Tester, Duplicate Code Detector, Dead Code Removal Agent, Daily Max Ai Credits Test, Daily Fact, Daily BYOK Ollama Test, Agent Performance Analyzer - Meta-Orchestrator	1 each	1 each	100%
Test Quality Sentinel	24	0	0%
Ponytail Reviewer	21	0	0%
Impeccable Skills Reviewer	21	0	0%
PR Code Quality Reviewer	20	0	0%
Matt Pocock Skills Reviewer	20	0	0%
Design Decision Gate 🏗️	20	0	0%
(remaining ~150 workflows)	—	0	0%

Recommendations

1. Investigate Windows Runner Integration Test's Setup Scripts failure — 100% failure rate, no tracking issue, likely a Windows-runner provisioning regression.
2. Look into the Feature Grower / Windows Grower API anomaly (total_count:1, empty workflow_runs[]) — may indicate these daily workflows are silently stuck rather than just quiet.
3. Continue monitoring the Execute GitHub Copilot CLI cluster (8 runs, 6 workflows) — still the single largest failure signature despite existing tracking ([aw-failures] [aw] Copilot-BYOK auth rejected by api-proxy — Codex 404 (Ponytail Reviewer, Issue Monster) + Claude 401 (Smoke Claude on Copilo #55413, [aw-failures] [P0] Cross-engine agent-CLI segfault (exit 139) — confirmed on Auto-Triage/Pi CLI, now spread to 8 workflows acros #54186); consider escalating if the cross-workflow spread grows.
4. No action needed on the day-over-day trend — both failure-rate metrics improved this cycle.

References:

• §33140830599 — Windows Runner Integration Test, Setup Scripts failure
• §33154183421 — Multi-Device Docs Tester, docs build failure
• §33216999126 — Avenger, Execute Claude Code CLI failure

Warning

Firewall blocked 2 domains

The following domains were blocked by the firewall during workflow execution:

• api.anthropic.com
• api.github.com

[!TIP]
api.github.com is blocked because GitHub API access uses the built-in GitHub tools by default. Instead of adding api.github.com to network.allowed, use tools.github.mode: gh-proxy for direct pre-authenticated GitHub CLI access without requiring network access to api.github.com:

tools:
  github:
    mode: gh-proxy

See GitHub Tools for more information on gh-proxy mode.

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.anthropic.com"
    - "api.github.com"

See Network Configuration for more information.

Generated by 🩺 Agent Job Health Monitor · claude · agent · 573.7 AIC · ⌖ 8.35 AIC · ⊞ 9K · ◷

• expires on Aug 31, 2026, 4:00 PM UTC-08:00

[github-actions[bot]]

Caveman say: build good. Test done. Discussion nice. Me leave mark here.

Warning

Firewall blocked 6 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• contentautofill.googleapis.com
• www.google.com
• www.gstatic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "contentautofill.googleapis.com"
    - "www.google.com"
    - "www.gstatic.com"

See Network Configuration for more information.

📰 BREAKING: Report filed by Smoke Copilot · copilot · auto · 35.4 AIC · ⌖ 2.38 AIC · ⊞ 9.2K · ◷
_{Comment /smoke-copilot to run again
Add label smoke to run again}