[agent-job-health] Agent Job Health: 2026-10-07 daily report (12.77 percent fleet rate, redact-secrets regression)

[github-actions[bot]]

Summary

• Window evaluated: 2026-10-06 23:00 UTC → 2026-10-07 23:00 UTC
• total_runs: 94 · distinct_workflows: 38 · agent_failures: 12
• Run-weighted fleet failure rate: 12.77% (12/94)
• Workflow-weighted median: 0.0% · workflow-weighted mean: 23.68% (27 of 38 workflows had zero failures; the mean is pulled up by eight workflows that each ran once today and failed once)
• Tracked vs. novel: 2 tracked (16.7%) · 10 novel (83.3%)
• Trend vs. 2026-10-06: run-weighted rate fell from 19.67% → 12.77% (-6.9pp, not a >10pp swing in either direction). Workflow-weighted mean is roughly flat (22.9% → 23.68%); median stayed at 0% both days.

Warning

One novel, fleet-wide infrastructure regression was found: a post-execution "Redact secrets in logs" step is failing across 3 different agentic engines (codex, copilot, claude) and 4 different workflows, always after the agent itself completed successfully. This is new today (not present in the 2026-10-06 report) and is the largest single cluster (33% of all agent-job failures). An issue has been filed.

Failure Rate by Step

Failing step	Runs	Share of agent_failures
Redact secrets in logs	4	33.3%
Execute GitHub Copilot CLI	3	25.0%
Execute Codex CLI	2	16.7%
Start MCP Gateway	1	8.3%
Warm up model	1	8.3%
Download logs from last 24 hours	1	8.3%

Tracked Failures

Workflow	Failing runs	Existing coverage
Daily Go Test Parallelizer	2	Same signature as #65950 ("Unsupported Responses custom tool call 'exec'. Only 'apply_patch' is supported through the Copilot compatibility adapter.", invalid_request_error, codex engine routed through the Copilot compatibility adapter). #65950 was auto-filed for this exact failure on 2026-10-05 and auto-closed/expired on 2026-10-06 without a fix landing, so it has recurred twice more today. Treated as tracked (known signature), not novel — but the underlying bug is still unresolved and the auto-expiry is hiding that.

No other failing workflow or error signature today matched an open issue found by search (search results were partially filtered by the GitHub integrity policy — some matches may not be visible to this report; treat "no open issue found" as best-effort, not a guarantee).

Novel Failure Clusters

1. Redact secrets in logs step failure — ERR_VALIDATION: Failed to remove artifact source after secret redaction failed

• Count: 4 runs, 4 distinct workflows — Daily AWF Spec Compiler Surfacing Review (codex), Daily Ambient Context Optimizer (copilot), Daily Caveman Optimizer (claude), Deep Report (claude)
• Failing step / engine: post-agent "Redact secrets in logs" step; reproduced on all three engines (codex, copilot, claude) — this is infra-level, not engine-specific
• Representative run: 37695677725 (Daily AWF Spec Compiler Surfacing Review)
• Error excerpt:

  Found 1218 file(s) to scan for secrets (1083 in /tmp/gh-aw, 135 in /home/runner/work/_temp/gh-aw)
  ##[warning]Replaced non-writable file for secret redaction: /tmp/gh-aw/sandbox/firewall/audit/awf-resolved-config.json
  ##[warning]Replaced non-writable file for secret redaction: /tmp/gh-aw/sandbox/firewall/audit/docker-compose.redacted.yml
  ##[warning]Replaced non-writable file for secret redaction: /tmp/gh-aw/sandbox/firewall/audit/image-manifest.json
  ##[warning]Replaced non-writable file for secret redaction: /tmp/gh-aw/sandbox/firewall/audit/policy-manifest.json
  ##[warning]Replaced non-writable file for secret redaction: /tmp/gh-aw/sandbox/firewall/audit/sensitive-paths.json
  ##[error]ERR_VALIDATION: Secret redaction failed: ERR_VALIDATION: Failed to remove artifact source after secret redaction failed
  

  The copilot-engine instance (37676391386) shows the identical signature, one line earlier: ##[warning]Replaced non-writable file for secret redaction: /tmp/gh-aw/mcp-logs/safeoutputs/server.log → ##[error]ERR_VALIDATION: Secret redaction failed: ERR_VALIDATION: Failed to remove artifact source after secret redaction failed.
• Suspected root cause: redact_secrets.cjs replaces non-writable sandbox/firewall-audit files in place (logged as warnings), then a later step tries to remove the original artifact source and fails — likely because the replacement left a file with different ownership/permissions, or a race between the firewall sandbox teardown and the redaction step's cleanup. The agent's own execution always completed fine in all 4 cases; only this post-processing step fails, but it still fails the whole job.
• Blast radius: fleet-wide / engine-agnostic. Every workflow run that hits this scenario (apparently tied to certain sandbox/firewall-audit file states) will fail at the finish line even though the actual agentic work succeeded.
• Recommended action: file created (see below) — the fix likely belongs in redact_secrets.cjs or the firewall-audit artifact handling, making artifact-source removal tolerant of files it just rewrote in place.

2. Execute GitHub Copilot CLI — engine timeout (agentic_engine_timeout)

• Count: 2 runs, 2 distinct workflows — Impeccable Skills Reviewer, Matt Pocock Skills Reviewer (both pull_request-triggered, both copilot engine)
• Representative run: 37682943473
• Evidence: conclusion-job telemetry: agent.execution categories=["agentic_engine_timeout"] errorCodes=[] errorTypes=[] exitCode=0
• Suspected root cause: Copilot CLI engine exceeding its step/turn timeout on these two PR-review workflows; exit code 0 suggests a clean timeout cutoff rather than a crash.
• Blast radius: copilot engine only, 2 PR-review workflows so far.
• Recommended action: no open issue found; worth a follow-up if it recurs — consider raising the per-turn timeout or trimming prompt/tool scope for these two reviewer workflows.

3. Execute GitHub Copilot CLI — HTTP 429 rate limit

• Count: 1 run — Daily Reliability Review (schedule-triggered, copilot engine)
• Representative run: 37699051771
• Evidence: agent.execution categories=[] errorCodes=[429] errorTypes=["rate_limit"] exitCode=1
• Note: this is a different root cause from cluster 2 above, despite failing at the same step name — do not conflate the two "Copilot CLI" failures.
• Blast radius: single workflow, single run. Likely transient; monitor for recurrence.

4. Unconfirmed-root-cause singletons

• Agentic Workflow Audit Agent — failing step "Start MCP Gateway" (37688836213, codex)
• Daily BYOK Ollama Test — failing step "Warm up model" (37696303776, copilot — expected flakiness surface for a workflow that tests local Ollama model warm-up)
• Daily Cache Strategy Analyzer — failing step "Download logs from last 24 hours" (37668481951, codex)
• Each is isolated to one workflow, one run, under 10% of total runs, and root cause was not conclusively isolated within this report budget (the generic agentic_engine_timeout/ai_credits_rate_limit_error output keys are declared in every job output list regardless of value, so their presence in logs is not itself evidence — do not over-read it elsewhere in this fleet). No issue filed for these; flagging for awareness only.

Schedule Heartbeat

245 workflows declare a schedule: trigger in frontmatter. All were checked against the most recent Actions run of any kind.

Workflow	Lock path	Last observed run	Expected cadence	Gap	State
daily-geo-optimizer	.github/workflows/daily-geo-optimizer.lock.yml	2026-08-18T16:31:58Z	daily	50.3 days	disabled_manually
daily-hippo-learn	.github/workflows/daily-hippo-learn.lock.yml	2026-07-27T07:24:10Z	daily around 7:00	72.7 days	disabled_manually
slide-deck-maintainer	.github/workflows/slide-deck-maintainer.lock.yml	2026-09-23T16:24:44Z	daily around 16:00 on weekdays	14.3 days	disabled_manually
smoke-ci	.github/workflows/smoke-ci.lock.yml	2026-07-18T08:57:59Z	every 2 days	81.6 days	disabled_manually

All four are explained by disabled_manually — not an undiagnosed cron misfire. Recommend re-enabling (gh workflow enable <name>.lock.yml) if these are still wanted, or removing their schedule: trigger if intentionally retired, so they stop showing up as blind spots in future reports.

Also observed: smoke-pi-auto, smoke-claude-auto, smoke-claude-copilot-auto, and smoke-codex-bare-auto have never run (zero runs ever) — all four were created 2026-10-07T17:07:58Z, i.e. a few hours before this report, and are active. These are new workflows that haven't reached their first scheduled fire yet, not blind spots — no action needed, but worth a follow-up check tomorrow if they still haven't fired.

View per-workflow breakdown

Workflow	Runs	Agent failures	Failure rate
Daily Go Test Parallelizer	3	2	66%
Impeccable Skills Reviewer	6	1	16%
Matt Pocock Skills Reviewer	6	1	16%
Agentic Workflow Audit Agent	1	1	100%
Daily AWF Spec Compiler Surfacing Review	1	1	100%
Daily Ambient Context Optimizer	1	1	100%
Daily BYOK Ollama Test	1	1	100%
Daily Cache Strategy Analyzer	1	1	100%
Daily Caveman Optimizer	1	1	100%
Daily Reliability Review	1	1	100%
Deep Report	1	1	100%
Issue Monster	9	0	0%
[aw] Failure Investigator (6h)	9	0	0%
PR Sous Chef	7	0	0%
Design Decision Gate 🏗️	6	0	0%
PR Code Quality Reviewer	6	0	0%
Ponytail Reviewer	6	0	0%
Test Quality Sentinel	6	0	0%
Smoke Claude Dynamic Workflow	3	0	0%
Auto-Triage Issues	1	0	0%
Avenger	1	0	0%
Code Scanning Fixer	1	0	0%
Contribution Check	1	0	0%
Copilot Agent PR Analysis	1	0	0%
Copilot PR Prompt Pattern Analysis	1	0	0%
Daily Code Debt Cleanup — Aider	1	0	0%
Daily Code Metrics and Trend Tracking Agent	1	0	0%
Daily Graft Intelligence	1	0	0%
Daily Project Performance Summary Generator (Using MCP Scripts)	1	0	0%
Daily Regulatory Report Generator	1	0	0%
Daily Safe Output Integrator	1	0	0%
Daily Safe Output Tool Optimizer	1	0	0%
Daily Spec Coverage Review	1	0	0%
Daily Team Evolution Insights	1	0	0%
Duplicate Code Detector	1	0	0%
Lockfile Statistics Analysis Agent	1	0	0%
Release	1	0	0%
Scout	1	0	0%

Recommendations

1. Fix the redact-secrets/artifact-cleanup regression (cluster 1) — fleet-wide, engine-agnostic, 33% of today's failures, 4 distinct workflows. Issue filed below.
2. Re-enable or formally retire daily-geo-optimizer, daily-hippo-learn, slide-deck-maintainer, and smoke-ci — all four are disabled_manually, not misfiring; a deliberate decision is needed either way so they stop appearing as blind spots.
3. Watch the Copilot-engine timeout on Impeccable Skills Reviewer / Matt Pocock Skills Reviewer (cluster 2) — only 2 occurrences so far, not yet issue-worthy, but worth a follow-up if it recurs tomorrow.
4. Re-open or re-file [aw] Daily Go Test Parallelizer engine driver failed before emitting a terminal safe output #65950 (Daily Go Test Parallelizer / codex-via-Copilot-adapter unsupported_custom_tool error) — the auto-generated issue expired before a fix landed, and the same failure has now recurred twice more. The expiry mechanism silently dropped a real, repeating bug.

Generated by 🩺 Agent Job Health Monitor · claude · agent · 1.4K AIC · ⊞ 6.7K · ◷

• expires on Oct 10, 2026, 3:46 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Agent Job Health Monitor.

A newer discussion is available at Discussion #67304.