[audit-workflows] Agentic Workflow Audit — 2026-08-03 (24h, post 28-day gap): 86.6% success, copilot-sdk-driver-failures still dominant

[github-actions[bot]]

Overview

Full ~24h audit window (2026-08-02T21:36Z → 2026-08-03T21:38Z), fetched in 3 paginated pages: 255 runs, 220 success / 35 failure (raw 86.3%, 86.6% excluding the intentional Daily Max Ai Credits Test guardrail check). Zero missing tools, missing data, or MCP failures fleet-wide — every failure traces to a driver/infra step, not agent logic.

⚠️ Audit cadence gap: the last recorded audit in repo-memory was 2026-07-06 — 28 days before this run. Trend continuity is broken for that stretch; treat pre-gap history as a stale baseline only, and see the recommendation below to confirm the daily schedule is actually firing.

Metric	Value
Total runs	255 (253 ex. intentional tests)
Success rate	86.3% raw / 86.6% ex-intentional
Prod-main	154 runs, 86.4% success
Non-main (PR)	101 runs, 86.1% success
AI credits (AIC)	12,078.6
Tokens	10,659,990
Action minutes	3,127
GitHub API calls	4,430
Engine mix	copilot 162, claude 46, pi 36, codex 10

Trend Charts

Success rate closed today at 86.6%, in line with the pre-gap baseline (mid-80s to low-90s). The big blank stretch from 07-08 to 08-01 is the 28-day audit gap, not a quiet period on the fleet — no data was collected, not zero runs.

Token reporting remains sparse (known token-usage-reporting-gap issue) — most days have no recovered total, so the moving average line bridges long gaps rather than reflecting continuous data. Today's 10.66M recovered total is the highest since the 07-01–07-06 window.

Key Findings

1. copilot-sdk-driver-failures — still the dominant, chronic issue (open since 2026-06-02)

PR Code Quality Reviewer failed 9 of 17 runs (53%) — one paginated sub-window flagged it at 8/10 (80%) via the tool's own failure-hotspot insight. Failures cluster tightly: 8 of 9 fails landed in a 2h15m window (11:11–13:23Z). Failing step is Execute GitHub Copilot CLI (7 runs) or Install GitHub Copilot CLI (2 runs), 0-turn, all other jobs (pre_activation, activation, detection, evals, safe_outputs) green.

Same signature also hit: Matt Pocock Skills Reviewer, Daily Agent of the Day Blog Writer, CI Optimization Coach, Layout Specification Maintainer, Linter Miner.

The claude engine shows the identical crash-after-startup pattern via Execute Claude Code CLI: Design Decision Gate (×2), Workflow Portfolio Analyst, arXiv Paper Researcher — confirming this is cross-engine (tracked separately in memory as claude-agent-job-fail-longrun, same root family).

This has been open and unfixed for >2 months. Recommend prioritizing root-cause work (or at minimum a fast-fail diagnostic that surfaces why Execute-CLI aborts 0-turn) over further feature workflows.

2. NEW: push_evals_state git-push race — 6 workflows, one 65-minute window

Tidy, Copilot Session Insights, Code Scanning Fixer, Daily Elixir Credo Snippet Audit, Daily Syntax Error Quality Check, daily-experiment-report — 6 unrelated workflows, different engines/schedules, all failed the same push_evals_state → Push evals results to git step within a 65-minute window (07:45–08:50Z). Every one had a fully green agent job; only the evals-state git push failed.

This time-clustering across otherwise-unrelated workflows points to a shared-branch push race (non-fast-forward rejections from concurrent pushes) rather than 6 independent bugs. Recommend adding retry-with-rebase to that push step, or moving to per-workflow eval-state paths.

3. NEW: Install AWF binary — 3 workflows, one 4-minute window

Glossary Maintainer, PR Description Updater, Test Quality Sentinel all failed Install AWF binary within 11:09–11:13Z. Same tight-clustering signature as the historical Playwright-install incident (2026-07-01) — most likely a transient release-artifact/CDN blip rather than a regression. Low priority; add retry-with-backoff if it recurs.

4. Recovery: Daily Cache Strategy Analyzer's chronic codex binary issue looks resolved

The codex-gh-aw-binary-not-found-for-mcp issue (chronic ~100% fail, open since 2026-06-15) did not recur today — Daily Cache Strategy Analyzer succeeded cleanly with 1.18M tokens (highest token run of the day). Downgraded to RECOVERED-WATCH in memory; needs one more clean window before closing.

5. Minor / singleton observations

• PR Sous Chef (pi engine) failed 2/16 runs, but — unlike the historic pi-gpt54 agent-startup pattern — the agent job itself succeeded both times; failures were downstream at Process Safe Outputs (1) and the evals job (1). Logged as a distinct downstream-tooling signal, not counted against the existing pi issue.
• Daily Container Image Security Scan failed at Enforce critical vulnerability and license gates — this may be the gate correctly catching a real finding rather than an infra bug. Flagging for human triage rather than auto-classifying as an incident.
• MCP Inspector Agent failed once at Start MCP Gateway — single occurrence, watching for recurrence.

Recommendations

1. HIGH — Root-cause or fast-fail-diagnose copilot-sdk-driver-failures, focused on PR Code Quality Reviewer (53% fail rate today, dominant fleet reliability drag, unfixed 2+ months).
2. MEDIUM — Add retry/rebase to the push_evals_state git-push step to eliminate the concurrent-push race seen across 6 workflows today.
3. MEDIUM — Confirm the daily audit schedule is actually firing; a 28-day silent gap means any incidents between 07-06 and today went unreviewed.
4. LOW — Add retry-with-backoff to Install AWF binary; only escalate if the transient cluster recurs.

Repo memory (audit-history.jsonl, known-issues.json, recommendations.json, anomalies.json, workflow-trends.json, metrics-summary.json) has been updated with today's findings, two new tracked issues, and one recovery.

References:

• github/gh-aw Actions

Generated by 🔍 Agentic Workflow Audit Agent · agent · 210.8 AIC · ⌖ 41.2 AIC · ⊞ 6.5K · ◷

• expires on Aug 4, 2026, 1:54 PM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-08-04T21:54:00.103Z.

Closed by Workflow