[audit-workflows] Agentic Workflow Audit — 2026-07-15: 81.8% (90% ex-Smoke CI), 18 fails all known CLI-driver issues, 0 novel

[github-actions[bot]]

🔍 Agentic Workflow Audit — 2026-07-15

Window: partial ~5.6h (15:38–21:14Z) · 100 runs fetched (99 concluded + 1 in-progress). The full -1d logs fetch timed out at the 120s bridge cap; 225 run dirs were downloaded and 100 indexed into summary.json — this is a representative partial slice, consistent with recent windows.

Health at a glance

Metric	Value
Concluded success rate	81.8% (81/99)
Ex–Smoke CI	90.0% (81/90)
Failures	18 (17 copilot + 1 codex)
Missing tools / data / MCP failures	0 / 0 / 0
Fleet tokens · AIC	1.59M · 3,937.8
Action-minutes · GitHub API calls	1,052 · 2,439

Every one of the 18 failures is a Execute *-CLI driver failure (0-turn or long-run-then-0-turn) that maps to an existing known issue. Zero novel anomalies, zero missing tools, zero MCP failures this window.

Per-engine reliability

Engine	Concluded	Success	Rate
pi	18	18	100%
claude	16	16	100% (+1 still in-progress)
copilot	62	45	72.6%
codex	3	2	66.7%

pi and claude are clean. All 18 failures are copilot (17) + codex (1) — the fleet's reliability drag is entirely the copilot/codex CLI-startup driver family.

Failure clusters (all known issues)

1. Smoke CI — 9 fails (50% of all failures) · DOMINANT chronic

copilot / claude-sonnet-4.6, fails on push (7) and pull_request (2) at Execute GitHub Copilot CLI, 0-turn / 0-tok, 4.3–5.4m each. 100% deterministic on every push to main.

• Known issue: smoke-ci-copilot-cli-100pct-fail-on-push — recurrence Weekly Research Report: AI Workflow Automation Landscape and Market Opportunities - August 2025 #7, chronic 06-30 → 07-15, unfixed.
• Removing Smoke CI lifts the window from 81.8% → 90.0%.
• Because it fires on every push, it is a constant red on main.
• Example: §29450871518

2. copilot-sdk-longrun — 8 fails · agent runs then Execute-CLI drops

Agent consumes tokens (40K–61K) over 6–29 min, then Execute GitHub Copilot CLI fails 0-turn. Affected this window:

Workflow	Duration	Tokens
Daily Security Observability Report	29.0m	57,887
Daily Project Performance Summary Generator (MCP Scripts)	23.7m	58,794
The Daily Repository Chronicle	18.0m	16,273
Tidy	15.9m	—
Daily Formal Spec Verifier	15.0m	60,679
Discussion Task Miner – Code Quality Improvement Agent	13.1m	40,062
Contribution Check	10.9m	23,349
Smoke Copilot Sub Agents	5.9m	15,327

• Known issue: copilot-sdk-driver-failures — recurrence Logs revamp. #25, chronic since 06-02.
• These burn real token/compute before dropping — the most wasteful failure mode.
• Example: §29431830781

3. codex-gh-aw-binary — 1 fail

Daily Cache Strategy Analyzer (codex), 7.0m, 0-turn at Execute Codex CLI.

• Known issue: codex-gh-aw-binary-not-found-for-mcp — recurrence add cli flag to guard dropping a agentic workflow instructinos file #6, ~100% fail, unfixed since 06-15.
• Example: §29441631700

Trend charts (30-day)

Fleet health is stable within its recent band (82–90%). Today's 81.8% sits at the low edge, but the ex–Smoke CI 90% is squarely on baseline — the headline dip is one chronically-broken workflow, not a fleet regression. The deep troughs on 06-20 (Skillet) and 07-02/07-03 (pi PR Sous Chef collapse) were prior one-off incidents that have since recovered.

The token chart shows the well-documented metrics.TokenUsage reporting outage: daily fleet tokens crater from 40–55M/day (early June) to 1–4M/day from ~06-20 onward. Per-run token_usage is still present (e.g. 60,679 on Daily Formal Spec Verifier), and AIC remains reliable — so cost is trackable, but raw token trend lines are not. This is a reporting gap, not a real usage collapse.

Recommendations (evidence-linked, all pre-existing)

1. Fix / quarantine Smoke CI (rec-investigate-distributed-0tok-degradation) — 100% deterministic red on main, 50% of all failures, 7 windows unfixed. Highest-leverage single fix: add a fast-fail diagnostic surfacing why Execute GitHub Copilot CLI aborts 0-turn, and compare its copilot config vs. copilot workflows that succeed. Evidence: §29450871518.
2. copilot-sdk-longrun driver (rec-sdk-toolperm-allowlist-or-relax-guard) — 8 workflows burn 15K–61K tokens then drop at Execute-CLI. Investigate the SDK headless-path session/tool-permission handling on long runs. Evidence: §29431830781.
3. codex gh-aw binary path (fix-cache-strategy-binary-path) — one chronically-100%-broken workflow since 06-15; ensure the gh-aw binary resolves for the codex MCP setup.
4. Restore metrics.TokenUsage aggregation (token-usage-reporting-gap) — or formally standardize on AIC as the fleet cost metric so token trend charts stop showing a phantom collapse.

Bottom line

No new problems, no missing tools, no MCP failures. The fleet is healthy on baseline once the single chronic Smoke CI offender is excluded (90%). The failure surface is fully explained by three long-standing, already-tracked CLI-driver issues — the actionable move remains fixing Smoke CI, which alone accounts for half of all failures and a constant red on main.

References:

• §29450871518 — Smoke CI 0-turn (dominant cluster)
• §29431830781 — copilot-sdk-longrun (Daily Security Observability, 29m/57.9K tok)
• §29441631700 — codex binary-not-found (Daily Cache Strategy Analyzer)

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by 🔍 Agentic Workflow Audit Agent · 213 AIC · ⌖ 40 AIC · ⊞ 7.4K · ◷

• expires on Jul 16, 2026, 1:49 PM UTC-08:00

[github-actions[bot]]

Smoke bot grunt.\nRun 29459778281 poke done.\nTiny fire still warm.

📰 BREAKING: Report filed by Smoke Copilot · 89.1 AIC · ⌖ 3.78 AIC · ⊞ 19K · ◷
_{Comment /smoke-copilot to run again
Add label smoke to run again}

[github-actions[bot]]

Smoke test: discussion interaction passed ✅

📰 BREAKING: Report filed by Smoke Copilot - AOAI (Entra) · 30.1 AIC · ⌖ 2.88 AIC · ⊞ 17.6K · ◷
_{Comment /smoke-copilot-aoai-entra to run again
Add label smoke to run again}