[audit-workflows] Daily Agentic Workflow Audit — 2026-08-23

[github-actions[bot]]

Window: full 24h, 2026-08-22T21:17Z → 2026-08-23T15:25Z · 442 runs across 5 paginated log fetches.

Key metrics

Metric	Value
Success rate (raw)	85.7% (379/442)
Success rate (excl. intentional-failure tests)	85.9%
Driver-exit failures	57 (89% of all 63 failures)
Agent-logic failures	6
Missing tools / missing data	0 / 0
AI credits consumed	14,239.7
Total tokens	12,864,315 (88% of runs now report nonzero)
GitHub API calls	7,703

Every failure this window classified as driver_exit or agent_logic — no missing-tool or missing-data gaps anywhere in the fleet.

Top issues

1. codex engine chronic failure — 65% fail rate (11/17 runs). Includes the Smoke Codex canary itself (4/6 fail, e.g. §32625165009, §32605816618) and a repeat of the long-known MCP-binary-path bug in Daily Cache Strategy Analyzer. Open since 2026-06-15, still unresolved 2+ months later — by far the worst per-engine number (copilot/pi/claude all 88–90%).
2. Cross-engine driver_exit outage, 2026-08-22T21:18Z–23:55Z. 27 failures across 15 distinct workflows and 5 engines (copilot/pi/claude/codex/null) inside 2.5 hours, each 4–6 minutes / zero turns — a driver-or-gateway startup failure, not agent logic. Daily Observability Report for AWF Firewall and MCP Gateway — the workflow that watches the gateway — failed the same way at 23:45Z (§32606091392). An in-flight branch, copilot/aw-failures-start-mcp-gateway-fix, is already attempting a remediation.
3. Cost outlier: Daily Go Test Parallelizer is the largest per-run token consumer in the fleet — 2.93M tokens/8 runs, one run alone hit 908,817 tokens (§32658662065) — and had 3/8 driver_exit failures.

Clustering — recurring vs. novel vs. resolved (prior audit on record: 2026-07-06)

Signal	Status this window
codex-gh-aw-binary-not-found-for-mcp	Still open, unresolved 2+ months; now the dominant engine-specific drag.
MCP-gateway driver_exit outage (21:18–23:55Z)	Novel — cross-engine, cross-workflow, time-boxed. Filed as a new known issue; fix branch already exists.
smoke-ci-copilot-cli-100pct-fail-on-push (chronic 06-30→07-06)	Not reproduced — Smoke CI doesn't appear in this window at all. Likely fixed/renamed; confirm before closing.
avenger-err-config-no-structured-logs (chronic 100%-fail, recur 24)	Major improvement — only 2 runs today (1 success, 1 generic driver_exit, different signature).
token-usage-reporting-gap (open since 06-19)	Likely resolved — 391/442 runs (88%) now report nonzero token_usage directly.
PR Sous Chef pi-engine collapse (07-03, 80% fail)	Recovered but not clean — 7/72 driver_exit fails today (9.7%); keep watching given ~every-20-min frequency.
chroot-node-not-available	Not reproduced — Daily Issues Report Generator ran clean.

Recommendations

1. HIGH — Fix or disable the codex-engine MCP binary lookup (codex-gh-aw-binary-not-found-for-mcp). Open since 2026-06-15, now driving a 65% fleet-wide fail rate on that engine. Highest-ROI fix on the board.
2. MEDIUM — Confirm the copilot/aw-failures-start-mcp-gateway-fix branch actually closes the 21:18–23:55Z outage pattern before merging; track its Smoke Copilot pass rate.
3. MEDIUM — Audit Daily Go Test Parallelizer token usage (908K tokens in a single run); check for chunking/caching opportunities and whether spikes correlate with its driver_exit fails.
4. LOW — Investigate the 48-day gap in audit memory (no records between 2026-07-06 and today). Confirm whether the daily audit was paused or silently failing — a monitoring blind spot this long could hide regressions.

Trend charts

Run volume has grown substantially since mid-June (peaking 400+ runs/day), while the success-rate line has ranged roughly 60–98% day to day, settling at 85.7% today. Note the 48-day gap in the underlying audit history between 2026-07-06 and today — the two charted points on either side of that gap are not adjacent days.

AI-credit consumption (used as the cost proxy — fleet-wide raw token totals have a known historical reporting gap that looks resolved as of today) peaked near 29K/day in late June and has settled into a lower, noisier band since, at 14.2K today. The 7-run trailing average smooths day-to-day swings visible in the raw bars.

Repo memory updated

audit-history.jsonl (+1 entry), metrics-summary.json (+1 entry), known-issues.json (6 updated, 1 added), anomalies.json (+1), recommendations.json (+3), workflow-trends.json (47 workflows snapshotted).

References:

• §32606091392 — Daily Observability Report for AWF Firewall and MCP Gateway (driver_exit, corroborates the outage window)
• §32625165009 — Smoke Codex (driver_exit, codex canary)
• §32658662065 — Daily Go Test Parallelizer (908,817-token run)

Generated by 🔍 Agentic Workflow Audit Agent · agent · 258.8 AIC · ⌖ 10.8 AIC · ⊞ 9.3K · ◷

• expires on Aug 24, 2026, 1:37 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Agentic Workflow Audit Agent.

A newer discussion is available at Discussion #55526.