[audit-workflows] Agentic Workflow Audit — 2026-08-02

[github-actions[bot]]

Window: last 24h (2026-08-01 21:30 → 2026-08-02 21:30 UTC), 293 runs across 3 paginated fetches.
⚠️ Audit gap: this workflow did not run for 27 days (2026-07-07 → 2026-08-01) — repo-memory has no entries in that span. Everything below is reconciled against the 2026-07-06 baseline, so some "resolved" calls rest on a single post-gap window rather than continuous observation.

Headline numbers

Metric	Value
Runs	293 (291 excl. 2 intentional-failure-test runs)
Success rate	94.2% raw / 94.5% excl. intentional failures
Real failures	16
Total AIC (cost)	13,690.5
Total tokens	11,655,665
Engine mix	Copilot 175, Claude 58, Pi 53, Codex 7
Missing tools / missing data / MCP failures	0 / 0 / 0

94.5% is a meaningful improvement over the 07-06 baseline (86.25% raw / 89.4% ex-Smoke-CI) — see charts below.

📊 Trend charts

Success runs, failure runs, and success rate over the observable window. The shaded band is the 27-day audit gap — note the line does not interpolate across it, since no data exists there. Success counts and rate both recovered strongly by 2026-08-02 versus the last observed point (2026-07-06).

Daily token totals had gone mostly null since ~06-19 (a previously tracked issue). Today's window reports a healthy 11.66M tokens with clean per-run values across all four engines — a good sign the reporting gap is closing, though we want one more window before calling it fixed.

Failure clustering

Every one of the 16 real failures shares the same signature: the agent completes its turns with ErrorCount=0 (no logged tool/agent errors), yet the run itself concludes failure. This is the same "driver-exit" pattern tracked in repo-memory as copilot-sdk-driver-failures and claude-agent-job-fail-longrun — now confirmed on a third engine (Pi):

Workflow	Engine	Fail rate this window
PR Code Quality Reviewer	Copilot	5/23 (21.7%)
PR Sous Chef	Pi (new for this engine)	4/29 (13.8%)
Design Decision Gate 🏗️	Claude Code	2/19 (10.5%)
Linter Miner, Daily Testify Uber Super Expert, Discussion Task Miner, Daily Container Image Security Scan, daily-experiment-report	Copilot	1 each

Why this is hard to root-cause from logs alone

run_summary.json / summary.json for every one of these failing runs shows 0 failed tool executions, 0 firewall blocks, and no error entries — the log schema exposed to this audit has no job/step-level breakdown, so we cannot tell which post-agent step (Execute-CLI, Process-Safe-Outputs, Parse-agent-logs) is exiting non-zero. Recommend instrumenting that step to emit its exit code/stderr tail into a structured field future audits can read.

Good news: 5 chronic issues resolved

All previously chronic (multi-week) failure clusters from repo-memory show 100% success in this window — moved to RESOLVED with a watch flag in case of relapse:

• Avenger (avenger-err-config-no-structured-logs) — 11/11 success, no ERR_CONFIG signature. Was chronic since 06-13, recurrence count 18.
• Daily Issues Report Generator (chroot-node-not-available) — 1/1 success. Was chronic since 06-12, recurrence count 11.
• Daily Cache Strategy Analyzer (codex-gh-aw-binary-not-found-for-mcp) — 1/1 success, all 7 Codex-engine runs fleet-wide succeeded. Was chronic since 06-15.
• Auto-Triage Issues (pi-gpt54-0tok-agentjob-fail) — 3/3 success.
• Code Scanning Fixer (code-scanning-fixer-0tok-agentjob-fail) — 3/3 success.

Unconfirmed: smoke-ci-copilot-cli-100pct-fail-on-push (previously the single dominant failure cause, 31% of all fails on 07-06) had zero runs this window — no push-to-main events fired it, so we can't confirm whether it's fixed or just quiet.

Cost & efficiency observations

Codex token efficiency vs. cost

Codex-engine runs average 682K tokens/run, ~40x Copilot's 16.7K average — but Codex's average AIC cost (13.6) is actually lower than Copilot's (37.9), consistent with cheaper per-token pricing. Claude Code has the highest average AIC (111.6) despite modest token counts, consistent with premium per-request pricing. Not an issue, just useful context for budget planning.

Single-run cost outlier

Daily VulnHunter Scan (run 30736015491, Claude Code) cost 992 AIC — 3x the next-highest run and ~7% of the entire fleet's 24h AIC total. Duration (15.2m) and tokens (66K) were unremarkable, so this reads as premium-pricing overhead rather than a runaway loop. Flagged as a watch-list anomaly, not a bug.

Recommendations (added to repo-memory)

1. HIGH — Instrument the post-agent driver/wrapper step to expose exit code + stderr on failure, so the dominant "ErrorCount=0 but run fails" cluster can finally be root-caused instead of re-observed every window.
2. MEDIUM — Investigate why this audit workflow itself went dark for 27 days; a month-long blind spot let 5 chronic issues go unwatched (they happened to resolve, but we had no visibility either way).
3. LOW — Merge copilot-sdk-driver-failures and claude-agent-job-fail-longrun into one cross-engine known-issue next cycle; they're the same signature and now span Copilot, Claude, and Pi.

Repo-memory (known-issues.json, recommendations.json, anomalies.json, metrics-summary.json, audit-history.jsonl) updated with today's findings and the 5 resolved-issue reconciliations.

Generated by 🔍 Agentic Workflow Audit Agent · agent · 280.4 AIC · ⌖ 38 AIC · ⊞ 8.3K · ◷

• expires on Aug 3, 2026, 1:53 PM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-08-03T21:53:28.626Z.

Closed by Workflow