[audit-workflows] 🔍 Agentic Workflow Audit — 2026-07-10 (partial 5.6h · prod 92.9% healthy)

[github-actions[bot]]

🔍 Agentic Workflow Audit — 2026-07-10

Window: partial ~5.6h (15:39–21:15Z) · 104 complete run summaries analyzed · engine logs bridge timed out at 120s so ~170 downloaded run dirs were empty (chronic tooling limit, see below).

Health at a glance

Metric	Value
Overall success	85 / 104 = 81.7%
Production (ex-smoke)	78 / 84 = 92.9% ✅
Smoke harnesses	7 / 20 = 35% ⚠️ (noisy by design)
Agent-logic failures	0
Missing-tools / missing-data / MCP failures / noops / errors	0 / 0 / 0 / 0 / 0
All 19 failures	0-turn (pre-agent driver, agent never ran)
Tokens (via token_usage_summary)	7.25M · AIC $4,673.62 · 1,059 action-min

Bottom line: Production fleet is healthy (92.9% ex-smoke, zero agent-quality issues). The 81.7% overall is dragged down entirely by smoke/CI test harnesses and chronic 0-turn copilot-CLI startup failures — no new agent-behavior regressions.

Engine reliability

Engine	Rate	Note
claude	15/15 = 100%	clean
pi	16/17 = 94.1%	1 fail (PR Sous Chef @ Process-Safe-Outputs)
gemini / antigravity	1/1 each	clean
codex	4/5 = 80%	1 token_usage.jsonl assert fail
copilot	48/62 = 77.4%	Smoke CI + longrun 0-turn cluster
opencode	0/3 = 0%	🆕 new engine debut, all failed

---

📊 Trend charts

Overall success has oscillated in the 81–92% band over 30 days; today's 81.7% sits at the low end of the healthy range, consistent with prior partial evening windows (07-02/07-03 troughs, 07-04/07-05 rebounds). The dip is composition-driven — a high share of noisy smoke runs in this slice — not a production-quality regression, which held at 92.9%.

Reported daily tokens landed at 7.25M, the highest single-window figure in two weeks and above the 7-day moving average — but this reflects reporting coverage (token_usage_summary populated) as much as true volume, since run.TokenUsage remains empty. Token accounting is only sparsely comparable day-to-day because of the ongoing reporting gap (below).

---

🧩 Failure clusters (all 0-turn / pre-agent)

Recurring — known & tracked

• Smoke CI (copilot) — chronic, DOMINANT (count 7): 6 fails (5 push→main, 1 PR), 0-turn ~4–5m at Execute GitHub Copilot CLI. 100%-fail-on-push for 7 consecutive windows (06-30 → 07-10). Largest single red source; masks real regressions. smoke-ci-copilot-cli-100pct-fail-on-push
• copilot-sdk longrun Execute-CLI 0-turn (count ~29): 5 prod-main fails that ran full length then died 0-turn — Daily Security Observability Report (31m), Daily Project Performance Summary Generator (25m), Daily Formal Spec Verifier (19m), PR Triage Agent, The Daily Repository Chronicle. Wasted compute. copilot-sdk-driver-failures
• Smoke Copilot / Small / Sub Agents (copilot) x3: same copilot-CLI 0-turn family (smoke).
• PR Sous Chef (pi) x1: single 0-turn fail at Process Safe Outputs (post-agent) — isolated; watch. safe-output-partial-failure-intolerance

🆕 New this window

• opencode engine debut — 0/3 = 100% fail: Smoke OpenCode appears in the fleet for the first time and fails every run — 2 at Assert token_usage.jsonl is non-empty, 1 at Execute OpenCode CLI. Engine-integration immaturity. opencode-engine-onboarding-100pct-fail
• Empty token_usage.jsonl → hard smoke-assert fails (x3): Smoke Codex (1) + Smoke OpenCode (2) fail because these engines emit an empty token_usage.jsonl and the smoke test hard-asserts it non-empty. This is the first time the token-reporting gap causes hard test failures rather than just null metrics. token-usage-jsonl-empty-hard-assert

Persistent tooling limits (audit infra)

• logs bridge 120s timeout: the MCP logs fetch for -1d timed out at 120s again; only 104 of ~275 run dirs had complete summaries. Recurring across recent audits — audit coverage is capped to whatever downloads before the timeout.
• run.TokenUsage empty: the authoritative per-run token field is empty for all 104 runs; only token_usage_summary carries data. Ongoing token-usage-reporting-gap, now escalated to breaking smoke tests.

---

✅ Recommendations

1. [P1] Fix or quarantine Smoke CI copilot push failures — 7 straight windows of 100%-fail-on-push. Fix the copilot-CLI startup path or mark the job non-blocking so it stops masking real regressions on main.
2. [P2] Wire (or gracefully skip) token_usage.jsonl for codex/opencode — either emit real usage or downgrade the "non-empty" assert to a warning for engines that can't report per-turn tokens, so smoke tests reflect real health.
3. [P2] Track opencode onboarding — expected teething for a new engine; monitor whether 0/3 stabilizes as integration matures before treating it as a regression.
4. [Infra] Audit logs coverage — the 120s bridge timeout keeps truncating the dataset. Consider count-limited or before_run_id paginated fetches to guarantee complete coverage per audit.

References:

• §29124847022 (this audit)
• §29114468147 (representative healthy claude run)

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by 🔍 Agentic Workflow Audit Agent · 198.4 AIC · ⌖ 36.1 AIC · ⊞ 7.4K · ◷

• expires on Jul 11, 2026, 1:50 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Agentic Workflow Audit Agent.

A newer discussion is available at Discussion #45015.