[audit-workflows] Agentic Workflow Audit — 2026-07-03: pi engine collapse (PR Sous Chef 80% fail, 0-turn)

[github-actions[bot]]

🔎 Agentic Workflow Audit — 2026-07-03

Window: ~5.4h evening cluster (15:51–21:16Z). Partial as usual — the logs MCP bridge hit its 60s cap; 47 of 118 run dirs completed full download (70 download-incomplete). Numbers are biased toward the active failure cluster and undercount total volume.

Metric	Value
Runs analyzed	47 (32 ✅ / 15 ❌) — 68.1%
Prod-main	27/41 = 65.9% (ex-2-clusters 86.2%, near baseline)
Non-main	5/6 = 83.3%
AI credits (AIC)	$3,539
Action minutes	539 (130 wasted on failures)
Tokens (in+out)	1,078,019 — recovered via token_usage_summary
missing-tools / missing-data / mcp-failures	0 / 0 / 0

🚨 Headline: pi engine collapsed to 30.8%

The pi engine crashed from 100% (06-30/07-01) → 30.8% (4/13) today, driven almost entirely by a sustained PR Sous Chef incident. This is the escalation of the NEW pi-0turn signal first seen 07-02 (then just 1/2).

• PR Sous Chef (pi / copilot/gpt-5.4): 8/10 = 80% fail. The 2 successes were early; the following 8 runs failed consecutively — a sustained mid-window incident, not scatter. All are 0-turn / 0-tok failures at the agent step (agent never produced output).
• Because pi runs the copilot/gpt-5.4 backend, this is the chronic copilot-sdk-driver-failures family now surfacing dominantly through the pi engine.

Every one of the 15 failures was a 0-turn pre-agent driver failure — no agent logic ran, no missing tools, no MCP errors.

All 15 failures (all 0-turn / 0-tok)

Workflow	Engine	Model	Event	Count	Class / Known issue
PR Sous Chef	pi	copilot/gpt-5.4	schedule/main	8 (8/10)	pi-0turn-copilot-backed → copilot-sdk-driver-failures (ESCALATING)
Smoke CI	copilot	claude-sonnet-4.6	push/main	2 (2/2=100%)	smoke-ci-copilot-cli-100pct-fail-on-push (chronic)
Documentation Unbloat	pi	copilot/gpt-5.4	schedule/main	1	doc-unbloat-empty-output — now on pi engine
Daily Formal Spec Verifier	copilot	claude-sonnet-4.6	schedule/main	1	copilot Execute-CLI 0-turn
Daily Safe Output Integrator	copilot	claude-sonnet-4.6	schedule/main	1	copilot Execute-CLI 0-turn
PR Code Quality Reviewer	copilot	claude-sonnet-4.6	pull_request	1	copilot Execute-CLI 0-turn
Daily Cache Strategy Analyzer	codex	—	schedule/main	1	codex-gh-aw-binary-not-found-for-mcp (chronic, unfixed since 06-15)

Engine breakdown

Engine	Runs	Success	Rate
claude	8	8	100% ✅
copilot	25	20	80.0%
pi	13	4	30.8% 🔴
codex	1	0	0%

claude was fully healthy. The fleet-wide degradation is concentrated in copilot-backed drivers (direct copilot and pi→copilot/gpt-5.4).

📈 Trend Charts

30-day health shows the fleet oscillating around a ~85–93% baseline on full windows, with sharp dips on partial evening-only windows (06-30, 07-02, 07-03) that over-sample the active failure cluster. Today's 68.1% is a partial-window artifact layered on a real pi/PR-Sous-Chef incident — prod-main ex-clusters (86.2%) sits near baseline.

AI-credit usage tracks window size, peaking on full-day windows (06-24/06-25 ≈ $24–27k) and low on today's partial slice ($3.5k). Raw token counts remain empty in metrics.TokenUsage fleet-wide (since ~06-19), so AIC from token_usage_summary is used as the consistent cost proxy — note: this run confirmed real token data (1.08M) is still present in token_usage_summary, only the metrics.TokenUsage artifact is zeroed.

✅ Healthy signals

• claude engine 8/8 = 100%.
• Zero missing-tools, missing-data, or MCP failures fleet-wide.
• Prod-main excluding the two known clusters = 86.2% (on baseline).

🎯 Recommended actions

1. PR Sous Chef (HIGH): investigate the sustained 0-turn/0-tok incident. As mitigation, consider pinning it off the copilot/gpt-5.4 backend (non-copilot engine/model). Root cause is the copilot-backed driver failing before the first turn.
2. Driver observability: capture and surface driver stderr on 0-turn exits to distinguish auth vs binary-missing vs startup — today 15/15 fails are indistinguishable "agent: failure, 0 turns".
3. Alerting: flag any workflow with ≥N consecutive 0-turn failures (PR Sous Chef went 8-in-a-row unnoticed within the window).
4. Chronic, still unfixed: Smoke CI push/main 100% fail, Daily Cache Strategy Analyzer (codex binary, since 06-15).

Notes & caveats

• Repo memory updated: audit-history.jsonl (full entry) + metrics-summary.json pushed. Non-essential memory files (anomalies/recommendations/known-issues) were reverted this run to stay under the 50KB patch limit — full failure detail, known-issue cross-refs, and the pi-collapse escalation are all captured inside the audit-history entry.
• Duration/branch data read from run.headBranch / run.Duration; engine from aw_info.json.engine_id (authoritative).

References:

• §28674765048 — PR Sous Chef pi 0-turn fail
• §28670963087 — Smoke CI push/main fail
• §28678217943 — Daily Cache Strategy Analyzer codex fail

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by 🔍 Agentic Workflow Audit Agent · 252.9 AIC · ⌖ 35 AIC · ⊞ 7.3K · ◷

• expires on Jul 4, 2026, 1:46 PM UTC-08:00