[audit-workflows] 🔍 Agentic Workflow Audit — 2026-07-14 (partial; tokens restored, Smoke CI chronic, new crush engine)

[github-actions[bot]]

🔍 Agentic Workflow Audit — 2026-07-14

Window: partial ~8.5h (12:45–21:14Z) · 8-day gap since last audit (07-06). Bridge logs call hit the 120s cap mid-download, so 105 of ~340 run dirs had full summaries (234 empty). Numbers below are a representative partial slice.

Metric	Value
Runs (with data)	105
Success / Failure	86 / 19
Overall rate	81.9%
Prod-main rate	83.1% (86.2% ex-SmokeCI+CreditTest)
Non-main rate	77.3%
AI Credits	5,708.6
Tokens	73.3M ✅ restored
Missing tools / data / MCP	0 / 0 / 0
Agent-logic errors	0

Headline: All 19 failures are 0-turn driver failures (agent never produced a turn) — zero agent-logic, missing-tool, missing-data, or MCP failures. Health is on-baseline once the two chronic clusters are excluded. pi (24/24) and claude (13/13) were 100%. Avenger — the chronic #1 offender — had 0 runs (paused again).

---

📊 Trends

Success rate sits at 81.9% (partial slice), below the ~90% healthy band but consistent with prior partial windows once the deterministic Smoke CI cluster is removed. Failures remain concentrated in the copilot 0-turn family rather than spread across engines.

Token reporting is restored: 73.3M this window — the highest since mid-June. The multi-week "token-gap" (06-19→07-13) was a reporting-path artifact: run.TokenUsage/metrics.TokenUsage are still 0, but real per-run tokens live in agent_usage.json (input+output+cache). Recommend the token chart officially read from that field.

---

🧩 Failure taxonomy (all 0-turn)

Full breakdown (19 fails)

1. copilot-sdk-longrun-0turn — ~9 fails (DOMINANT, recur count 25)
Agent does real work for 9–33 min, then Execute GitHub Copilot CLI aborts 0-turn:

• Daily Security Observability Report 32.8m (new name)
• Daily Project Performance Summary Generator 20.3m
• The Daily Repository Chronicle 16.6m (new)
• Daily Formal Spec Verifier 16.4m
• Discussion Task Miner – Code Quality 15.2m (new)
• PR Triage Agent 14.6m
• CI Optimization Coach 12.8m (new)
• Daily Issues Report Generator 10.3m (was chroot-node; now longrun signature)
• Daily SPDD Spec Planner 9.4m + Smoke Copilot Sub Agents 6.2m

2. Smoke CI 3/3 = 100% (chronic, recur count 7)
copilot/claude-sonnet-4.6, 0-turn 4.6–5.0m at Execute Copilot CLI, push→main + PR. Deterministic red on main across 7 observed windows (06-30→07-14). Muting/fixing lifts prod-main 83.1%→~86%.

3. Smoke Crush 4/4 = 100% — 🆕 NEW crush engine (isolated)
anthropic/claude-sonnet-4, 0-turn ~6m, all on branch copilot/aw-fix-smoke-crush-failure via workflow_dispatch. A new engine entered the fleet; failures are confined to an active fix branch → PR-under-review signal, not fleet infra. Flag to branch author; watch for crush reaching prod-main.

4. Daily Cache Strategy Analyzer (codex) — recur count 6
0-turn 6.2m at Execute Codex CLI (gh-aw binary-not-found for MCP). ~100% fail for ~1 month (06-15→07-14).

5. Daily Credit Limit Test — intentional.

---

✅ Recommendations (evidence-linked)

1. [HIGH · count 7] Fix/mute Smoke CI copilot-CLI 0-turn on main — single highest-ROI deterministic red; persisted unchanged across the 8-day gap → not self-healing. Add a fast-fail diagnostic dumping why the CLI aborts 0-turn.
2. [HIGH · count 25] Root-cause the copilot Execute-CLI 0-turn family — ~68% of fleet fails; 4 new affected workflows this window. Instrument stderr/exit-code so 0-turn stops being opaque.
3. [MEDIUM · count 6] Fix or disable Daily Cache Strategy Analyzer (codex) — ~1 month of ~100% fail; burns ~6min/day with zero output. Align the MCP-helper binary lookup path with the build-step placement.
4. [LOW · RESOLVED] Point token chart at agent_usage.json — real per-run tokens are present there (73.3M fleet); run.TokenUsage remains 0. Close the "token-gap" by reading the correct field.
5. [WATCH] New crush engine — currently isolated to its fix branch; confirm crush startup before it reaches main/schedule.

References: §29337445427 · §29369632935

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by 🔍 Agentic Workflow Audit Agent · 217.8 AIC · ⌖ 36 AIC · ⊞ 7.4K · ◷

• expires on Jul 15, 2026, 1:45 PM UTC-08:00

[github-actions[bot]]

Smoke tests executed by Copilot. Results forthcoming.

📰 BREAKING: Report filed by Smoke Copilot - AOAI (Entra) · 19.9 AIC · ⌖ 3.04 AIC · ⊞ 17.6K · ◷
_{Comment /smoke-copilot-aoai-entra to run again
Add label smoke to run again}

[github-actions[bot]]

Smoke bot here. Fire good. Run 29372693679 stomp by.

📰 BREAKING: Report filed by Smoke Copilot · 139.4 AIC · ⌖ 4.22 AIC · ⊞ 19K · ◷
_{Comment /smoke-copilot to run again
Add label smoke to run again}