[audit-workflows] 🔍 Agentic Workflow Audit — 2026-06-27 (90.9% overall; Avenger 10/10 fail is the whole prod-main gap)

[github-actions[bot]]

🔍 Agentic Workflow Audit — 2026-06-27

Window: 2026-06-26 21:52Z → 2026-06-27 21:24Z (~23.5h, 320 runs, 5 stitched log batches)
Note: the 06-26 audit cycle was missed/unrecorded; this window bridges it.

Metric	Value	vs 06-25
Overall success	90.9% (291/320)	🟢 +5.6pt (was 85.3%)
Prod-main success	86.3% (165/191)	🟢 +5.3pt (was 81.0%)
Prod-main ex-2-intentional	87.3%	🟢 +5.5pt
Prod-main ex-Avenger	91.2%	healthy baseline
Missing tools / data / MCP failures	0 / 0 / 0	✅ clean
Action-minutes	3,412	—
GitHub API calls	3,530	—
Tokens	unavailable	⚠️ artifact empty fleet-wide

Headline: Fleet recovered from the 06-25 distributed trough but sits below the 06-22/24 ~92% baseline — and the entire gap is Avenger. Avenger failed 10/10 (100%) this window; remove it and prod-main is a healthy 91.2%. Zero new failure classes, zero missing tools/data, no shared incident.

---

📊 Trend Charts

Success rate rebounded to 90.9% after the 06-25 dip (81% prod-main), tracking back toward the late-June ~92% band. The recovery is broad — the 06-25 degradation was an aggregation of independent agent-startup fails across engines, and most of those workflows are green again today. The remaining drag is concentrated, not distributed.

Token telemetry flatlined to zero fleet-wide since ~06-19 — the token_usage artifact ships empty (TokenUsage=0) for every run, so the chart shows no daily volume past 06-20 and the 7-day moving average decays to the last valid reading. This is an observability gap, not a usage change; cost is approximated via aic (≈23.3K this window). Restoring token emission is needed to resume cost/efficiency trending.

---

🚨 Dominant Issue — Avenger 100% failure (the whole prod-main gap)

Avenger: 10/10 hard-fail, 173 action-min/day wasted

• Signature: claude engine, model=agent, Turns=0, TokenUsage=0, ErrorCount=0, runs ~14–19 min then fails. Classic avenger-err-config-no-structured-logs (recur 16, open since ~06-08).
• Cadence: fires ~every 90 min on schedule and fails every single time — 10 consecutive failures in 24h.
• Impact: alone accounts for 38% of all 26 prod-main failures and 173 action-minutes/day of pure waste with zero successful output.
• History: went absent 06-22/23 (streak "broken" by pause, not fix), resumed 2/2 on 06-24, now fully back at 10/10. The pause is clearly lifted and the root cause was never touched.

Run refs: §28301247575, §28300001017

🔁 Recurring failure families (all known, none new)

Long-run 0-turn agent-job fails — now CROSS-ENGINE

The "runs 20–33 min of real compute, logs 0 turns, then the agent job reds" pattern continues — and this window confirms it spans both engines:

• copilot: Daily SPDD Spec Planner (26m), Daily Safe Output Integrator (25m, chronic), Daily Firewall Logs Collector (33m, 533 aic — largest single-fail compute)
• claude: Daily Rendering Scripts Verifier (25m), Daily yamllint Fixer (19m)

Same copilot-sdk-driver-failures long-run variant signature (recur 18) now mirrored on claude (claude-agent-job-fail-longrun). Fix direction unchanged: flush already-emitted safe-output on session-idle/terminal instead of reddening the long-run job.

Other recurring (each 1–2 occurrences, all previously catalogued)

Workflow	Engine	Class	Recur
PR Sous Chef ×2	copilot	safe-output-partial-failure-intolerance	7
Daily Cache Strategy Analyzer	codex	model-param-config (gpt-5-codex model)	5
Daily Issues Report Generator	copilot	chroot-node-not-available (likely)	8
Code Simplifier / Sub-Agent Model Resolution Audit	copilot/codex	misc 0-turn agent-job	—
Go Logger Enhancement / Design Decision Gate ×2 / Changeset Generator	claude/codex	short single-occurrence fails (PR/dispatch)	—

Intentional & config-test fails (expected, excluded from health)

• Intentional: Daily Credit Limit Test (Intentionally Broken), Daily Max AI Credits Test (Intentionally Fails)
• Config/smoke tests: Daily BYOK Ollama Test, GitHub Remote MCP Authentication Test

---

🏷️ Engine mix

copilot 210 · claude 69 · pi 26 · codex 10 · antigravity 2 · gemini 2 · null-metadata 1 (from aw_info.json engine_id)

🔧 Top compute consumers (all green except Avenger)

PR Code Quality Reviewer 348min (27 runs, 0 fail) · PR Sous Chef 290min (2 fail) · Matt Pocock Skills Reviewer 270min (0 fail) · Test Quality Sentinel 214min (0 fail) · Avenger 173min (10/10 fail) · Design Decision Gate 184min (2 fail).

---

✅ Recommended actions (priority order)

1. [P1 — TOP ROI] Fix or pause Avenger. It is the single largest reliability drain — 10/10 fail, 38% of prod-main fails, 173 action-min/day burned for zero output. Either fix the err-config/no-structured-logs root cause or disable the schedule until fixed. This one change lifts prod-main from 86.3% → 91.2%.
2. [HIGH] Long-run agent-job flush. Flush emitted safe-output on session-idle/terminal instead of reddening the job — now affecting copilot and claude long-runners (5 fails, ~128 compute-min).
3. [HIGH] Restore token telemetry. token_usage artifact empty fleet-wide since ~06-19 — blocks all cost/efficiency trending. Likely a single emission/upload regression.
4. [MED] Long-tail recurring fixes unchanged: PR Sous Chef safe-output partial-failure intolerance (recur 7), codex Cache Strategy model-config (recur 5), Daily Issues Report Gen chroot-node (recur 8).

References: §28301247575 · §28298659490 · §28277055171

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by 🔍 Agentic Workflow Audit Agent · 291.1 AIC · ⌖ 28.1 AIC · ⊞ 7.2K · ◷

• expires on Jun 28, 2026, 2:01 PM UTC-08:00

[github-actions[bot]]

Smoke poke. Query and comment path work. Run §28303379787.

Warning

Firewall blocked 6 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• contentautofill.googleapis.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "contentautofill.googleapis.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

📰 BREAKING: Report filed by Smoke Copilot · 506.2 AIC · ⌖ 16.4 AIC · ⊞ 19.8K · ◷

[github-actions[bot]]

Smoke test initiated for run 28303380682

Warning

Firewall blocked 6 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• contentautofill.googleapis.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "contentautofill.googleapis.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

📰 BREAKING: Report filed by Smoke Copilot - AOAI (Entra) · 235.9 AIC · ⌖ 6.32 AIC · ⊞ 17.9K · ◷

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-06-28T22:01:31.576Z.

Closed by Workflow