[audit-workflows] Workflow Audit — 2026-05-27 (partial day)

[github-actions[bot]]

Overview

Mid-day audit covering 37 runs from the last 24h (4 still in_progress including this run). Real completion rate is 30 succeeded / 33 completed = 90.9% — back to the typical band after yesterday's 85.7%. Three failures: one new critical (Smoke Claude Playwright FFmpeg download blocked by firewall), two recurring (the add_comment target=* pattern in Contribution Check, and Changeset Generator still on the broken gpt-5.3-codex model).

Good news: yesterday's Claude CLI step-timeout did not recur (0 in 9 Claude runs), awf-squid stayed quiet (0 in 22 copilot runs), and Smoke Copilot's upload_artifact failure also resolved. The PR Sous Chef turn-variance issue tightened from 0-217 → 10-48 turns with 3/3 runs succeeding.

Summary

• Runs: 37 (30 success, 3 failed, 4 in_progress)
• Success rate: 90.9% (excluding in_progress)
• Tokens: 17.3M raw / 152.1M effective
• Cost (est.): ~$12 partial — extrapolates toward typical $25-26 once Daily Safe Output Tool Optimizer completes
• Action minutes: 290 across 290 min of CI time
• Engines: Copilot 22, Claude 9, Codex 3, Antigravity 1, Gemini 1, Pi 1
• Firewall: 1682 requests, 419 blocked (24.9% — up from 16.4% yesterday, driven by Playwright CDN + Smoke Pi internal proxies)
• Errors: 3 / Missing tools: 0 / Missing data: 0

Critical Issues

🆕 Smoke Claude — Playwright FFmpeg download timeout (NEW)

Run 26537099387 — Step Install Playwright CLI skills ran playwright-cli install --skills, which began downloading FFmpeg from cdn.playwright.dev. The CDN resolved through three Azure edge hosts (playwright-akamai.azureedge.net, playwright-verizon.azureedge.net, playwright.azureedge.net) — none of which are in GH_AW_ALLOWED_DOMAINS — so the firewall blocked them and the step hit the 10-min GitHub Actions step timeout.

• Fix: Add the 3 Azure CDN hosts (or *.azureedge.net) to allowed_domains in Smoke Claude, OR pre-cache FFmpeg in the runner image.

🔁 Changeset Generator — still on gpt-5.3-codex (UNACTIONED RECOMMENDATION, DAY 7)

Run 26537099239 — model: gpt-5.3-codex in aw_info.json; CLI auto-routes to gpt-5-codex-alpha-2025-11-07 → 404 at proxy. All 5 retries failed. Other codex workflows (AI Moderator, Smoke Codex, Daily Cache Strategy Analyzer) all succeeded today on gpt-5.4.

• Fix: One-line frontmatter edit — set model: gpt-5.4 in .github/workflows/changeset-generator.md and re-compile. This recommendation has been outstanding for 7 days.

🔁 Contribution Check — add_comment target="*" without item_number (RECURRED, down from 15 → 2)

Run 26539586973 — 2 add_comment safe-outputs failed with Target is "*" but no item_number/issue_number/.... The yesterday-recommended prompt fix appears to have partially landed (15 occurrences → 2) but the pattern is not fully eliminated.

• Fix: Tighten validation at MCP boundary to reject add_comment with target="*" AND missing item_number — closes the gap permanently regardless of prompt drift.

Trend Charts

Workflow Health (8-day trend)

Success rate has stabilized in the 85-95% band since the May 23 cliff (41.6%, driven by the now-resolved harness-enoent-set-output + copilot-anthropic-beta-header bugs). Today's 90.9% is mid-pack and partial. Both new criticals today are isolated to single workflows — not a fleet-wide issue.

Token Usage & Cost (8-day trend)

Daily cost is plateauing around $23-26/day even as runs vary widely (38 → 174). The 3-day moving average is flat. Yesterday's $9.90 Daily Safe Output Tool Optimizer spike is pending verification — that workflow is still in_progress at audit time. If it lands at $9+ for a second day, escalate.

Detailed Findings & Observability Insights

Top runs by effective tokens

Workflow	Run	Eff. tokens	Turns
PR Code Quality Reviewer #2	26538033723	16.5M	30
Daily Project Performance Summary	26539409760	12.3M	24
Matt Pocock Skills Reviewer #2	26538033652	11.3M	21
Smoke Codex	26537099294	11.2M	—
Test Quality Sentinel #2	26538033651	10.8M	22

Longest runs by action minutes

Workflow	Run	Minutes	Notes
AI Moderator	26536024624	17	Codex/gpt-5.4 success
PR Sous Chef #3	26539523153	14	48 turns, success
Smoke Claude	26537099387	14	Failed — Playwright timeout
Contribution Check	26539586973	13	Failed — add_comment
Smoke Copilot	26537099471	13	Success (resolved upload-artifact bug)

Observability insights (auto-detected)

• Failure hotspot: Changeset Generator 100% fail (1/1) — same gpt-5.3-codex root cause
• Execution drift: PR Sous Chef variance 10-48 turns (improvement from 0-217 yesterday)
• Network friction hotspot: Smoke Pi 20/22 blocked (91%) on internal api-proxy:10000/10002 — workflow still succeeded; likely test-intentional but worth verifying
• Anomaly events: 6 high-anomaly events flagged across 37 runs (mostly new log templates / rare clusters)
• Actuation mix: 36/37 runs read-only; 1 wrote a safe-output

Firewall block composition (419 total)

Domain	Blocked
(unknown) — DNS-failure blanket blocks	299
localhost:8080 / www.google.com:443	21 each
content-autofill.googleapis.com:443	20
accounts.google.com:443	14
api-proxy:10002 (Smoke Pi)	12
api-proxy:10000 (Smoke Pi)	8
playwright-akamai/verizon/azureedge.net	3 (root cause of Smoke Claude fail)

Resolved since last audit

• claude-cli-step-timeout-10min — 0 occurrences across 9 Claude runs today
• awf-squid-unhealthy — 0 occurrences across 22 copilot runs today (still intermittent historically)
• smoke-copilot-upload-artifact-empty — Smoke Copilot succeeded today including the previously-broken test
• copilot CLI 1.0.52 — 4-day stability streak confirmed (no ENOENT, no anthropic-beta errors)

Recommendations (priority order)

1. Critical — Add Playwright Azure CDN hosts to Smoke Claude allowed_domains (or pre-cache FFmpeg)
2. Critical — Update Changeset Generator workflow: model: gpt-5.3-codex → model: gpt-5.4 (7-day backlog)
3. High — Add MCP-boundary validation: reject add_comment with target="*" + missing item_number
4. Medium — Verify Smoke Pi api-proxy:10000/10002 config: intentional test-block or stale hostnames?
5. Medium — Re-audit Daily Safe Output Tool Optimizer cost tomorrow (was $9.90, today in_progress)
6. Low — Pin copilot CLI 1.0.52 as minimum (day 4 stability)

References:

• Workflow runs (today)
• Smoke Claude failure run
• Changeset Generator failure run
• Contribution Check failure run

Generated by 🔍 Agentic Workflow Audit Agent · opus47 18M · ◷

• expires on May 28, 2026, 9:58 PM UTC

[github-actions[bot]]

This discussion has been marked as outdated by Agentic Workflow Audit Agent.

A newer discussion is available at Discussion #35583.