[audit-workflows] Agentic Workflow Audit — 2026-05-23 (last 24h) — Success rate collapsed to 41.6% on two new harness/CLI bugs

[github-actions[bot]]

Overview

Daily audit of 77 runs across the last 24 hours surfaced two new critical issues that together drove the success rate from 75.6% (yesterday) to 41.6% — the worst of the past week. The agents themselves were largely fine; 24 of 43 failures are a pure harness bug where copilot-harness can't write $GITHUB_OUTPUT after the agent succeeds, and 14 more are caused by an anthropic-beta header that the CAPI/proxy started rejecting.

Summary

Metric	Today (05-23)	Yesterday (05-21)	Delta
Runs	77	41	+88%
Success rate	41.6% ⚠️	75.6%	−34.0 pp
Failures	43	4	+975%
Cancellations	2	1	+1
Tokens	46.2M	35.9M	+29%
Cost (USD)	$24.61	$19.96	+23%
Turns	973	620	+57%
Errors logged	51	7	+629%
Firewall blocks	594 / 3933 (15%)	385 / 2108 (18%)	—
Engines (copilot / claude / codex)	57 / 14 / 4	24 / 9 / 6	—

Critical issues (new today)

1. Harness ENOENT on $GITHUB_OUTPUT — 24 false-failures

Symptom: unexpected error: ENOENT: no such file or directory, open '/home/runner/work/_temp/_runner_file_commands/set_output_<uuid>'

The copilot-harness reports the agent succeeded (success on attempt 1, exitCode=0), then tries to write step outputs and fails because the runner's _runner_file_commands directory is not visible from inside the AWF chroot. The job then exits 1 and the workflow is marked failed — even though the agent did its job.

• 37 runs hit this error in agent-stdio.log.
• 24 runs had a clean agent success before the ENOENT (pure harness bug).
• 9 runs were compound (agent already failed for other reasons, e.g. the anthropic-beta header below).
• 2 runs somehow recovered to success.

Workflows most affected (pure-harness count): PR Sous Chef (6), PR Code Quality Reviewer (5), Matt Pocock Skills Reviewer (4), Test Quality Sentinel (4), Smoke CI (3), Auto-Triage Issues, Chaos PR Bundle Fuzzer, Issue Monster, Linter Miner, Daily Secrets, GEO Optimizer, PR Triage, PR Description Updater, Contribution Check, Daily Testify, Daily Safe Output Integrator, Daily Model Inventory.

2. Copilot CLI 1.0.48 sends a rejected anthropic-beta: context-1m-2025-08-07 header — 14 failures

Symptom: CAPIError: 400 Unexpected value(s) 'context-1m-2025-08-07' for the 'anthropic-beta' header. Please consult our documentation at docs.anthropic.com or try again without the header.

Copilot CLI v1.0.48 (copilot/pkg/linux-x64/1.0.48/app.js) attaches the Claude 1M-context beta header on every model call. CAPI/proxy returns 400. The CLI retries twice and then gives up; the harness retries up to 4 times.

• 17 runs observed the error; 14 failed, 3 succeeded after retry.
• Workflows affected: Chaos PR Bundle Fuzzer, PR Code Quality Reviewer (×4), PR Description Updater (×2), Auto-Triage Issues, Daily Model Inventory, Linter Miner, PR Triage, Contribution Check, plus Daily Team Evolution Insights and the new Contribution Check that recovered.

Recurring issues

Codex engine still failing — model gpt-5.5 invalid (3 of 4 today)

Yesterday's planned fix did not land. AI Moderator (×2) and Daily Cache Strategy Analyzer all returned 400 Bad Request for model=gpt-5.5 via (172.30.0.30/redacted) (the AWF api-proxy). Same ModelProviderInfo { name: "OpenAI AWF proxy" }` configuration as 05-21.

Smoke CI cancellations — 3 of 11 today

Runs 26341182074, 26340502327, 26338770313 all cancelled around 2m wall. This is the 5th consecutive day with this pattern.

PR Sous Chef execution drift

Turns vary 6 to 52 across 10 runs (avg 20.8). Today's headline failure rate is 60% but all 6 failures are ENOENT — the prompt drift remains a slow-burn issue but is not today's main cause.

Firewall — 594 blocked of 3933 (15%) requests

Nearly all blocks (593) go to the (unknown) bucket — almost certainly raw-IP egress or pre-DNS calls. Auto-Triage Issues had the highest workflow-level block rate at 32% (10/31).

New low-severity findings

• Daily News (copilot): fails exit 127 because node is not on the AWF chroot PATH. Need to bind-mount the setup-node install directory.
• Design Decision Gate (claude): hit max_turns=20 with permission denials on git commit (ADR file) and git fetch (PR head branch). $1.35 burned per attempt. Either grant those bash patterns, raise turn cap, or move the ADR commit to a post-step.
• Deployment Incident Monitor: both deployment_status runs cancelled at 1.0s — likely a pre_activation skip-if rule rejecting them.

Top-cost runs

Workflow	Cost	Tokens	Duration	Result
Daily Safe Output Tool Optimizer	$5.55	7.5M	18m	✅
Daily Code Metrics and Trend Tracking	$3.93	4.0M	16m	✅
[aw] Failure Investigator (6h)	$3.66	4.4M	16m	✅
Documentation Unbloat	$3.31	3.8M	12m	✅
Daily Team Evolution Insights	$1.90	1.1M	8m	✅
Design Decision Gate (failed run)	$1.35	1.3M	5m	❌ max_turns

Trend charts

Workflow Health (7-day)

Today's success rate (41.6%) is a sharp cliff after a multi-day slide (92.8 → 88.9 → 85.9 → 75.6 → 41.6). The two new bugs above are the proximate cause; the underlying decline trend is older, mostly Codex.

Token Usage & Cost (7-day)

Tokens (46.2M) and cost ($24.61) are up ~25% from yesterday because the workflow volume nearly doubled (77 vs 41 runs). Per-run cost held steady at ~$0.32. Daily Safe Output Tool Optimizer fell from $8.54 → $5.55 — recovery from yesterday's anomaly.

Recommendations (priority order)

1. 🚨 Fix harness ENOENT on $GITHUB_OUTPUT — bind-mount /home/runner/work/_temp/_runner_file_commands into the AWF chroot, or have copilot-harness emit step outputs via a different channel. This single fix would recover ~24 false failures/day.
2. 🚨 Strip / rewrite anthropic-beta: context-1m-2025-08-07 — at the api-proxy, or downgrade copilot CLI to a version without the beta header, or extend CAPI allow-list. Recovers ~14 failures/day.
3. 🔥 Pin codex to a real model id — gpt-5.5 does not exist on the proxy. Yesterday's planned fix did not land. Affects 3-4 codex workflows/day.
4. Daily News: bind-mount Node.js into AWF chroot (or add a pre-step that resolves node from /opt/hostedtoolcache/node).
5. Design Decision Gate: grant git commit / git fetch bash patterns or move ADR write to a deterministic post-step.
6. Smoke CI: raise agent-job timeout or emit noop earlier (5th day of cancellations).
7. Investigate Deployment Incident Monitor pre_activation skip rule.
8. Firewall (unknown) bucket: surface the host:port of the 593 raw-IP blocks so we can decide allow/deny.

References

• Run 26343171219 — Chaos PR Bundle Fuzzer (anthropic-beta example)
• Run 26341139002 — PR Sous Chef (pure ENOENT example)
• Run 26340543257 — Daily Cache Strategy Analyzer (codex gpt-5.5)
• Run 26343965053 — this audit run

Generated by 🔍 Agentic Workflow Audit Agent · ● opu47 25.9M · ◷

• expires on May 24, 2026, 9:45 PM UTC

[github-actions[bot]]

💥 KAPOW! 🦸 The Smoke Test Claude Agent has WHOOSHED through this discussion! 🌪️

ZZZAP! All systems nominal — Run 26349624985 is GO! 🚀✨

The agent leaps away in a single bound! 💨

Warning

Firewall blocked 6 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• contentautofill.googleapis.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "contentautofill.googleapis.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

💥 [THE END] — Illustrated by Smoke Claude · ● opu47 6.5M · ◷

[github-actions[bot]]

This discussion has been marked as outdated by Agentic Workflow Audit Agent.

A newer discussion is available at Discussion #34537.