[audit-workflows] Daily Workflow Audit 2026-06-08 — 🔴 Active copilot PAT-400 incident (88%→58%)

[github-actions[bot]]

Overview

🔴 ACTIVE PRODUCTION INCIDENT. A new copilot auth-token failure (PAT-not-supported-400) began at 2026-06-08T19:40Z and was still firing in the most recent runs (21:37Z). It collapsed fleet success from a healthy 88.0% (pre-incident) to 57.9% (post-incident). ~15 runs failed identically, including production-main schedules. An in-flight fix branch — pelikhan/fix-pat-400-retry — was observed, so the team is already aware.

Window: 2026-06-08T18:21–21:40Z (~3.3h evening cluster, partial — the logs MCP tool timed out at the 120s cap; 93 of 260 run dirs had full summaries). EffectiveTokens came back 0/unpopulated for every run this window, so effective-token cap proximity could not be assessed.

Summary

Metric	Value
Runs (with summaries)	93 (88 completed, 5 in-progress)
Success / Failure	66 / 22 — 75.0% completed
Pre-incident (<19:40Z)	44/50 = 88.0% ✅
Post-incident (≥19:40Z)	22/38 = 57.9% 🔴
Prod-main	30/40 = 75.0%
Tokens	24.37M
Cost (claude-measured)	$16.83
Errors / Missing-tools / Missing-data / MCP-failures	0 / 1 / 0 / 0
Firewall blocked	192 / 4162 = 4.6% (very light)
Engines	copilot 69, claude 16, codex 5, pi/gemini/antigravity 1 each

🔴 Critical: copilot PAT-not-supported-400 (NEW, active)

Both the copilot-harness and copilot-sdk-driver token-check paths POST a Personal Access Token to an endpoint that now rejects it:

400 checking third-party user token: bad request: Personal Access Tokens are not supported for this endpoint

The run exits 1 with turns=0 (agent never starts). Onset 19:40:19Z, continuous through the latest observed run (21:37Z). Affected prod-main schedules: Daily Ambient Context Optimizer, Contribution Check, Daily Project Performance Summary Generator — note these reddened via PAT-400, not their usual classes (token-429 / safe-output). Also hit many PRs/smokes: Matt Pocock Skills Reviewer ×3, PR Code Quality Reviewer ×3, PR Description Updater ×2, Agent Container Smoke Test, Smoke Copilot ×2, Daily Max AI Credits Test.

Recommendation (CRITICAL): Land/expedite pelikhan/fix-pat-400-retry or roll back the token-source change for the copilot token-check path; confirm whether the endpoint or the credential changed. Add a harness smoke assertion that the token-check uses a credential type the endpoint accepts.

Trend Charts

The 30-day health view shows a sharp spike in failures (22, highest since the 05-23 outlier) and the success rate dropping to 75% — entirely attributable to the evening PAT-400 incident; the day was tracking at 88% before 19:40Z.

Token usage (24.4M) sits below the 7-day moving average and continues a two-week downward drift — note this is a partial 3.3h window plus many turns=0 failed runs that consumed zero tokens, so true daily usage is understated.

Other failure classes (7 runs)

• cli-proxy DIFC liveness-probe-failed (NEW, infra, ×2): awf-cli-proxy container errored at startup — DIFC proxy liveness probe to localhost:18443 refused → agent never starts (turns=0). Workflows: PR Description Updater (27158226210), Issue Monster (27167452907). Both pre-incident (18:xx) — looks like a separate infra flake. Rec: bounded retry/health-wait on the probe + surface proxy startup logs.
• copilot-sdk session.idle-timeout-after-valid-output (RECUR, ×1): PR Code Quality Reviewer (27160990743) emitted a complete, valid review (findings JSON present) then hit Timeout 870000ms waiting for session.idle → exit1, discarding the work and reddening the job. Rec: treat "output already emitted + idle-timeout" as success-with-warning.
• copilot-sdk tool-permission-lockout (RECUR, ×1): Daily Safe Output Integrator (27161455868) — permission denied: read(compiler_safe_outputs_config_test.go).
• codex (×2): AI Moderator (27158930229, issue_comment, turns=0); Daily Cache Strategy Analyzer (27160677178) — recurring day 4 (plugin/remote-catalog auth warnings, gpt-5-mini).
• smoke probes (by-design/non-prod, ×3): Smoke Gemini (EACCES /tmp scandir), Smoke Copilot-AOAI ×2 (o4-mini-aw, transient_bad_request retries — possibly PAT-400-adjacent).

Positives & deltas vs prior days

• token-budget-429 ABSENT again (no effective-token cap breaches observed).
• model-param-config (Caveman effort-400) did NOT fail this window after 3 consecutive days.
• 0 errors, 0 missing-data, 0 MCP-failures; only 1 missing-tool.
• Firewall very light (4.6% blocked) — no firewall-caused failures.
• Pre-19:40Z the fleet was healthy at 88%; the incident is a clean, time-bounded regression rather than broad degradation.

Top cost & data-quality notes

Top cost (claude-measured): [aw] Failure Investigator (6h) $3.21 · Daily Code Metrics and Trend Tracking Agent $2.12 · Daily Safe Output Tool Optimizer $2.09 — all successful.

Data quality: (1) logs tool timed out at 120s again — only newest ~93/260 runs had full summaries (~18h likely unobserved). (2) EffectiveTokens field was 0 for all runs this window. (3) Cost is claude-only; copilot/codex report $0.

Recommendations

1. 🔴 CRITICAL — Expedite pelikhan/fix-pat-400-retry (or roll back the token change) for the copilot token-check path; add a smoke assertion on the token/credential type.
2. MEDIUM — Add retry/health-wait to the awf-cli-proxy DIFC liveness probe + surface its startup logs.
3. MEDIUM — sdk-driver: flush safe-outputs and mark success-with-warning when a terminal agent message precedes a session.idle timeout.

References: §27164602401 (Ambient/PAT-400) · §27160990743 (sdk idle-timeout) · §27158226210 (cli-proxy)

Generated by 🔍 Agentic Workflow Audit Agent · 308.4 AIC · ⌖ 32 AIC · ⊞ 8.2K · ◷

• expires on Jun 9, 2026, 1:57 PM UTC-08:00

[github-actions[bot]]

Smoke cave bot here. Me poke tools. Fire still warm.

Warning

Firewall blocked 6 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• contentautofill.googleapis.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "contentautofill.googleapis.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

📰 BREAKING: Report filed by Smoke Copilot · 232.7 AIC · ⌖ 16.1 AIC · ◷

[github-actions[bot]]

This discussion has been marked as outdated by Agentic Workflow Audit Agent.

A newer discussion is available at Discussion #38221.