[audit-workflows] Daily Agentic Workflow Audit — 2026-08-01 (92.5% success, 26-day audit gap flagged)

[github-actions[bot]]

Overview

First audit since 2026-07-06 — a 26-day gap in audit history that this report flags as its own anomaly (schedule likely stalled; see Anomalies below). Window: 2026-07-31T21:30Z → 2026-08-01T21:11Z (~23.7h), 240 runs across 6 engines.

Key metrics

Metric	Value
Total runs	240
Success / Failure	222 / 18
Raw success rate	92.5%
Success rate (excl. 2 intentional-failure tests)	93.28%
AI credits (AIC)	9,405.5
Tokens	18,450,594
Missing tools / data / MCP failures	0 / 0 / 0

Engine mix: copilot 145, pi 42, claude 36, codex 11, antigravity 3, gemini 3, null 1 (engine classified via aw_info.json engine_id, not .lock.yml scanning).

All 18 real failures resolve to two already-known recurring issues plus one novel signature — no unattributed failures this window.

Recurring known issues (confirmed again today)

copilot-sdk-driver-failures (recurrence #25, open since 2026-06-02) — 8 runs failed at "Execute GitHub Copilot CLI" 0-turn in the agent job. PR Code Quality Reviewer is worst-hit at 10/15 = 66.7% success (e.g. §30689984827, §30715678987), plus single fails on daily-experiment-report, Discussion Task Miner, and Tidy. This remains the #1 unresolved failure signature fleet-wide; root cause is still opaque because the CLI step emits no stderr/exit-code diagnostics.

safe-output-partial-failure-intolerance (recurrence #9, open since ~2026-05) — 7 runs where the agent job succeeded but the safe_outputs job failed at "Process Safe Outputs," reddening an otherwise-successful run. Smoke Copilot was fully red (0/3) (e.g. §30716152552); also hit Smoke Codex, PR Sous Chef, Code Scanning Fixer. Confirmed not a GitHub API rate-limit issue (core_remaining stayed 11.2K–15.0K of 15K throughout). Smallest actionable fix: make the safe_outputs job tolerate partial item failures instead of reddening the whole job for one bad item.

Novel signature — Contribution Check evals-install failure (1 run, watch only)

§30693860813 — Contribution Check's evals job (distinct from the agent job) failed at "Install GitHub Copilot CLI." This is a CLI-installation failure, not the familiar Execute-CLI-crash-after-install signature. Only one instance observed — logged as a new known issue to watch for recurrence, not yet treated as a cluster.

Positive signals

Three previously-chronic issues showed zero recurrence today: the AOAI apikey/Entra transient-error loop (6/6 success), the chroot-node-not-available issue on Daily Issues Report Generator (1/1 success), and the codex-gh-aw-binary-not-found issue on Daily Cache Strategy Analyzer (1/1 success). Only one clean day of evidence after the audit gap, so none are marked RESOLVED — worth confirming at the next audit.

Anomalies

• Smoke CI silence: previously the single largest chronic failure driver (100% fail on every push to main, 31% of all fleet fails as of 07-06), Smoke CI had zero runs in today's entire window. Its workflow definition is still compiled/active with push+PR+schedule triggers per agenticworkflows status, so this isn't a removed workflow — it's ambiguous whether the underlying issue was fixed and no qualifying push fired, or whether the trigger itself stopped. Flagged as a watch item, not a confirmed fix.
• PR Code Quality Reviewer degraded to 66.7% success today (10/15), consistent with its history as the worst-hit workflow for copilot-sdk-driver-failures.
• 26-day audit-history gap (07-07 → 07-31): trend charts below render this audit's data point as an isolated marker rather than connecting it with a line across the gap, since no data exists to support a continuous trend claim.

Recommendations filed this audit

1. HIGH — Instrument "Execute GitHub Copilot CLI" to emit stderr/exit code into run artifacts (copilot-sdk-driver-failures, unfixed 60+ days).
2. HIGH — Make the safe_outputs job tolerate partial item failures instead of failing the whole job on one bad item (safe-output-partial-failure-intolerance).
3. MEDIUM — Investigate why the daily audit schedule produced no runs for 26 days; restore cadence.
4. MEDIUM — Watch Smoke CI: confirm at next audit whether it resumes running and whether it passes.
5. LOW — Watch Contribution Check evals-job CLI-install failures for recurrence before escalating.

Trend charts

Workflow Health — Runs & Success Rate

Success/failure counts and rate across ~7 weeks of audit history, with the 07-07..07-31 gap left unbridged. Today's 92.5% sits comfortably above the 90% reference line and close to the pre-gap baseline (86.25% on 07-06), suggesting no major regression occurred during the unaudited period — though that can't be confirmed without daily data. The isolated marker on 08-01 (rather than a connecting line from 07-06) is intentional: it reflects that this is a single new observation after a long silent stretch, not a continuous trend.

Token Usage — Daily & Moving Average

Daily token totals plotted against a 3-audit moving average computed only over days with observed data (many days in this history have null token counts — a known, separately-tracked reporting gap). Today's 18.5M tokens is on the higher end of the observed range, but with so few surrounding data points the moving average should be read cautiously rather than as a firm trend.

Next actions

• Confirm Smoke CI's status (running-and-passing vs. still silent) at the next audit before drawing conclusions.
• Investigate the audit-schedule gap so daily cadence resumes.
• Prioritize stderr/exit-code instrumentation on the Execute-CLI step — it remains the highest-value fix on the board after 60+ days unresolved.

References:

• §30689984827 — PR Code Quality Reviewer, copilot-sdk-driver-failures
• §30716152552 — Smoke Copilot, safe-output-partial-failure-intolerance
• §30693860813 — Contribution Check, novel evals-install failure

Generated by 🔍 Agentic Workflow Audit Agent · agent · 362.1 AIC · ⌖ 35.1 AIC · ⊞ 8.3K · ◷

• expires on Aug 2, 2026, 1:56 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Agentic Workflow Audit Agent.

A newer discussion is available at Discussion #49857.