[audit-workflows] Agentic Workflow Audit — 2026-08-28 (github/gh-aw)

[github-actions[bot]]

Agentic Workflow Audit — github/gh-aw

Window: 2026-08-27 22:14 UTC → 2026-08-28 22:26 UTC (~24h, 463 runs, 5 paginated fetches)

⚠️ Audit gap notice: the last recorded audit in repo-memory is dated 2026-07-06 — this run picks up after a 53-day gap with no intervening history. Trend charts below show that gap explicitly rather than interpolating over it. Additionally, /tmp/gh-aw/repo-memory/default/ was mounted read-only in this run, so the findings below could not be persisted back to known-issues.json / metrics-summary.json / etc. — treat this report as the durable record until write access is restored.

Fleet snapshot (24h)

Metric	Value
Total runs	463 (1 still in-progress at cutoff)
Raw success rate	331/462 = 71.6%
Excl. intentional_failure runs	330/461 = 71.6% (only 1 run flagged intentional_failure=true this window — Daily Credit Limit Test, which succeeded, so the adjustment is negligible)
Technical failure rate (excl. gate/content-block failures — see Finding 2)	38/462 = 8.2% real infra/agent failures
Total tokens	31,558,297 (74% of runs reported token data; 119/463 still null)
Total AI credits (AIC)	14,534
Total action-minutes	10,111 (~168.5h)
GitHub API calls	6,714
Missing tools / missing data / MCP failures	0 / 0 / 0
Agent-level errors	40

Engine mix: copilot 188 (91.5% success), pi 71 (93.0%), codex 57 (86.0%), claude 44 (81.8%), goose 3, opencode 2, aider 2, crush 2, gemini 1. The remaining 93 "unknown-engine" runs are exactly the gate-bloc + AI Moderator failures in Finding 2 — they never got far enough to populate an engine.

📊 Trends

Success/failure counts and the success-rate line (right axis) across every day we have data for, June 13 – Aug 28. The shaded band marks the 53-day audit gap. Today's raw rate (71.6%) looks like a regression vs. the ~86–90% baseline seen in early July, but Finding 2 below shows most of that drop is a new class of fast, shared-precondition PR-gate failures rather than a broad reliability decline — the technical-failure-only rate (8.2%) is actually in line with (better than) the July baseline once the July "Smoke CI" chronic failure is accounted for.

Daily token totals (bars) with a 3-sample moving average. Today's 31.6M-token day is the highest ever recorded for this repo, driven by a handful of very token-dense runs (see Finding 6). The June token-reporting gap (token-usage-reporting-gap, still open in known-issues.json) appears to have substantially improved — 74% of today's runs now report token data vs. near-total nulls in late June.

---

Findings

1. Avenger — 100% failure rate, unresolved for ~12 weeks (HIGH, recurring)
Every one of today's 5 scheduled runs (~every 4h, main branch) failed with failure_kind: driver_exit, TokenUsage=0, Turns=0 — the agent never executes a single turn. This is the exact signature of avenger-err-config-no-structured-logs in prior memory (first_seen: 2026-06-13, recurrence_count: 18 as of 2026-07-06). It is now confirmed still failing, 100% of runs, on 2026-08-28 — at least 11 weeks unfixed with zero mitigation. This is the single most durable chronic issue in the fleet and should be prioritized or the schedule paused until root-caused.

2. NEW — Six PR-gate reviewer workflows fail together as a bloc in 1–9 seconds (HIGH, novel this window)
Design Decision Gate, Impeccable Skills Reviewer, Matt Pocock Skills Reviewer, PR Code Quality Reviewer, Ponytail Reviewer, and Test Quality Sentinel fire on the same pull_request event and, on a given PR, either all succeed together or all fail together — e.g. all 6 failed in 1–4s on copilot/fix-integration-workflow-misc-part-2, all 6 failed in 3–4s on copilot/debug-daily-code-debt-cleanup, all 6 succeeded together on copilot/fix-fallback-issue-warning. Across today's data this bloc pattern accounts for 57 of the 93 "unknown-engine" failures and never shows a partial split (e.g. 3-of-6). This strongly indicates a shared precondition or shared included dependency (not 6 independent AI judgments) that occasionally fails before any reviewer agent starts — worth checking whatever common template/composite step (e.g. a shared shared/*.md include or setup action) these 6 workflows depend on. Failure rates individually: Design Decision Gate 34.4%, Matt Pocock/PR CQR 31.3%, Test Quality Sentinel 28.6%, Impeccable/Ponytail 28.1%.

3. AI Moderator — 75.6% failure rate (31/41), but likely a content gate, not a bug (MEDIUM, needs confirmation)
Unlike Finding 2, AI Moderator runs standalone with real duration (18min–4h), so its failures are probably legitimate "flagged content, block PR" gate outcomes rather than infra failures — none carry a driver_exit/agent_logic failure_kind. However the duration spread (some runs took up to 4 hours) is unusual for a moderation check and worth a look — that could indicate the agent is retrying/looping rather than a clean fail-fast gate decision.

4. Windows Runner Integration Test — 100% failure (3/3), all driver_exit, GitHub Copilot CLI (MEDIUM, new/recurring)
All 3 runs today failed in 3–6 minutes with the driver-exit signature. Not present in prior known-issues.json — likely a new or freshly-recurring Windows-runner-specific Copilot CLI startup problem, worth its own investigation thread.

5. Visual Regression Checker — mixed hang/fail pattern (LOW-MEDIUM)
2 of 3 failures ran 1.6–2.0 hours before failing (no failure_kind, i.e. not a clean driver crash — looks like a hang/timeout), and the third failed cleanly via driver_exit in 4 minutes. Worth checking whether this workflow has an effective timeout — multi-hour hangs before failure waste both wall-clock and probably AI credits.

6. Cost/token outliers worth reviewing (LOW, informational)

• Daily Cache Strategy Analyzer — 3.14M tokens in 24.4 min (Codex, succeeded) — by far the most token-dense run today, ~2.4x the next highest (Dev, 2.24M tokens in 10.3 min).
• Daily VulnHunter Scan — highest single-run AI-credit cost at 1,029 AIC (Claude, 14.4 min, succeeded).
• Agent Job Health Monitor — 743 AIC over 31 min (Claude, succeeded).
  These aren't failures, just the top of the cost distribution — flagging in case token/cost budgets are being tracked per-workflow.

7. Daily Max Ai Credits Test failed via driver_exit, not its intended guardrail trip (LOW)
This workflow is designed to intentionally trip the max-ai-credits firewall, but today's failure is not marked intentional_failure: true in the data (unlike the sibling Daily Credit Limit Test, which succeeded and is marked intentional) — meaning this run crashed for a real reason rather than by design. Worth a quick look to confirm the guardrail test itself is still working as intended.

8. Transient multi-engine smoke-test cluster at 03:08–03:14 UTC (LOW, resolved same window)
Smoke Gemini, Smoke Copilot, Smoke Copilot - AOAI (apikey), Smoke Copilot - AOAI (Entra), and Smoke OTEL all failed within a ~90-second span across 4 different engines (Gemini CLI, Copilot CLI, Codex). This cross-engine, tightly-clustered pattern matches the playwright-install-cli-failure shape from prior memory (infra/network blip, not engine-specific) — no recurrence seen later in the window, so likely a transient runner/registry hiccup rather than an ongoing issue.

Resolved / improved since 2026-07-06 memory (good news)

• Smoke CI — the dominant chronic issue in the last recorded audit (100% fail on every push, 31% of all fleet failures) had zero runs in today's 24h window — either retired, renamed, or fixed outright.
• Daily Cache Strategy Analyzer — previously chronic (codex-gh-aw-binary-not-found-for-mcp, unfixed since 06-15) — succeeded today (and was the top token-usage run).
• Daily Issues Report Generator — previously blocked by chroot-node-not-available (recurrence_count 11) — succeeded today (11.6 min, real token usage).
• copilot-sdk-driver-failures longrun-0turn pattern — previously a dominant 18–28 minute "real work then 0-turn crash" signature on PR Code Quality Reviewer/Impeccable Skills Reviewer/etc. — today's failures on those same workflows are all 1–35 seconds (Finding 2's bloc pattern), a different and much faster signature, suggesting the old longrun crash is gone even though a new fast-fail issue has emerged.
• Token-usage reporting gap — improved from "mostly null" to 74% of runs reporting tokens, though token-usage-reporting-gap should stay open at lower severity until it's fully closed.

Recommendations

1. Root-cause or pause Avenger's schedule — 11+ weeks at 100% failure with zero successful runs is pure wasted compute; either fix the err-config/structured-log issue or disable the schedule until it's diagnosed.
2. Investigate the shared dependency behind the 6-workflow PR-gate bloc failures (Finding 2) — check whatever common included template/composite action Design Decision Gate, Impeccable/Matt Pocock/Ponytail Reviewers, PR Code Quality Reviewer, and Test Quality Sentinel all rely on; a fix here would resolve the majority of today's "unknown-engine" failures at once.
3. Add a timeout to Visual Regression Checker — 1.6–2h hangs before failure suggest a missing/ineffective timeout.
4. Confirm Daily Max Ai Credits Test's driver_exit is not masking a broken guardrail — compare against Daily Credit Limit Test's successful/intentional run this same window.
5. Restore repo-memory write access for this workflow — the audit could not persist known-issues.json/metrics-summary.json/etc. updates this run because /tmp/gh-aw/repo-memory/default/ was read-only; without a fix, the 53-day-gap problem will recur.

---

Generated by the Agentic Workflow Audit Agent. Data source: agenticworkflows logs (5 paginated fetches, 463 unique runs merged) + /tmp/gh-aw/repo-memory/default/ historical files (read-only this run).

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• api.anthropic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.anthropic.com"

See Network Configuration for more information.

Generated by 🔍 Agentic Workflow Audit Agent · claude · agent · 194.2 AIC · ⌖ 7.7 AIC · ⊞ 9.3K · ◷

• expires on Aug 29, 2026, 2:31 PM UTC-08:00