[audit-workflows] Agentic Workflow Audit — 2026-08-09 (323 runs, 96.6% success, 34-day gap since last audit)

[github-actions[bot]]

Summary

Audit window: 2026-08-08 21:34 UTC → 2026-08-09 21:06 UTC (full ~24h, 323 runs collected across 4 paginated pages). Note: the prior audit ran on 2026-07-06, leaving a 34-day monitoring gap — this run is the first since then, so day-over-day trend claims are limited to a single new data point after the gap.

• 323 runs, 312 success, 11 failures → raw success rate 96.6%
• Excluding the 2 designed-to-fail credit-guardrail stress tests (Daily Credit Limit Test, Daily Max Ai Credits Test) from the denominator: 96.9% adjusted rate
• Total tokens: 12,498,032 across runs reporting usage; total AI-credits (aic): 11,561.1; total action minutes: 2,771
• 0 missing-tool, missing-data, or MCP-failure signals in this window

Failure clustering

Cluster 1 — Cross-engine Execute-CLI driver-exit (7 runs)

The dominant failure mode: the agent step exits at the driver level (0-turn, before any real agent work) rather than failing on agent logic. This is a reconfirmation of the existing known issue copilot-sdk-driver-failures (known-issues.json), now showing evidence of engine spread beyond Copilot:

• copilot engine: 3 occurrences
• claude engine: 3 occurrences (matches known issue claude-agent-job-fail-longrun)
• kiro engine: 1 occurrence — new engine affected, not previously recorded against this issue

This cross-engine pattern suggests the root cause is more likely in shared harness/driver plumbing (tool-permission bootstrap, session startup) than in any single engine's SDK.

Cluster 2 — Safe-outputs partial-failure intolerance (2 runs, PR Sous Chef)

Both failures are PR Sous Chef runs where the agent step completed successfully but the safe_outputs processing step failed partially, and the current design treats any partial safe-output failure as a full run failure. This reconfirms known issue safe-output-partial-failure-intolerance — same workflow recurring since last audit.

Cluster 3 — Legitimate security-gate failure (1 run, not an infra issue)

Daily Container Image Security Scan failed because it correctly caught a real vulnerability/license violation in a scanned image — this is the gate doing its job, not a bug. Recorded for completeness, excluded from "issues to fix."

The remaining 1 failure is the intentional Daily Max Ai Credits Test stress test (by design, excluded from adjusted rate).

Token/AIC outliers

3 workflows in this window used disproportionately high tokens relative to their peers (see rec-token-outlier-review-3wfs in recommendations.json, updated this run) — flagged for a closer look at whether their prompts/loops are over-consuming context relative to task complexity.

Recommendations

1. Investigate cross-engine Execute-CLI driver-exit (copilot, claude, kiro) — the kiro occurrence is new evidence this is a shared-harness bug, not engine-specific. Suggest checking tool-permission allowlist bootstrapping and session-init timing across engines.
2. Relax or scope safe-outputs partial-failure handling for PR Sous Chef — a partial safe-output failure currently fails the whole run; consider distinguishing "safe-output degraded" from "run failed."
3. Review the 3 token-usage outliers flagged this run for prompt/loop efficiency.
4. Verify the audit workflow's own schedule — the 34-day gap between this run and the prior one (2026-07-06) suggests the daily audit schedule itself may have been disabled, broken, or silently failing between then and now. Recommend confirming the cron/schedule trigger is active and adding self-monitoring so a missed audit is itself detected.
5. Repo-memory storage format — recommendations.json and workflow-trends.json are stored as single-line minified JSON. Because git diffs are line-based, any edit to these files costs roughly 2× the full file size in patch bytes (whole line removed + whole line re-added), which pushed this run's total patch over the 50KB hard limit. This run's updates to those two files were reverted before push to stay within the limit — their content still reflects the 2026-07-06 snapshot. Recommend either reformatting them to indent=2 (accepting a one-time larger diff in exchange for small surgical diffs going forward) or splitting workflow-trends.json into smaller per-workflow files.

Trend charts

Workflow Health Trends (success/failure counts + success rate)

The top panel stacks daily success (green) and failure (red) run counts; the shaded band marks the 34-day gap with no audit data (2026-07-07 to 2026-08-08). The bottom panel shows the success-rate line, deliberately broken across that gap rather than interpolated — the rate moved from 86.25% on 2026-07-06 to 96.6% today, but with zero visibility into the path between those two points, so no gradual-trend claim can be made. Both of the two recorded data points sit above the 90% reference line.

Token Usage Trends (daily tokens + moving average)

Only ~11 days across the full 30-day lookback window have any recorded token data, clustered in a few short bursts (e.g. Jun 13, Jun 20, Jul 03–05) plus today. The moving-average line is computed and drawn only within each contiguous cluster (gap ≤ 2 days) rather than across the large gaps between them, to avoid implying a continuous trend where there isn't one. Today's 12.5M-token total is the highest single-day figure in the available data; whether that's a one-off (catch-up after the gap) or a new baseline can't be determined without tomorrow's run.

Repo-memory updates

• known-issues.json — 3 existing issues reconfirmed with updated last_seen, incremented recurrence_count, new affected workflows/engines noted (indent=2, format preserved)
• anomalies.json — 2 new anomalies recorded: the kiro-engine Execute-CLI driver-exit, and the 34-day audit-monitoring gap itself (indent=2, format preserved)
• metrics-summary.json — today's window and history entry appended (indent=2, format preserved)
• audit-history.jsonl — full audit entry appended (compact JSONL, append-only)
• recommendations.json, workflow-trends.json — not updated this push (reverted) due to the patch-size issue described in Recommendation 5 above; still reflect the 2026-07-06 snapshot

Methodology notes

• Engine classification came from each run's engine_id/agent field and summary.engine_counts, not from scanning .lock.yml files.
• Success-rate rollups exclude the 2 intentional-failure credit-guardrail stress tests from the adjusted-rate denominator; raw rate is reported alongside.
• Data collected via the agenticworkflows MCP CLI (status then logs with start_date: -1d, 4 paginated pages, 323 unique runs after dedup by run_id).

Generated by 🔍 Agentic Workflow Audit Agent · agent · 272.3 AIC · ⌖ 47.8 AIC · ⊞ 8.3K · ◷

• expires on Aug 10, 2026, 1:44 PM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-08-10T21:44:28.551Z.

Closed by Workflow