Agentic Workflow Audit — 2026-03-26

[github-actions[bot]]

Daily audit of agentic workflow runs for the last 24 hours (2026-03-25 → 2026-03-26).

Summary

Metric	Value
Total tracked runs	28 (of ~202 in period)
✅ Successful	11 (39%)
❌ Failed	14 (50%)
⏭️ Skipped	1
Estimated Total Cost	$7.15
Total Tokens	11.1M

⚠️ Today's 44% success rate is the lowest in the 30-day record — previous range was 72–91%.

Workflow Health Trend

The sharp drop today is driven primarily by widespread Copilot authentication failures (7+ runs) and GitHub API rate limiting during a concurrent execution burst at ~20:25 UTC.

Token & Cost Trend

Today's $7.15 cost is the highest in the historical record, driven by multiple high-turn Claude runs (Smoke Claude ran twice at ~$1.10 each, plus Daily Doc Updater at $1.11).

---

🔴 Critical Issues

1. Copilot Authentication Failures (7 runs)

Root cause: Authentication failed (Request ID: ...) — Copilot token is invalid, expired, or missing the Copilot Requests permission.

Affected workflows:

• issue-monster — 3 failures (§23614004922, §23615946471, §23617232281)
• smoke-copilot — 2 failures (§23614909987, §23615931012)
• agent-container-smoke-test — 2 failures (§23614910062, §23615931059)
• metrics-collector (§23614384478)
• daily-difc-integrity-filtered-events-analyzer (§23615370549)
• daily-workflow-updater (§23617471074)

Recommended fix: Refresh COPILOT_GITHUB_TOKEN (or GH_AW_GITHUB_TOKEN) secret. Verify the Fine-Grained PAT has the Copilot Requests permission enabled. Check with gh auth status.

2. GitHub API Rate Limiting — safe_outputs failures

Root cause: Concurrent PR workflows targeting PR #23160 triggered the installation rate limit at 2026-03-26T20:25 UTC. The safe_outputs job received: API rate limit exceeded for installation — 6 safe-output messages failed.

Affected workflows:

• smoke-claude (§23615931027) — agent succeeded ✅, safe_outputs failed ❌ (add_comment, update_pull_request, create_pull_request_review_comment ×2, add_reviewer, submit_pull_request_review)
• daily-doc-updater (§23616275567) — agent succeeded ✅, safe_outputs failed ❌

Recommended fix: Stagger schedule times for workflows that run on the same PR. Consider adding retry logic with exponential backoff in safe_outputs for rate limit errors (HTTP 429).

---

⚠️ Performance Concerns

High-Cost Runs (Top 5)

Workflow	Cost	Tokens	Turns	Run
Smoke Claude	$1.26	2.14M	39	§23617381060
Daily Documentation Updater	$1.11	2.03M	49	§23616275567
Smoke Claude	$1.09	1.84M	45	§23615931027
Sergo	$1.06	1.32M	33	§23616076030
Static Analysis Report	$1.01	1.64M	24	§23614592028

The audit system flagged Smoke Claude as resource_heavy_for_domain with partially_reducible assessment: ~91% of turns are data-gathering that could move to deterministic pre-agent steps. Consider moving file reads and PR metadata fetching to frontmatter steps to reduce inference cost.

Agentic Behavior Concerns

• Smoke Claude: 45 turns, exploratory execution, write_heavy actuation. Baseline comparison shows turns increased from 40 → 45 vs prior successful run. poor cost_efficiency rating.
• Smoke Codex: Flagged resource_heavy_for_domain for a Triage task with 14 tool types and 13.6m duration despite 0 agent turns.
• Daily Documentation Updater: 49 turns, $1.11 — consider whether a subset of this work can be deterministic.

---

🔧 Missing Tool

add_smoked_label was requested by Smoke Codex (§23614909967) but is not configured. The agent correctly reported this via missing_tool and the run succeeded. This tool should be added to the workflow's safe-output permissions.

---

✅ Healthy Runs

Workflow	Event	Duration	Tokens	Cost
AI Moderator (×3)	pull_request	1.2m	—	—
Smoke Codex	pull_request	13.6m	—	—
Changeset Generator (×2)	pull_request	—	—	—
Smoke Copilot	workflow_dispatch	—	—	—
Smoke Claude	workflow_dispatch	—	2.14M	$1.26
Sergo	schedule	—	1.32M	$1.06
Static Analysis Report	schedule	—	1.64M	$1.01
Step Name Alignment	schedule	—	548K	$0.66

---

Recommendations

1. Immediate: Rotate/refresh the Copilot token used by copilot-engine workflows. This is causing 50%+ of today's failures.
2. Short-term: Investigate API rate limiting patterns — consider staggering workflow schedules that target the same PRs at the same time.
3. Optimization: Add add_smoked_label to the Smoke Codex workflow safe-output configuration.
4. Cost efficiency: Review Smoke Claude and Daily Documentation Updater prompts — move data-fetching to deterministic pre-agent steps (see [Deterministic & Agentic Patterns guide]).

References:

• §23614909967 — Smoke Codex (success, missing tool)
• §23615931027 — Smoke Claude (rate limit failure)
• §23614004922 — Issue Monster (auth failure)

AI generated by Agentic Workflow Audit Agent · history

• expires on Mar 27, 2026, 9:20 PM UTC

[github-actions[bot]]

This discussion has been marked as outdated by Agentic Workflow Audit Agent.

A newer discussion is available at Discussion #23276.