Agent Performance Report - Week of 2026-08-18

[github-actions[bot]]

Executive Summary

• Agents analyzed: 62 active (of 284 total workflows)
• Total safe outputs (10h partial window): 368
• Overall success rate: 95% (↑ from 87.5% Aug 15, 90.9% Aug 14, 69.5% Aug 13)
• Top performers: PR Sous Chef (81 items/32 runs), PR Code Quality Reviewer (27 items/12 runs, 100% success), Matt Pocock Skills Reviewer (43 items/11 runs, 100% success)
• Needs improvement: Smoke Copilot - AOAI (apikey) (0% success, 2/2 runs failed), Avenger (50% success), Daily Team Evolution Insights (0% success), Smoke Gemini (0% success)

Correction to prior report (Jul 8): The last agent-performance-latest.md flagged Matt Pocock Skills Reviewer, Impeccable Skills Reviewer, PR Code Quality Reviewer, and Test Quality Sentinel as 100% activation-refused due to a "CLI hang-on-exit" bug. That data is 6 weeks stale. Current metrics and direct PR inspection (PRs #53601, #53672, #53680, today) confirm all four are running successfully at 100% success rate today, posting substantive inline review comments (e.g., correct #nosec G204 placement catch, Windows path-traversal edge case, missing test coverage) and submitting reviews. The CLI hang-on-exit fix (#44254, merged) resolved this. Do not re-file or re-flag this cluster — remove it from active watch lists.

Performance Rankings (today's ~10h window)

Top Performing Agents 🏆

1. PR Code Quality Reviewer — 27 items / 12 runs, 100% success, ~$147.60 cost, avg 402s runtime
2. Matt Pocock Skills Reviewer — 43 items / 11 runs, 100% success. Verified quality: inline comments correctly identify gosec annotation placement bugs and Windows-specific path traversal gaps with concrete fix suggestions — high actionability.
3. PR Sous Chef — 81 items / 32 runs, 96.9% success (highest volume agent; still worth watching for scope given volume — see Patterns below)
4. Issue Monster — 26 items / 5 runs, 100% success, low cost ($11.88 for 5 runs) — efficient triage-to-assignment agent
5. Contribution Check — 12 items / 3 runs, 100% success

Agents Needing Improvement 📉

1. Smoke Copilot - AOAI (apikey) — 0% success (0/2), 1224s avg runtime, 2.98M tokens burned with no successful output. This is a smoke/CI test, not a production agent, but the token burn on failed runs is high — worth a runtime cap.
2. Avenger — 50% success (2/4 runs). Existing issue [aw] Avenger failed #53238 (closed) and [deep-report] Avenger chronic driver_exit (err-config-no-structured-logs) unresolved after 19 recurrences, 4 prior closed fixes #53251 (deep-report, chronic driver_exit/err-config-no-structured-logs, 19 recurrences across 4 prior "fixes") — DO NOT RE-FILE, but flag as a systemic reliability gap: repeated fixes are not sticking.
3. Daily Team Evolution Insights — 0% success (0/1). Issue [aw] Daily Team Evolution Insights failed #53483 already open (failed today) with related fix PRs merged (Fix Goose engine safeoutputs ENOENT failure in daily-team-evolution-insights workflow #51872, Print explicit window_start/window_end UTC timestamps in Daily Status and Team Evolution reports #53545, Add gh-aw-detection to Daily Team Evolution Insights and MCP Inspector Agent #53556) — appears to be an ongoing stabilization effort, monitor next run.
4. Smoke Gemini — 0% success (0/1). Smoke test infrastructure, not user-facing; low priority.
5. Design Decision Gate — 100% success in latest metrics window (13/13), but two new issues filed today ([aw-failures] [P1] Design Decision Gate hangs silently mid-tool-use in Claude Code CLI step #53619 P1 hang mid-tool-use in Claude Code CLI step during large-diff Read; Surface stalled Claude CLI failures and bound Design Decision Gate diff input #53677 companion fix to bound diff input and surface stalled Claude CLI failures). This is a live, unresolved regression distinct from the stale Jul 8 note — treat as current P1, not stale.

Inactive/Chronic-Failure Agents

• Smoke Copilot Sub Agents — 0% success (0/1) today; long-running "100% red" pattern per historical alerts.
• Linter Miner — 0% success (0/1) today.

Behavioral Patterns

• PR Sous Chef (over-creation watch): 81 safe items from 32 runs (2.5 items/run) is the highest ratio among reviewed agents — combined with 3 open issues today about chronic safe-outputs job failures (Prevent recurring PR Sous Chef safe-output failures #53676, [deep-report] PR Sous Chef: root-cause chronic safe_outputs job failures (16 open duplicate auto-filed issues) #53615) and a new scope-expansion request (Allow PR Sous Chef to approve CJS and CGO action-required runs #53679, approve CJS/CGO action-required runs). Recommend auditing whether its per-run output volume is proportionate to review value, and consolidating the 16 open duplicate auto-filed issues referenced in [deep-report] PR Sous Chef: root-cause chronic safe_outputs job failures (16 open duplicate auto-filed issues) #53615 before they compound.
• Avenger (inconsistency): 4 prior "fixes" for the same driver_exit root cause have not resolved it (per [deep-report] Avenger chronic driver_exit (err-config-no-structured-logs) unresolved after 19 recurrences, 4 prior closed fixes #53251). This is a config/logging gap (no structured logs to diagnose root cause), not a prompt issue — needs an infrastructure fix, not another patch-and-close cycle.
• Design Decision Gate (inconsistency): Was stable per aggregate metrics (100% success, 13 runs) but has 2 fresh failure reports today from silent hangs on large diffs. Aggregate success-rate metrics can mask acute, recently-introduced regressions — cross-check metrics against today's issue stream before declaring an agent healthy.
• No clear scope-creep or repetition detected in the sampled top agents beyond the Sous Chef volume noted above.

Data Quality Note

The metrics/latest.json snapshot covers only ~10h of a 24h window (collection tool timeouts on batch 10) and lacks per-type (issues/PRs/comments) breakdown — only aggregate total_safe_items was available. Effectiveness scores (PR merge rate, issue close time) were null across all workflows in this snapshot; this analysis relied on direct gh queries against a handful of top/bottom agents to compensate. Recommend the Metrics Collector workflow (flagged previously as having stale data since Jan 2026, though now producing daily files through Aug 17) add per-type breakdown to close this gap.

Recommendations

High Priority

1. Retire the stale "CLI hang-on-exit" flag for Matt Pocock/Impeccable/PR Code Quality Reviewer/Test Quality Sentinel from all shared-alert and health-tracking documents — confirmed resolved, currently 100% success with high-quality output. Continuing to track this as an open concern wastes future analysis cycles.
2. Escalate Design Decision Gate's new silent-hang issue ([aw-failures] [P1] Design Decision Gate hangs silently mid-tool-use in Claude Code CLI step #53619) — this is a genuine, currently-unresolved P1 (large-diff Read hangs with zero error signal in the Claude Code CLI step). Fix PR Surface stalled Claude CLI failures and bound Design Decision Gate diff input #53677 already proposes bounding diff input; prioritize merging it since ADR enforcement PRs are currently blocked by this hang.
3. Root-cause Avenger's driver_exit failures properly — 4 closed "fixes" have not stopped recurrence (19 instances per [deep-report] Avenger chronic driver_exit (err-config-no-structured-logs) unresolved after 19 recurrences, 4 prior closed fixes #53251). Recommend adding structured logging around the driver exit path (as [deep-report] Avenger chronic driver_exit (err-config-no-structured-logs) unresolved after 19 recurrences, 4 prior closed fixes #53251 proposes) before attempting another prompt/config patch, since patching symptoms hasn't worked.

Medium Priority

1. Consolidate PR Sous Chef's duplicate auto-filed issues (16 open per [deep-report] PR Sous Chef: root-cause chronic safe_outputs job failures (16 open duplicate auto-filed issues) #53615) before the backlog compounds further; add a dedup check that also considers issue title/topic similarity, not just exact match.
2. Improve Metrics Collector to capture per-type safe-output breakdown (issues/PRs/comments/discussions individually) so effectiveness scores (merge rate, close time) can be computed instead of showing null across the board.

Low Priority

1. Cap runtime/token budget for smoke-test workflows with 0% success (AOAI apikey, Gemini, Sub Agents) — they're burning significant tokens (2.98M for AOAI apikey) on failures with no diagnostic value beyond "still broken."

Trends

• Overall success rate: 95% today (↑ from 87.5% Aug 15, 90.9% Aug 14, 69.5% Aug 13) — clear positive trend over the past 5 days
• Active workflow count fluctuating (51-111 range over past week) — normal variance tied to event-driven triggers, not a concern
• Matt Pocock/Impeccable/PR Code Quality Reviewer cluster: fully recovered from the CLI hang-on-exit bug that plagued them through early July

Actions Taken This Run

• Verified and corrected a stale 6-week-old false-positive flag on 4 agents (Matt Pocock, Impeccable, PR Code Quality Reviewer, Test Quality Sentinel) via direct PR inspection
• Identified 1 new live P1 regression (Design Decision Gate silent hang) already tracked by fresh issues [aw-failures] [P1] Design Decision Gate hangs silently mid-tool-use in Claude Code CLI step #53619/Surface stalled Claude CLI failures and bound Design Decision Gate diff input #53677 — no duplicate filed
• No new issues filed this run (all identified problems already have open tracking issues); calling noop with this summary in lieu of creating a discussion, since the primary finding is a memory correction rather than a new external-facing report

Next Steps

1. Update agent-performance-latest.md and shared-alerts.md to remove the stale CLI-hang-on-exit flag for the 4 cleared agents
2. Monitor Design Decision Gate fix PR Surface stalled Claude CLI failures and bound Design Decision Gate diff input #53677 merge and confirm hang no longer recurs
3. Track Avenger's structured-logging fix for driver_exit root cause
4. Re-run Metrics Collector improvements to get typed safe-output breakdown for next week's report

---

Analysis period: 2026-08-17T16:29 to 2026-08-18T12:57 UTC (~10h partial window per available metrics + direct issue/PR verification)
Next report: 2026-08-25

Generated by ⚡ Agent Performance Analyzer - Meta-Orchestrator · auto · 64.5 AIC · ⌖ 12.5 AIC · ⊞ 16.8K · ◷

• expires on Aug 19, 2026, 5:04 AM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-08-19T13:04:39.533Z.

Closed by Workflow