Agent Performance Report - Week of 2026-08-03

[github-actions[bot]]

Agent Performance Report — Week of 2026-08-03

Executive Summary

• Data caveat: Shared metrics store (metrics/latest.json, metrics/daily/*.json) is stale — last successful collection 2026-01-18 ("Metrics Collector" engine failure, tracked in [aw] Metrics Collector failed #43292/persistent-issues list). Historical Q/E scores below are carried from the last full analysis (2026-07-08, Q:63/100, E:64/100, Health:63/100) and cross-checked against live issue-tracker evidence gathered this run (2026-08-01 → 2026-08-03).
• Live evidence gathered: 300 most-recent issues scanned via gh issue list --json; targeted gh issue view on the bottom-10 candidate agents.
• Persistent bottom performers confirmed still failing (Aug 1–3): Design Decision Gate, Matt Pocock Skills Reviewer, Impeccable Skills Reviewer, PR Code Quality Reviewer, Test Quality Sentinel.
• New root causes identified this run (not previously logged with this specificity):
  • Design Decision Gate: engine max-runs exceeded (claude engine loop/retry budget exhausted) — issue [aw] Design Decision Gate 🏗️ failed #49989, recurring daily ([aw] Design Decision Gate 🏗️ failed #49786, [aw] Design Decision Gate 🏗️ failed #49494).
  • Matt Pocock Skills Reviewer: skill installation failure — gh skill install mattpocock/skills/to-prd@<sha> exits 1 (issue [aw] Matt Pocock Skills Reviewer failed #49954), separate from the earlier CLI-hang-on-exit cluster.
• Confirmed unresolved architectural decision: PR Deprecate auto-run paths for Matt Skills Reviewer and Design Decision Gate to stop chronic failure-issue churn #48730 ("Deprecate auto-run paths for Matt Pocock Skills Reviewer and Design Decision Gate") was closed unmerged, not landed (verified via [Agent Perf] Deprecation PR #48730 for Matt Pocock Skills Reviewer / Design Decision Gate was closed unmerged — chronic churn co [Content truncated due to length] #49577). Both workflows still have live on.pull_request triggers and no report-failure-as-issue: false, so the churn continues.

Bottom-10 Prompt/Workflow Deficiency Audit

Agent	Recent failures (Aug 1–3)	Deficiency category	Evidence
Design Decision Gate	#49989, #49786, #49494 (daily)	Missing self-assessment loop / stale tool guidance	Claude engine repeatedly exhausts max-runs guardrail — no retry/loop-budget safeguard in prompt; root cause unrelated to original CLI-hang fix (#44254)
Matt Pocock Skills Reviewer	#49954, #49867, #49780, #49743, #49662 (9 in scan window)	Stale tool guidance / low actionability	New failure mode: gh skill install for mattpocock/skills/to-prd@<sha> fails outright — pinned skill ref may be stale/inaccessible; workflow has no fallback when skill install fails
Impeccable Skills Reviewer	#49979, #49779, #49661, #49380	Low actionability (rate-limit only, no content issue)	All 4 recent issues are "hit AI credits rate limit" — not a prompt-quality problem per se, but recurring enough to warrant either a lower cadence or shared credit pool fix
PR Code Quality Reviewer	#49953, #49868, #49850, #49795, #49778 (11 in window)	Generic task framing / missing tool	Mix of failure/timeout/"missing required tool" — inconsistent failure modes suggest brittle setup, not a stable prompt issue
Test Quality Sentinel	#49983, #49957, #49666, #49383	Low actionability (rate-limit + generic failure)	Same AI-credit-rate-limit pattern as Impeccable; co-occurs with PR Code Quality Reviewer, suggesting shared engine/credit contention rather than independent prompt defects

Explicit redesign-vs-deprecation candidates (per initiative directive):

• Design Decision Gate — recommend redesign: the failure mode (engine max-runs exceeded) is fixable by raising max-runs or trimming prompt-driven retries; deprecation PR was rejected once already (Deprecate auto-run paths for Matt Skills Reviewer and Design Decision Gate to stop chronic failure-issue churn #48730 closed unmerged), so a scoped fix is more likely to land than a second deprecation attempt.
• Matt Pocock Skills Reviewer — recommend deprecation or skill-ref fix: this is the second distinct failure cluster in a month (CLI-hang, now skill-install-failure) for a maintainer-facing style-linter that has never landed a merged deprecation despite repeated proposals.
• Impeccable Skills Reviewer — recommend investigate credit allocation, not prompt redesign: failures are uniformly "rate limit," implying it's a scheduling/quota problem shared with Test Quality Sentinel.

Recommendations

High Priority

1. Land or re-propose PR Deprecate auto-run paths for Matt Skills Reviewer and Design Decision Gate to stop chronic failure-issue churn #48730-equivalent for Design Decision Gate / Matt Pocock Skills Reviewer. The prior deprecation PR was closed unmerged — this is the single highest-leverage fix, since it directly stops ~2 failing daily workflows and the issue churn they generate. Owner: maintainer decision + Copilot SWE Agent.
2. Fix Design Decision Gate max-runs guardrail ([aw] Design Decision Gate 🏗️ failed #49989 cluster): either raise max-runs for the claude engine step or reduce per-run invocations (trim retry loops in the ADR-gate prompt). Expected impact: eliminates the daily "[aw] Design Decision Gate failed" issue (3+ occurrences this week alone).
3. Fix or repin mattpocock/skills/to-prd@<sha> skill reference ([aw] Matt Pocock Skills Reviewer failed #49954): confirm the skill repo/branch/commit is still valid and accessible with the configured token; add a fallback path so a skill-install failure doesn't hard-fail the whole run.

Medium Priority

1. Investigate shared AI-credit rate-limiting affecting Impeccable Skills Reviewer + Test Quality Sentinel simultaneously — likely a scheduling/credit-pool contention issue rather than two independent prompt defects. Consolidating or staggering their run schedules may resolve both.
2. Stabilize PR Code Quality Reviewer — its failure modes are inconsistent (timeout, missing tool, generic failure) across 11 recent issues, suggesting brittle environment setup rather than a single root cause; worth a dedicated triage pass separate from the skills-reviewer cluster.

Data Infrastructure

1. Metrics Collector is down since Jan 18, 2026 (tracked persistently as [aw] Metrics Collector failed #43292) — restore this before the next full quantitative agent-performance run; all Q/E scores in this and recent reports are extrapolated from live issue-tracker sampling, not fresh metrics.

Actions Taken This Run

• Sampled 300 most recent issues + targeted gh issue view on 5 underperforming agents to identify concrete, current root causes.
• Confirmed PR Deprecate auto-run paths for Matt Skills Reviewer and Design Decision Gate to stop chronic failure-issue churn #48730 (proposed deprecation) was closed unmerged, invalidating prior "resolved" status for Design Decision Gate / Matt Pocock Skills Reviewer in shared memory.
• No new duplicate issue filed — [Agent Perf] Deprecation PR #48730 for Matt Pocock Skills Reviewer / Design Decision Gate was closed unmerged — chronic churn co [Content truncated due to length] #49577 already tracks the deprecation-churn root cause comprehensively; filing another would duplicate existing tracking (violates DO-NOT-RE-FILE convention).

Next Steps

1. Maintainer decision needed on Design Decision Gate / Matt Pocock Skills Reviewer: land deprecation vs. invest in fixes.
2. Restore Metrics Collector for quantitative trend tracking.
3. Re-run full quality/effectiveness scoring once metrics pipeline is restored.

---

Analysis period: 2026-08-01 to 2026-08-03 (live issue-tracker sample); historical scores from 2026-07-08 baseline (last full metrics run)
Data limitation: shared metrics store stale since 2026-01-18; this report substitutes live GitHub issue sampling for quantitative scoring

Generated by ⚡ Agent Performance Analyzer - Meta-Orchestrator · auto · 33.3 AIC · ⌖ 10.5 AIC · ⊞ 17K · ◷

• expires on Aug 4, 2026, 5:41 AM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-08-04T13:41:47.595Z.

Closed by Workflow