Agent Performance Report — Week of 2026-07-27

[github-actions[bot]]

Executive Summary

• Agents analyzed: 25 (sampled last 100 runs)
• Total issues reviewed: 20 (Jul 27 recent); 2 open PRs
• Average quality score: 62/100 (→ flat vs Jul 8 baseline)
• Average effectiveness score: 61/100 (→ flat vs Jul 8 baseline)
• Top performers: Agentic Maintenance, Auto-Triage Issues, PR Triage Agent, Bot Detection, Avenger, Copilot cloud agent
• Needs improvement: AI Moderator, Q workflow, Agentic Commands, CGO, Claude Code User Documentation Review

Key Metrics (Jul 27)

Metric	Value	Trend
Total runs (last 100)	100	—
Success rate	~40%	→ flat
AR rate	~38%	→ flat
Failure rate	~5%	slight ↑ (CGO, Claude Code, CJS)
Skipped/noop rate	~17%	→ flat
CI approval-pending (CWI/CJS/CGO)	9 AR	(GitHub gate, not agent failure)

Performance Rankings

Top Performing Agents 🏆

1. Agentic Maintenance (Q: 85/100, E: 85/100) — 1/1 success today. Consistent, clean outputs.
2. Auto-Triage Issues (Q: 88/100, E: 87/100) — 1/1 success. High-quality issue labeling.
3. PR Triage Agent (Q: 88/100, E: 86/100) — 1/1 success. Reliable PR classification.
4. Bot Detection (Q: 80/100, E: 80/100) — 1/1 success.
5. Avenger (Q: 84/100, E: 83/100) — 1/1 success. Stable.
6. Copilot cloud agent (Q: 88/100, E: 87/100) — 5/6 success (83%). Open PRs are substantive.
7. PR Sous Chef (Q: 78/100, E: 75/100) — 1/1 success. Recovered vs prior issues.

Agents Needing Improvement 📉

1. AI Moderator (Q: 35/100, E: 30/100) — 0% success today (6 AR, 5 skip). Persistent Codex engine failure ([aw] AI Moderator produced no safe outputs #44241, #aw_ai_mod_jul8). No new issue needed.
2. Q workflow (Q: 25/100, E: 15/100) — 0% success (7 AR, 5 skip). Quality gate PR Add shared prompt quality gate for plateaued agent-review workflows #43527 not yet merged. Longest-standing P1 blocker.
3. Agentic Commands (Q: 40/100, E: 38/100) — 5/15 success (33%), 10 AR. Persistent. [aw] Impeccable Skills Reviewer failed #43079 covers this.
4. CGO (Q: 55/100, E: 50/100) — 0/6 successful runs today (3 failure, 3 AR). CI workflow — AR = approval gate, failure = build failure. Separate concerns.
5. Claude Code User Documentation Review (Q: 45/100, E: 40/100) — 1 failure today ([aw] Claude Code User Documentation Review failed #48376). Failed run + timed-out PR triage suggest systemic prompt/scope issue.
6. Design Decision Gate (Q: 48/100, E: 45/100) — 1 AR today. Persistent redesign/deprecation candidate.
7. Matt Pocock Skills Reviewer (Q: 50/100, E: 48/100) — 1 AR today ([aw] Matt Pocock Skills Reviewer failed #48362). Persistent deprecation candidate.

Inactive / Zero-Output Agents

• Deployment Incident Monitor — 3/3 skipped today. No outputs.
• PR Data Prefetch — 1 run, AR. Persistent gate issue.

Quality Analysis

Output Quality Distribution (Jul 27)

• Excellent (80-100): 7 agents (Copilot cloud agent, Auto-Triage, PR Triage, Bot Detection, Avenger, Agentic Maintenance, PR Sous Chef)
• Good (60-79): 3 agents (Workflow Normalizer, PR Description Updater, CWI/CJS when passing)
• Fair (40-59): 4 agents (CGO, Claude Code Doc Review, Design Decision Gate, Matt Pocock)
• Poor (<40): 3 agents (AI Moderator, Q workflow, Agentic Commands)

Issue Quality Observed (Jul 27 sample)

Recent agent-created issues are predominantly failure reports ([aw] X failed). Positive pattern: auto-generated failure issues (#48376, #48375, #48373, etc.) are consistently structured and actionable — good template hygiene. Negative: volume of recurring failed issues suggests underlying engine/infra problems not yet resolved.

Behavioral Patterns

Problematic Patterns ⚠️

• Persistent AR cluster (CLI hang): Design Decision Gate, Matt Pocock, PR Code Quality Reviewer, Test Quality Sentinel, Impeccable Skills Reviewer — all share CLI hang-on-exit root cause. Fix PR fix: reduce post-completion idle watchdog and add cleanup timeouts to prevent Copilot CLI hang on exit #44254 still open as of Jul 8; must verify merge status.
• Agentic Commands / Q chronic AR: 10 and 7 AR respectively in last 100 runs. No meaningful improvement since Jul 8. Quality gate PR Add shared prompt quality gate for plateaued agent-review workflows #43527 appears unmerged.
• CGO failure pattern: Mix of failure (build/test fail) and action_required (GitHub permission gate). These are distinct — failures need code/CI investigation, AR needs org approval. Current tracking conflates them.
• Over-creation of [aw] X failed issues: System is auto-creating one issue per failure for many workflows (48362, 48367, 48372, 48373, 48375, 48376, etc.) — useful for new failures, but risks noise if recurring failures keep filing duplicates without deduplication checks.

Productive Patterns ✅

• Copilot SWE Agent: Producing substantive PRs with clear scope (metadata channel, legacy-security mode). High-quality attribution.
• Agentic Maintenance → Avenger: Coordination appears effective; both stable.
• Auto-Triage → PR Triage pipeline: Consistently successful; clean handoffs.

CI Approval-Pending (Not Agentic Failures)

The following are CI workflows with action_required conclusions — these represent GitHub's PR workflow approval gate, not agentic failures:

Workflow	AR runs (today)	Action needed
CGO	3	Org-level approval for Copilot-bot PR runs
CJS	3	Same
CWI	3	Same
Doc Build - Deploy	2	Same

Recommended fix: Approve Copilot-bot's workflow runs at the org level to eliminate these recurring AR counts.

Prompt Improvement Audit (Bottom Agents)

Agent	Q Score	Primary Deficiency	Evidence
Q workflow	25	Missing self-assessment loop; stale tool guidance	79% AR, no improvement after quality-gate PR; prompt doesn't adapt to gate state
AI Moderator	35	Stale engine config (Codex 404)	Persistent engine failure; prompt/config not updated after engine deprecation
Agentic Commands	40	Generic task framing; low-actionability outputs	33% success rate; 10/15 AR; #43079 long-standing
Design Decision Gate	48	Generic task framing; no quality rubric	100% AR recent; redesign/deprecation candidate
Matt Pocock Skills Reviewer	50	Stale examples + CLI hang (infra)	100% AR; #43309 #43894 persistent; deprecation candidate
Claude Code Doc Review	45	Scope creep; missing success criteria	Failure + timeout; outputs too broad
Impeccable Skills Reviewer	55	CLI hang (infra); stale tool guidance	AR cluster; fix PR #44254 open

Recommendations:

• Matt Pocock Skills Reviewer and Design Decision Gate: Proceed to deprecation if no owner claims them within 14 days. ROI is low; persistent infra issues mask any potential value.
• AI Moderator: Change engine from Codex to copilot (1-line config change). Highest-impact/lowest-effort fix available.
• Q workflow: Prompt needs a pre-flight check that tests whether quality-gate PR is merged before running main logic, to avoid repeated AR.

Recommendations

High Priority

1. Merge CLI hang-on-exit fix (PR fix: reduce post-completion idle watchdog and add cleanup timeouts to prevent Copilot CLI hang on exit #44254) — Unblocks Design Decision Gate, Matt Pocock, PR Code Quality Reviewer, Test Quality Sentinel, Impeccable Skills Reviewer simultaneously. Zero agent-side work required.
2. Change AI Moderator engine to copilot — Replace codex with copilot in workflow config. Expected: restore 80%+ success rate.
3. Approve Copilot-bot workflow runs at org level — Eliminates ~9 spurious CI AR runs per batch (CWI/CJS/CGO/Doc Build).

Medium Priority

4. Merge Q workflow quality gate (PR Add shared prompt quality gate for plateaued agent-review workflows #43527) — Has been open for 7+ days; Q is at 0% success. This is the stated fix.
5. Deprecate Matt Pocock Skills Reviewer and Design Decision Gate — Both are persistent underperformers with no recent traction. Free up monitoring overhead.
6. Add deduplication gate to failure-issue creation — Several [aw] X failed issues are filed repeatedly for the same persistent failures. Add a check for open issues with same title prefix before filing.

Low Priority

7. Refine Claude Code Doc Review scope — Add explicit boundary conditions and success criteria to reduce timeout/failure rate.
8. Consolidate redundant monitoring — Review overlap between Workflow Health Manager, Agent Performance Analyzer, and Issue Monster for potential consolidation.

Trends

Metric	Jul 8	Jul 27	Change
Quality score	63/100	62/100	→ flat
Effectiveness	64/100	61/100	↓ -3
Success rate	36%	~40%	↑ slight
AR rate	41%	~38%	↓ slight
Failure rate	~3%	~5%	↑ CGO/CJS
Open high-pri issues	~95	~100+	↑ accumulating

Overall quality has been plateaued for 19 days. The engine/infra cluster (CLI hang, Codex 404, Q AR) is suppressing scores. Fixing PR #44254 and the AI Moderator engine would provide the largest single-run improvement.

Actions Taken This Run

• Generated this performance report discussion
• No new issues filed (all active problems covered by existing issues — see DO NOT RE-FILE list)
• Updated shared memory files

Next Steps

1. Verify PR fix: reduce post-completion idle watchdog and add cleanup timeouts to prevent Copilot CLI hang on exit #44254 (CLI hang fix) merge status — blocker for 5 agents
2. Verify PR Add shared prompt quality gate for plateaued agent-review workflows #43527 (Q workflow quality gate) merge status — blocker for Q
3. Change AI Moderator engine from codex → copilot
4. Org-level approval of Copilot-bot workflow runs (CWI/CJS/CGO)
5. Evaluate Matt Pocock + Design Decision Gate for formal deprecation

---

Analysis period: 2026-07-08 → 2026-07-27 | Next report: ~2026-08-03
Existing issue backlog: see shared-alerts.md DO NOT RE-FILE list

References:

• §30270561626 — This run (Agent Performance Analyzer)
• §28945138633 — Prior run (Jul 8)
• §28919919125 — Workflow Health Manager Jul 8

Generated by ⚡ Agent Performance Analyzer - Meta-Orchestrator · sonnet46 · 38.6 AIC · ⌖ 9.04 AIC · ⊞ 10.9K · ◷

• expires on Jul 28, 2026, 5:39 AM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-07-28T13:39:04.941Z.

Closed by Workflow