Agent Performance Report - Week of 2026-08-13

[github-actions[bot]]

Executive Summary

• Agents analyzed: 63 active workflows (top 20 by run volume/output reviewed in depth)
• Total outputs reviewed: safe items across sample ≈ 210 (PR Sous Chef 101, PR Triage Agent 21, Matt Pocock 19, Contribution Check 17, Impeccable 16, Test Quality Sentinel 12, Issue Monster 12)
• Average quality score (proxy = success_rate×100) across sampled agents: ~64/100
• Average effectiveness score: ~62/100 (weighted down by 0-token/0-output failed runs)
• Top performers: Contribution Check (100%, 4/4), Auto-Triage Issues (100%, 2/2), Outcome Collector (100%, 2/2), AI Moderator (88.9%, 8/9)
• Needs improvement: Test Quality Sentinel (38.9%, 18 runs), Ponytail Reviewer (33.3%, 15 runs), Matt Pocock Skills Reviewer (54.5%, 11 runs), Impeccable Skills Reviewer (64.3%, 14 runs), Design Decision Gate (81.8% but down from 100% Aug 6), PR Sous Chef (86.4%, recovering from 16% crash Aug 11)

Collection note carried from metrics: 2026-08-13 snapshot is partial (logs capped at 200 runs/workflow, ~12h window gap; per-workflow issue/PR/comment breakdown unavailable via API fallback for 63 active workflows). Scores below use success_rate as the primary quality/effectiveness proxy where direct engagement data was not computed.

Performance Rankings

Top Performing Agents 🏆 (≥2 runs, highest success rate)

1. Contribution Check — 100% (4/4 runs), 17 safe items, $238.53 cost — consistent, high output volume per run.
2. Auto-Triage Issues — 100% (2/2), 4 safe items, low cost ($8.66) — efficient.
3. Outcome Collector — 100% (2/2), 2 safe items.
4. AI Moderator — 88.9% (8/9 runs) — high token cost ($813,684 tokens / $84.17) but 0 safe items recorded this window; recovering from earlier Codex-engine 404 (persistent issue [aw] Daily Sub-Agent Model Resolution Audit failed #43335, [aw-failures] Daily Sub-Agent Model Resolution Audit 100% red — Codex gpt-5-codex-alpha-2025-11-07 404s (same alpha-snapshot d [Content truncated due to length] #42033). Watch: high token burn with zero output is a red flag for wasted spend even as success rate looks healthy.
5. PR Sous Chef — 86.4% (22 runs), 101 safe items (highest output volume of any agent this week) — recovered from a 16% crash on Aug 11 (25 runs, only 4 succeeded) back to 86% by Aug 13. Volatility warrants a stability watch even though current numbers are good.

Agents Needing Improvement 📉 (bottom by success rate, ≥2 runs)

1. Test Quality Sentinel — 38.9% (18 runs, only 7 succeeded), 12 safe items, 124K tokens / $292.92 cost.

   • Trend: 100% (Aug 4–6) → 93.8% (Aug 7) → 100% (Aug 8–10) → 83.3% (Aug 11) → 38.9% (Aug 13) — sharp regression, not a slow drift.
   • Prompt deficiency: missing self-assessment loop. The 432-line prompt has an elaborate pre-fetch/cache-validation script (lines 1-60) but the review-generation section provides no explicit gate for "is this PR too large / is context truncated / did the sub-agent step fail silently" — a fragile external dependency (cli-proxy, cache invalidation logic) is the likely single point of failure driving run failures, and the prompt has no fallback path when pre-fetch cache is stale or absent.
   • Evidence: elaborate bash cache-comparison logic in the prompt body increases the chance of hard failures (set -euo pipefail) rather than falling back gracefully; failed runs likely error out during data prep, before any review is produced.
   • Recommendation: add an explicit fallback ("if cache miss and re-fetch fails, call noop with reason") and a completion self-check before finishing.

2. Ponytail Reviewer — 33.3% (15 runs, only 5 succeeded), just 2 safe items across 15 runs (huge under-output relative to run volume).

   • Prompt deficiency: generic task framing + low-actionability outputs. The 68-line prompt is a thin wrapper around a single external skill (ponytail-review) with almost no repo-specific guidance, no success criteria section, and no fallback when the skill produces nothing — compare to Matt Pocock/Impeccable, which both define explicit "Success Criteria" sections. Ponytail has none.
   • Evidence: 2 safe items from 15 runs (13% output rate) suggests the skill is either silently no-opping most of the time or failing before producing review comments — no visible completeness/self-check gate.
   • Recommendation: add a "Success Criteria" section, an explicit noop-vs-silence rule, and instrumentation to distinguish "no complexity issues found" (valid noop) from "skill failed to run" (error).

3. Matt Pocock Skills Reviewer — 54.5% (11 runs, only 6 succeeded), 19 safe items, $291.35 cost.

   • Trend: 100% (Aug 4–6, 9) → 83.3–77.8% (Aug 7, 11) → 54.5% (Aug 13).
   • Prompt deficiency: stale examples / stale tool guidance. The prompt still instructs "Invoke the pr-triage agent and capture its JSON response" as a hard dependency for skill selection (Step 3) — any failure/timeout in that sub-agent call likely cascades into a full run failure, and there's no documented fallback if pr-triage doesn't return valid JSON.
   • Recommendation: add a fallback skill-selection heuristic (e.g., file-extension-based default) if the pr-triage sub-agent call fails or times out, so one flaky dependency doesn't sink the whole review.

4. Impeccable Skills Reviewer — 64.3% (14 runs, 9 succeeded), 16 safe items, $223.39 cost.

   • Trend: 100% (Aug 5–6, 9–10) → 68.8% (Aug 7) → 91.7% (Aug 8) → 64.3% (Aug 11, 13) — recurring dips correlated with Aug 7/11/13, same days Matt Pocock and Test Quality Sentinel also regressed. This is a shared pattern, not agent-specific.
   • Prompt deficiency: has a Success Criteria section (good) but missing self-assessment loop for the pre-fetch verification — Step 1 only checks file existence, not staleness against the current head SHA (unlike Matt Pocock/Test Quality Sentinel, which do SHA comparison). A stale cache could silently produce a review against the wrong diff.
   • Recommendation: add head-SHA validation to the pre-fetch check, matching the more defensive pattern used by sibling reviewers.

5. Design Decision Gate — 81.8% (22 runs) but down from 100% (Aug 6) — volatile, ranging 50–92% week to week.

   • Prompt deficiency: low-actionability outputs risk — this agent gates merges via ADR enforcement and uses push-to-pull-request-branch; when it fails mid-run, PRs may get stuck without a clear ADR draft or a clear explanation of what's missing.
   • Recommendation: verify success/failure is not being caused by the same "CLI hang-on-exit" cluster that hit Impeccable/PR Code Quality/Test Quality Sentinel (fix PR fix: reduce post-completion idle watchdog and add cleanup timeouts to prevent Copilot CLI hang on exit #44254 merged 2026-07-08 — confirm this fix is still deployed and not regressed for Design Decision Gate specifically, since it's on the claude engine, not copilot, and would not have benefited from that Copilot-CLI-specific fix).

Shared Root-Cause Pattern (P1 — affects 3+ agents)

"Aug 7 / Aug 11 / Aug 13 correlated failure spikes" — Test Quality Sentinel, Matt Pocock, and Impeccable Skills Reviewer all show simultaneous success-rate dips on the same three dates (Aug 7, Aug 11, Aug 13), while Design Decision Gate (different engine: claude vs copilot) shows a milder, offset pattern. This points to a shared Copilot-engine or shared-import failure (all three use shared/pr-review-base.md + shared/pr-diff-data-fetch.md + cli-proxy: true + copilot engine) rather than independent per-agent prompt regressions. Recommend the Workflow Health Manager investigate shared/pr-diff-data-fetch.md and the Copilot cli-proxy gateway for infra-level flakiness before assuming prompt-only fixes will resolve it.

Inactive/Zero-Output Agents (window: latest snapshot)

• PR Code Quality Reviewer: 21 runs, 76.2% success, but 0 tokens and 0 safe items recorded — likely a metrics/logging gap rather than true inactivity; needs verification, not a prompt fix.
• Daily Regulatory Report Generator: 0% (3 runs, 0 tokens) — full failure.
• Code Scanning Fixer: 0% (2/2 failed), 21K tokens burned with no output — wasted spend.
• Daily Code Debt Cleanup — Aider, Daily BYOK Ollama Test, Daily Documentation Healer, Daily Ambient Context Optimizer, Daily CLI Performance Agent, Daily Code Metrics and Trend Tracking Agent: all 0% success, 0 tokens — consistent with known persistent engine/infra issues already tracked in shared-alerts.md (BYOK Ollama api-proxy 503, etc.) — not re-filing, per Do-Not-Re-File list.

Behavioral Patterns

Problematic Patterns ⚠️

• PR Sous Chef — inconsistency: swung from 100% (Aug 9–10) to 16% (Aug 11, 25 runs) back to 86% (Aug 13) despite being the single highest-output agent (101 safe items this week). Volatility at this output volume is a bigger operational risk than a chronically low agent.
• Test Quality Sentinel, Matt Pocock, Impeccable — inconsistency (shared): correlated multi-agent dips on Aug 7/11/13 as detailed above — classified as a shared infra/import issue, not isolated per-agent drift.
• Ponytail Reviewer — under-creation: 2 safe items across 15 runs is disproportionately low output relative to run count; can't distinguish "nothing to flag" from "silently failing."
• AI Moderator — resource inefficiency: 813,684 tokens consumed across 9 runs with 0 recorded safe items — high token burn without demonstrable output; flag for cost review even though success_rate looks fine.

Productive Patterns ✅

• PR Triage Agent + Matt Pocock/pr-triage sub-agent: Matt Pocock explicitly delegates change-type classification to a pr-triage sub-agent — good separation of concerns, though currently a single point of failure (see deficiency above).
• Contribution Check + Auto-Triage Issues + Outcome Collector: all 100% success this week, low-to-moderate cost, consistent output — solid baseline agents worth using as template references for other reviewers' Success Criteria structure.

Coverage / Redundancy Notes

• Four PR-review-focused skill agents (Matt Pocock, Impeccable, Ponytail, Test Quality Sentinel) all import shared/pr-review-base.md and shared/pr-diff-data-fetch.md and run on ready_for_review — this shared-infra concentration explains the correlated failure pattern above and is a redundancy/coordination point worth consolidating pre-fetch logic further (e.g., a single shared "PR data freshness" self-check helper all four import, rather than each re-implementing SHA-comparison logic slightly differently).

Recommendations

High Priority

1. Investigate shared PR-review infra flakiness (Aug 7/11/13 correlated dips) — affects Test Quality Sentinel, Matt Pocock, Impeccable Skills Reviewer. Check shared/pr-diff-data-fetch.md and Copilot cli-proxy gateway health before any prompt-only fix. Coordinate with Workflow Health Manager.
2. Add fallback/self-assessment gates to Test Quality Sentinel's pre-fetch script — replace hard set -euo pipefail failures with a graceful noop fallback when cache/pre-fetch data is unavailable or stale.
3. Add a Success Criteria section + noop-vs-silence distinction to Ponytail Reviewer — currently the thinnest prompt of the PR-review family (68 lines, no success criteria), producing implausibly low output (2/15 runs).
4. Add fallback skill-selection heuristic to Matt Pocock Skills Reviewer — remove hard dependency on the pr-triage sub-agent call succeeding.

Medium Priority

1. Add head-SHA freshness validation to Impeccable Skills Reviewer's pre-fetch check (currently only checks file existence, not staleness).
2. Verify PR Code Quality Reviewer's 0-token/0-safe-item metric gap — likely a metrics collection issue, not an agent failure, given a 76.2% run success rate.
3. Review AI Moderator's token efficiency — 813K tokens with 0 recorded outputs across 9 runs warrants a cost/output audit.
4. Monitor PR Sous Chef stability — investigate the Aug 11 crash (16% success, 25 runs) to prevent recurrence given its high output volume.

Low Priority

1. Confirm Design Decision Gate is unaffected by the Copilot-CLI-hang fix window (it runs on claude, not copilot) since its volatility profile differs from the other three PR reviewers.

Trends

• Full trend/engagement metrics (issues/PRs/comments breakdown, merge rates) were not computable this run — the 2026-08-13 metrics snapshot is explicitly marked collection_status: partial (logs capped at 200 runs/workflow per policy; ~12h window gap; per-workflow breakdown unavailable for 63 active workflows via API fallback). Only aggregate success_rate and total_safe_items proxies were used for scoring.
• Daily success-rate history for the four PR-review agents (Aug 4–13) is included above and shows a clear correlated regression on Aug 7/11/13 vs. a stable Aug 4–6, 8–10 baseline.

Actions Taken This Run

• No new issues filed — all identified systemic issues either duplicate persistent, already-tracked items (CLI hang-on-exit cluster, fixed by PR fix: reduce post-completion idle watchdog and add cleanup timeouts to prevent Copilot CLI hang on exit #44254 merged 2026-07-08) or represent new prompt-level findings without a corresponding numbered issue yet in the Do-Not-Re-File list.
• Recommend Workflow Health Manager or a follow-up run open one issue for "shared PR-review infra flakiness Aug 7/11/13" if it recurs beyond this reporting window, since it spans 3 agents and isn't yet tracked as a discrete item.
• This report itself is the primary output; no discussion tool was configured to be checked as available in this run's tool set beyond the noop/report path, so findings are being delivered via this response — see note below on tool call.

Next Steps

1. Coordinate with Workflow Health Manager to check shared/pr-diff-data-fetch.md / Copilot cli-proxy gateway health for the Aug 7/11/13 correlated dip.
2. Apply the four prompt fixes above (Test Quality Sentinel fallback gate, Ponytail success criteria, Matt Pocock skill-selection fallback, Impeccable SHA freshness check).
3. Audit AI Moderator token spend vs. output.
4. Re-run this analysis once the 2026-08-13 metrics collection gap (12h missing window) is resolved for a fuller picture.

---

Analysis period: 2026-08-04 to 2026-08-13 (metrics/daily snapshots)
Next report: next scheduled weekly run

Generated by ⚡ Agent Performance Analyzer - Meta-Orchestrator · auto · 42.7 AIC · ⌖ 11.2 AIC · ⊞ 16.4K · ◷

• expires on Aug 14, 2026, 5:06 AM UTC-08:00