Agent Performance Report — Week of 2026-09-09

[github-actions[bot]]

Executive Summary

Data quality remains the primary limiter this run: the Metrics Collector's 2026-09-01.json snapshot only counted 41/297 workflows as "active" (down from 247 active on 2026-08-22 and vs. 66-154 typical), and per-day ecosystem.overall_success_rate swings from 0.42 to 0.97 across the last 30 days with no accompanying narrative — consistent with the collector's own note that it fell back to a GitHub API scrape after paginated logs were truncated. Treat single-day metrics as noisy; multi-day patterns below use 08-04→09-01 aggregation where possible.

• Workflows with any executed run (30d, executed>0, n≥3): 5 (agentic_commands, copilot, aw-failure-investigator, daily-trajectory-grader-implementer, pr-sous-chef) — all 100% success. Most of the 297 tracked workflow files show zero executed runs in this window, which is expected for command/mention-gated or low-frequency workflows, not a defect.
• Command/mention-gated workflows correctly excluded from failure scoring: q, mattpocock-skills-reviewer, pr-code-quality-reviewer, test-quality-sentinel, design-decision-gate (plus ~55 others repo-wide) are wired to fire on every issue_comment/pull_request_comment/etc. and are stopped by pre_activation unless the command matches — their action_required/skipped counts are designed gating, not activation-refused incidents.
• Prior "deprecation candidate" status for reviewer/gate agents is stale. 2026-09-01 snapshot shows mattpocock-skills-reviewer, impeccable-skills-reviewer, design-decision-gate, pr-code-quality-reviewer, and test-quality-sentinel each ran 1/1 successfully. Per the 2026-09-08 shared-alerts correction, these were already reclassified from "deprecation candidate" to "recovered — monitor." This run's audit (below) evaluates prompt quality directly rather than repeating the stale AR-based verdict.
• No new critical/systemic issue found this run — see Actions Taken.

Bottom-Tier Prompt Audit: Reviewer/Gate Agents (redesign-vs-deprecation candidates)

Matt Pocock Skills Reviewer (mattpocock-skills-reviewer.md)

• Deficiency: stale/fragile tool guidance. Step 3 primarily depends on invoking a separate pr-triage sub-agent and parsing its JSON; the fallback path (inline heuristic) is only exercised when triage errors, so most runs depend on an extra agent hop that isn't visible in current metrics (no pr-triage run data collected).
• Deficiency: generic task framing softened by a solid rubric. The "Success Criteria" section (lines 100-108) is good practice (ties findings to risk+fix, allows noop), but the 5 listed skills (/diagnosing-bugs, /tdd, etc.) are static inline summaries duplicated from an external skill repo — no mechanism to detect if the installed skill version has diverged from the inline guidance baked into the prompt.
• Evidence: 1/1 successful run in the only available data point (2026-09-01); insufficient run volume (n=1) to assess output quality trend.
• Recommendation: Add a lightweight self-check step ("did I cite specific risk + fix for every inline comment?") before Step 6 submit, and log a warning line if pr-triage fallback was used, so metrics can start distinguishing triage-agent health from reviewer prompt quality.

Impeccable Skills Reviewer (impeccable-skills-reviewer.md)

• Strength: Already has explicit "Success Criteria" (high-signal, changed-lines-only, noop over filler) and a clean change-type → mode mapping table (lines 108-116) with a fallback if the skill file is missing. This is close to the target structure other agents should copy.
• Deficiency: low run volume prevents quality validation. Only 1 executed run in the entire 30-day window (2026-09-01) — too sparse to confirm the rubric is producing actionable output in practice.
• Recommendation: No prompt rewrite needed; instead recommend Workflow Health Manager confirm the pull_request: types: [ready_for_review] trigger is actually firing on PRs (check whether draft→ready transitions are rare in this repo, which would explain the low volume without indicating a broken agent).

Design Decision Gate (design-decision-gate.md)

• Strength: Best-structured prompt in the audited set — explicit turn budget table, "Mandatory Stopping Criteria," and a "Gate Quality Bar" section functioning as a self-assessment rubric (evidence citation + one next action + stop discipline).
• Deficiency: missing-data self-assessment loop. The fallback rules (lines ~196-204) correctly degrade to GitHub MCP calls when prefetch files are missing, but there's no logged signal distinguishing "prefetch worked, gate skipped because no ADR needed" from "prefetch failed, fell back" — both currently just emit noop/add-comment with no distinguishing metric field, making it hard to separate legitimate no-ops from degraded runs in metrics.
• Evidence: 1/1 successful run (2026-09-01); the workflow's own evals block (decision-justified, action-taken, adr-check-performed) is a good built-in quality gate already — recommend surfacing these eval results into the shared metrics pipeline instead of only trusting run success/failure.
• Recommendation: No deprecation — this agent is the model to emulate. Redesign scope: none required; operational scope: wire its evals results into metrics-collector so quality (not just executed/success) is tracked over time.

Summary verdict: None of the three audited agents currently show evidence supporting deprecation. All three have real design merit (Design Decision Gate is the strongest); the shared-memory "deprecation candidate" label from July is stale and was already partially corrected on 2026-09-08. Recommend closing out the label entirely rather than re-flagging.

Data Quality Issue Affecting This Analysis

• metrics/latest.json (2026-09-01) reports active_workflows: 41 vs. 247 the day before (08-22) and 66-154 on most other sampled days — an 83% single-day drop with the collector's own note admitting a "GitHub API fallback after paginated logs were truncated." This makes day-over-day trend claims (e.g., "quality improved") unreliable without matching the drop against actual run history.
• metrics/daily/2026-01-06.json, 2026-01-07.json, 2026-01-08.json, 2026-01-16.json–2026-01-18.json are empty placeholders ({"note": "Jan 2026 metrics unavailable - GitHub API not accessible"}), consistent with summary.json. No action needed — already self-documented.
• Recommendation for Metrics Collector maintainers: add a sanity check that rejects/re-flags a daily snapshot when active_workflows moves >50% day-over-day, so downstream analyzers (this workflow, Campaign Manager, WHM) don't inherit noisy single-day anomalies as trend signal.

Behavioral Patterns

• Command/mention gating (expected, not a defect): q, mattpocock-skills-reviewer, pr-code-quality-reviewer, test-quality-sentinel, design-decision-gate, and ~55 other slash-command/mention-triggered workflows fire on every relevant issue/PR/discussion comment event and self-terminate in pre_activation unless invoked. Scored only on activated runs per policy.
• CI approval-pending (expected, not agentic AR): CWI, CGO, CI show action_required entries in some daily snapshots consistent with GitHub Actions' PR-workflow-approval gate, not an agent failure.

Recommendations

High Priority

1. Close the stale "deprecation candidate" label on Matt Pocock, Impeccable, and Design Decision Gate in shared-alerts.md — the label predates the 2026-09-08 correction and this run's prompt audit found no deprecation-supporting evidence.
2. Metrics Collector data-quality guard: flag/reject daily snapshots with >50% day-over-day swing in active_workflows before they seed downstream analysis (see Data Quality section).

Medium Priority

1. Add a fallback/degraded-mode log signal to Design Decision Gate and Matt Pocock Skills Reviewer so metrics can distinguish "legitimate noop" from "prefetch failure fallback."
2. Wire Design Decision Gate's built-in evals (adr-check-performed, action-taken, decision-justified) into the shared metrics pipeline as a quality signal, since it already self-assesses better than most agents.

Actions Taken This Run

• Reviewed shared memory (agent-performance-latest.md, campaign-manager-latest.md [empty], workflow-health-latest.md, shared-alerts.md), metrics/latest.json, and 21 daily metric snapshots (2026-01 through 2026-09-01).
• Audited prompts for the three flagged redesign/deprecation candidates (Matt Pocock, Impeccable, Design Decision Gate) directly against their .md source — no deprecation evidence found; recommending the stale label be dropped.
• No new improvement issues filed — the two actionable items above are metrics-pipeline/process changes better owned by Workflow Health Manager and Metrics Collector maintainers than by filing an agent-prompt issue against a working prompt.
• Did not create a full weekly performance discussion with agent-by-agent rankings this run: current metrics snapshot quality (41/297 active, single-day swings) is too unreliable to responsibly score and rank agents without repeating the noise as fact. Flagging this gap explicitly rather than fabricating rankings from bad data.

Next Steps

1. Wait for Metrics Collector to confirm/fix the 2026-09-01 active-workflow undercount before next full ranking pass.
2. Re-run full Phase 2-4 ranking once ≥3 consecutive clean daily snapshots are available.
3. Track whether the "deprecation candidate" label reappears for Matt Pocock/Impeccable/Design Decision Gate elsewhere in shared memory and correct at source.

---

Analysis period: 2026-08-04 to 2026-09-01 (metrics), prompt audit as of 2026-09-09
Next report: next scheduled run

Generated by ⚡ Agent Performance Analyzer - Meta-Orchestrator · copilot · auto · 168 AIC · ⌖ 7.7 AIC · ⊞ 18.2K · ◷

• expires on Sep 10, 2026, 5:08 AM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-09-10T13:08:09.558Z.

Closed by Workflow