Agent Performance Report - 2026-08-05

[github-actions[bot]]

Agent Performance Report — 2026-08-05

Executive Summary

• Workflows in metrics window: 31 active / 276 total (partial 24h sample, 50 most-recent runs; see caveat below)
• Weighted success rate: 90% (of the 50-run sample)
• Total safe outputs recorded: 0 across all 31 workflows — this figure is a known instrumentation gap, not real agent inactivity (see Critical Finding below)
• Top token/cost consumers: Daily Observability Report for AWF Firewall/MCP Gateway (851K tokens, $18), PR Code Quality Reviewer ($381, 3 runs, 1 failure), Failure Investigator ($241), Avenger ($246)
• Persistent bottom-3 redesign/deprecation candidates (per standing initiative): Design Decision Gate 🏗️, Matt Pocock Skills Reviewer, Impeccable Skills Reviewer — all currently green in the sampled runs, but historically the most chronic failure/churn cluster in the repo (60+ auto-filed failure issues since June)

🚨 Critical Finding: Metrics Collector cannot see safe-output data this run

metrics/latest.json and every metrics/daily/*.json file in shared memory report safe_outputs: {issues_created: 0, prs_created: 0, comments_added: 0, discussions_created: 0} for all 31 active workflows, and ecosystem.total_safe_outputs: 0. The collector's own collection_note explains why:

"agenticworkflows logs tool timed out for full -1d window (>60s at count>=100); data limited to most recent 50 runs... Safe-output/engagement counts unavailable from this data source in this run."

This has repeated on at least two of the last three collection days (Aug 4, Aug 5). Ground-truth gh issue list shows dozens of real issues/comments created by agents in the same window (e.g. #50506 Safe Outputs Conformance, #50516/#50515 Code Quality issues, #50502 model-resolution audit, #50519 Failure Investigator), so this is a data-collection defect, not an ecosystem-wide output freeze. Every downstream quality/effectiveness score computed purely from this snapshot (merge rate, engagement, completion rate) is currently unreliable. I filed a new issue for this (see below) since no open issue currently tracks the recurring truncated-window failure mode.

Performance Rankings (partial data only — token/success proxy)

Highest resource consumers (cost proxy, not quality)

Workflow	Runs	Success	Tokens	Cost ($)
Daily Observability Report (AWF Firewall/MCP GW)	1	100%	851,253	18.38
PR Code Quality Reviewer	3	67%	207,745	380.90
Avenger	2	100%	26,194	245.92
Failure Investigator (6h)	1	100%	64,227	240.96
Daily Firewall Logs Collector	1	100%	37,185	189.88
Daily Documentation Healer	1	100%	20,777	162.98

Low-cost, stable agents (Q proxy from run history)

Workflow	Runs	Success
Smoke Aider	4	100%
Auto-Triage Issues	1	100%
Code Scanning Fixer	1	100%
Release	1	100%

Currently degraded

Workflow	Runs	Success	Note
PR Code Quality Reviewer	3	67%	1 failure in sample (separate from historical CLI-hang cluster, which appears resolved — 2 of 3 latest runs succeeded)
Smoke Crush	3	33%	Persistent — do not re-file, tracked historically

Prompt Improvement Initiative — Bottom-3 Redesign/Deprecation Candidates

Per the standing initiative, these three remain the chronic underperformers even though their most recent sampled runs are green:

1. Design Decision Gate 🏗️ (.github/workflows/design-decision-gate.md)

   • Deficiency: stale tool/guidance — the workflow still hard-codes a bash pre-fetch step (Pre-fetch ADR gate PR context) that fails before the agent even starts (open issue [aw-failures] Design Decision Gate: 'Pre-fetch ADR gate PR context' step fails before agent starts #49144), and a separate failure mode exhausts the LLM invocation cap on shell-expanded file paths ([aw-failures] Design Decision Gate exhausts LLM invocation cap after permission-denied retries on shell-expanded file paths #49096). Most recent run (§31007716082, today) hit Engine Max Runs Exceeded — a third distinct failure mode layered on top of the first two.
   • Evidence: [aw] Design Decision Gate 🏗️ failed #50552 (today's failure, max-runs exceeded), [aw-failures] Design Decision Gate: 'Pre-fetch ADR gate PR context' step fails before agent starts #49144, [aw-failures] Design Decision Gate exhausts LLM invocation cap after permission-denied retries on shell-expanded file paths #49096, [aw-failures] Design Decision Gate: push_to_pull_request_branch rejects fixes touching actions/setup/** (outside allowed-files) #50142 (push-to-pull-request-branch rejects fixes outside allowed-files).
   • Recommendation: A deprecation PR (Deprecate auto-run paths for Matt Skills Reviewer and Design Decision Gate to stop chronic failure-issue churn #48730) was previously proposed and closed unmerged (confirmed via [Agent Perf] Deprecation PR #48730 for Matt Pocock Skills Reviewer / Design Decision Gate was closed unmerged — chronic churn co [Content truncated due to length] #49577) — no decision has landed. Either (a) merge a scoped-down deprecation (drop the on.pull_request auto-trigger, keep slash_command-only invocation) or (b) fix the three known bugs (bash pre-fetch failure, shell-expansion invocation-cap exhaustion, allowed-files scope) before re-enabling auto-run. Recommend (a) given 3 independent bug clusters and no owner progress in 5+ weeks.

2. Matt Pocock Skills Reviewer (.github/workflows/mattpocock-skills-reviewer.md)

   • Deficiency: generic task framing + missing self-assessment loop. The prompt lists 7 skills but gives the model no rubric for when to skip a low-value review; it delegates skill selection to a pr-triage sub-agent (Step 3) but has no fallback instruction if that sub-agent's classification is low-confidence, which historically produced boilerplate reviews on trivial diffs.
   • Evidence: deprecation was already proposed and closed unmerged ([deep-report] Evaluate Matt Pocock Skills Reviewer for deprecation (100% activation-refused, top API burner, redundant) #47608 evaluation → Deprecate auto-run paths for Matt Skills Reviewer and Design Decision Gate to stop chronic failure-issue churn #48730 deprecation PR, closed 2026-07-28; [Agent Perf] Deprecation PR #48730 for Matt Pocock Skills Reviewer / Design Decision Gate was closed unmerged — chronic churn co [Content truncated due to length] #49577 confirms it never merged); 7+ new [aw] ... failed issues since ([aw] Matt Pocock Skills Reviewer failed #48774, [aw] Matt Pocock Skills Reviewer failed #48844, [aw] Matt Pocock Skills Reviewer failed #49004, [aw] Matt Pocock Skills Reviewer failed #49121, [aw] Matt Pocock Skills Reviewer failed #49270, [aw] Matt Pocock Skills Reviewer failed #49405, [aw] Matt Pocock Skills Reviewer failed #49954, [aw] Matt Pocock Skills Reviewer failed #50238, [aw] Matt Pocock Skills Reviewer failed #50431).
   • Recommendation: Land the previously-drafted deprecation (or re-scope to slash_command-only like Design Decision Gate) rather than re-attempting prompt fixes on a workflow already flagged 3× as redundant with Impeccable/PR Code Quality Reviewer.

3. Impeccable Skills Reviewer (.github/workflows/impeccable-skills-reviewer.md)

   • Deficiency: low-actionability outputs during failure — recent history shows repeated "hit AI credits rate limit" ([aw] Impeccable Skills Reviewer hit AI credits rate limit #49380, [aw] Impeccable Skills Reviewer hit AI credits rate limit #49661, [aw] Impeccable Skills Reviewer hit AI credits rate limit #49779, [aw] Impeccable Skills Reviewer hit AI credits rate limit #49979) and "produced no safe outputs" ([aw] Impeccable Skills Reviewer produced no safe outputs #48494, [aw] Impeccable Skills Reviewer produced no safe outputs #48118) closures rather than genuine reviews; the prompt has no credit-budget circuit breaker or early-exit check before starting expensive skill discovery (find /tmp/gh-aw/.github/skills ...).
   • Evidence: overlaps near-100% with Matt Pocock Skills Reviewer and PR Code Quality Reviewer on trigger (pull_request: ready_for_review) and mission ("select relevant skills, review changed lines, submit review") — 3 workflows doing materially the same job on every PR.
   • Recommendation: Consolidate into a single skills-review workflow (already partially recommended in closed issues [deep-report] Consolidate shared GitHub reads across the three skills-reviewer workflows (~10k API quota/day) #47345, [deep-report] Reduce GitHub API consumption of the PR review-agent cluster via caching #45290, [deep-report] Scope PR-review workflows with path filters to cut redundant API consumption #46872, [Code Quality] Cache PR file-lists and diffs in top-4 review workflows to reduce REST API consumption #48089) that runs skill-selection once and dispatches to Matt Pocock / Impeccable skill sets conditionally, instead of running 3 separate full-cost agent invocations per PR.

Cross-cutting pattern: all three candidates share the same root cause — deprecation/consolidation decisions were drafted (issues + PRs) but never merged. This is a decision-execution gap, not a prompt-quality gap in isolation. Recommend a maintainer/Copilot SWE Agent task to land PR #48730's intent (or a fresh equivalent) rather than another round of failure-issue triage.

Behavioral Patterns

• Coordination gap, not model gap: the recurring theme across the bottom-3 is "fix was proposed, PR closed unmerged, churn resumed" ([Agent Perf] Deprecation PR #48730 for Matt Pocock Skills Reviewer / Design Decision Gate was closed unmerged — chronic churn co [Content truncated due to length] #49577) — a process/ownership issue more than an agent-quality issue.
• PR Code Quality Reviewer shows partial recovery (2/3 green in latest sample) after historical CLI-hang-on-exit issues — worth reconfirming next cycle before re-flagging.
• Metrics blind spot: current snapshot cannot detect over/under-creation, duplication, or engagement patterns because safe_outputs and engagement are zeroed by the collector's truncated-window bug — noted above, not re-analyzed here to avoid drawing false conclusions from zeroed data.

Recommendations

High Priority

1. Fix Metrics Collector's 50-run/context deadline exceeded truncation so safe-output/engagement fields stop reporting false zeros — blocks all downstream quality scoring. Issue filed this run.
2. Land or formally re-close the Design Decision Gate / Matt Pocock deprecation intent (Deprecate auto-run paths for Matt Skills Reviewer and Design Decision Gate to stop chronic failure-issue churn #48730 lineage) — stops recurring low-value failure-issue creation and unblocks the fleet quality plateau tracked since [agent-performance] Agent Quality Plateau: Q=61/E=62 for 3+ Weeks — Prompt Improvement Initiative #43387/[Agent Perf] Deprecation PR #48730 for Matt Pocock Skills Reviewer / Design Decision Gate was closed unmerged — chronic churn co [Content truncated due to length] #49577.
3. Consolidate the 3 PR skills-reviewers (Matt Pocock, Impeccable, PR Code Quality Reviewer) into one gated workflow — reduces per-PR cost (currently ~$380–450 combined per PR cycle in this sample) and removes duplicate review noise.

Medium Priority

4. Re-baseline agent quality/effectiveness scores once metrics collection is fixed — current 90% success figure is real (from run status, not from the broken safe-output field) but completion/merge/engagement scores cannot be computed this cycle.

Actions Taken This Run

• Filed 1 new issue: Metrics Collector safe-output/engagement data gap (truncated-window blind spot)
• No duplicate issues filed — verified via gh api search/issues that Design Decision Gate, Matt Pocock, and Impeccable Skills Reviewer failure modes are already tracked in existing open issues ([aw-failures] Design Decision Gate: 'Pre-fetch ADR gate PR context' step fails before agent starts #49144, [aw-failures] Design Decision Gate exhausts LLM invocation cap after permission-denied retries on shell-expanded file paths #49096, [aw-failures] Design Decision Gate: push_to_pull_request_branch rejects fixes touching actions/setup/** (outside allowed-files) #50142, [aw] Impeccable Skills Reviewer hit AI credits rate limit #49380, etc.)
• Skipped creating separate deprecation-nudge issues for the bottom-3, since [Agent Perf] Deprecation PR #48730 for Matt Pocock Skills Reviewer / Design Decision Gate was closed unmerged — chronic churn co [Content truncated due to length] #49577 (closed 2026-08-03) already fully documents the unmerged-PR situation — re-filing would duplicate it

Next Steps

1. Maintainer/Copilot SWE Agent: revive or re-scope PR Deprecate auto-run paths for Matt Skills Reviewer and Design Decision Gate to stop chronic failure-issue churn #48730 (Design Decision Gate + Matt Pocock deprecation)
2. Fix Metrics Collector truncated-window issue (filed this run)
3. Re-run this analysis once metrics collection is restored to get real quality/effectiveness/engagement scores
4. Track PR Code Quality Reviewer's recovery (2/3 green) over the next 2–3 cycles before re-flagging

---

Analysis based on: metrics/latest.json (2026-08-05T03:38:39Z partial snapshot), gh issue list/gh api search/issues ground-truth cross-checks, and current workflow source files.
Note: quantitative quality/effectiveness scores are omitted this cycle because the safe-output/engagement metrics field was empty due to the collector bug described above; scoring will resume once that data is restored.

Generated by ⚡ Agent Performance Analyzer - Meta-Orchestrator · auto · 117.6 AIC · ⌖ 7.84 AIC · ⊞ 17K · ◷

• expires on Aug 6, 2026, 5:28 AM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-08-06T13:28:24.746Z.

Closed by Workflow