[prompt-clustering] Prompt Clustering Analysis - 2026-10-21

[github-actions[bot]]

Summary

Analysis Period: Last 30 days (2026-09-21 to 2026-10-21)
Total Tasks Analyzed: 630
Clusters Identified: 6
Overall Success Rate: 79.7% (502 merged / 95 closed / 33 open)

The merge rate dropped to 79.7% today, down 3.6 points from Oct 18's 83.3% and back near the 75–78% band last seen in early October (Sep 29 – Oct 4), after two weeks holding in the 81–85% range.

Full Analysis Report

General Insights

• Most Common Task Type: Cluster 1 — Safe-Output, Agent Session & MCP Validation (203 tasks, 32.2% of total)
• Highest Success Rate: Cluster 2 — Repository Checkout, Workflow Targeting & Regression Coverage (83.5% merged)
• Most Complex Tasks: Cluster 4 — gh-aw CLI, Firewall & Version Pinning (114.3 avg files changed — the widest diff footprint by far)
• Outliers: 0 clusters crossed the formal flag (≥15 PRs and ≥10 points below the 79.7% overall rate). Cluster 4 is the closest near-miss at 70.3% (a 9.4-point gap) — see root-cause section below.
• Workflow metrics: aw_info.json turn/cost data was not collected for this run (no workflow-logs directory present in the sandbox); this analysis relies on PR-level signals only (comments, reviews, files changed), consistent with prior runs.

Cluster Analysis

Cluster 1: Safe-Output, Agent Session & MCP Validation

• Size: 203 tasks (32.2% of total)
• Success Rate: 79.3%
• Avg Comments / Reviews / Files Changed: 6.6 / 4.3 / 47.3
• Top Keywords: agent, safe, output, detection, job, add, failure, mcp
• Characteristics: The largest cluster by far — Docker-runtime migration, safe-output setting preservation, engine auth/token handling, and MCP tool-exposure fixes for Copilot SDK sessions.
• Example PRs: Migrate agentic workflows to Docker runtime #62406, Preserve imported engine auth with root engine preferences #62413, Add github-token-for-extra-empty-commit: none to keep GH_AW_CI_TRIGGER_TOKEN out of compiled workflows #62416, Expose overridden web_fetch tool to Copilot SDK sessions #62417, Fix Codex engine.config heredoc indentation #62420

---

Cluster 2: Repository Checkout, Workflow Targeting & Regression Coverage

• Size: 176 tasks (27.9% of total)
• Success Rate: 83.5% (highest of all clusters)
• Avg Comments / Reviews / Files Changed: 7.1 / 4.0 / 38.9
• Top Keywords: pr, repository, workflow, checkout, coverage, target, add, regression
• Characteristics: Branch/target validation, checkout-credential hardening, concurrency-group fixes, and regression-coverage additions — mechanical, well-scoped work that converts reliably.
• Example PRs: Validate Git branch arguments before command execution #62378, Add an opt-in shell-free go-repository tool profile for Copilot SDK workflows #62412, Remove push_repo_memory concurrency group so fan-out memory writes are not dropped #62414, Add Dependency Analysis gallery page #62418, Preserve step-installed skills during base-branch restore; allow Claude's Skill tool #62419

---

Cluster 3: Model Routing & Engine Configuration

• Size: 114 tasks (18.1% of total)
• Success Rate: 78.1%
• Avg Comments / Reviews / Files Changed: 7.7 / 4.9 / 53.3
• Top Keywords: model, copilot, routing, claude, models, pi, codex, awf
• Characteristics: Engine-routing changes, CLI version pins for Copilot/Gemini/opencode, and model-specific configuration across the Copilot/Claude/Codex/pi engines — includes several urgent "pin back" regression fixes.
• Example PRs: Prevent Copilot CLI staging for Gemini ARC/DinD workflows #62415, Pin Copilot CLI default back to 1.0.83 to fix instant HTTP 400 regression in 1.0.85 #62421, Pin Copilot CLI to 1.0.80 to avoid MCP regressions #62482, Migrate Daily Reliability Review off the hanging opencode engine #62490, Update default CLI and service versions #62497

---

Cluster 4: gh-aw CLI, Firewall & Version Pinning

• Size: 64 tasks (10.2% of total)
• Success Rate: 70.3% (lowest of all clusters — near-miss outlier)
• Avg Comments / Reviews / Files Changed: 5.5 / 2.7 / 114.3 (most files changed of any cluster, ~2x the next-highest)
• Top Keywords: aw, gh aw, gh, v0, version, update, firewall, pins
• Characteristics: gh-aw-firewall version bumps, binary-path/integration-test consolidation, and CLI self-maintenance tasks — the widest generated-diff footprint (compiled .lock.yml churn) combined with the lowest review engagement of any cluster.
• Example PRs: Document Gemini Vertex WIF adapter dependency #62488, Bump gh-aw-firewall to v0.28.22 #62529, Bump gh-aw-firewall to v0.28.23 #62748, Consolidate duplicated gh-aw binary-path lookups in integration tests #62763, Use portable path for OTLP helper guidance #62940

---

Cluster 5: CI Job Failure Remediation (WIP auto-fix)

• Size: 44 tasks (7.0% of total)
• Success Rate: 81.8%
• Avg Comments / Reviews / Files Changed: 7.1 / 4.6 / 25.3
• Top Keywords: work, queue, actions, job, issue, fix, pr, id
• Characteristics: The "[WIP] Fix failing GitHub Actions job ..." auto-triggered remediation tasks — repeated failures on the same integration jobs (Workflow Misc Part 2 appears 3x among examples).
• Example PRs: [WIP] Fix failing GitHub Actions job Integration: Workflow Misc Part 2 #62715, [WIP] Fix failing GitHub Actions job Integration: CLI Logs Live Cache #62716, [WIP] Fix failing GitHub Actions job Integration: Workflow Misc Part 2 #63235, [WIP] Fix failing GitHub Actions job 'build' #64053, [WIP] Fix failing GitHub Actions job Integration: Workflow Misc Part 2 #64076

---

Cluster 6: Repo-Memory Ledger & Artifact Validation

• Size: 29 tasks (4.6% of total)
• Success Rate: 82.8%
• Avg Comments / Reviews / Files Changed: 8.9 / 5.0 / 28.0 (highest review/comment intensity of any cluster)
• Top Keywords: ledger, repo, validation, add, artifact, safe, output, daily
• Characteristics: Smallest, newest cluster — distributed-ledger and repo-memory backend features (standalone safe-output-backed ledgers, replay projections, daily AIC guardrail accounting).
• Example PRs: Add repo-memory backend for daily AIC guardrail #62958, [WIP] Add distributed ledger capability to repo-memory #64199, Move daily workflow review history to repo-memory ledgers #64337, Add standalone safe-output-backed ledger configuration #64354, Add replay projections to standalone ledgers #64420

Success Rate by Cluster

Cluster	Tasks	Success Rate	Avg Comments	Avg Files	Top Keywords
1 — Safe-Output, Agent Session & MCP	203	79.3%	6.6	47.3	agent, safe, output, mcp
2 — Checkout, Targeting & Coverage	176	83.5%	7.1	38.9	repository, checkout, coverage, regression
3 — Model Routing & Engine Config	114	78.1%	7.7	53.3	model, copilot, routing, claude
4 — gh-aw CLI, Firewall & Pinning	64	70.3%	5.5	114.3	gh aw, version, firewall, pins
5 — CI Job Failure Remediation	44	81.8%	7.1	25.3	work, actions, job, fix
6 — Repo-Memory Ledger & Artifacts	29	82.8%	8.9	28.0	ledger, repo, validation, artifact

Key Findings

1. Merge rate pulled back sharply this period (79.7%, -3.6 points from Oct 18): after two weeks in the 81–85% band, today's reading lands closer to the early-October low (75–78%), driven mainly by Cluster 4's weak conversion and a slightly lower rate across the two largest clusters (1 and 3) than in the prior report.
2. CLI/firewall/version-pinning work remains the weakest performer (70.3%) and has by far the largest diff footprint (114.3 avg files): these PRs touch roughly 2x the generated .lock.yml surface of any other cluster while getting the least review engagement (2.7 avg reviews) — the same dynamic flagged as a near-miss outlier in the Oct 18 report, now slightly worse (73.6% → 70.3%).
3. Checkout/targeting/coverage work is still the safest bet (83.5%): well-scoped, mechanical fixes with clear acceptance criteria continue to convert most reliably.
4. Repo-Memory Ledger tasks draw the heaviest review traffic (8.9 comments, 5.0 reviews avg) yet still merge well (82.8%) — a newer feature area attracting active scrutiny rather than rejection.
5. CI auto-remediation PRs recur on the same jobs: 3 of 5 sampled examples in Cluster 5 target "Integration: Workflow Misc Part 2," suggesting a flaky or under-resourced test lane rather than one-off failures.

Lowest-Merge Cluster Root Cause

• Cluster: Cluster 4 — gh-aw CLI, Firewall & Version Pinning
• Merge Rate Gap: 70.3% vs 79.7% overall (gap: 9.4 points — just under the 10-point/15-PR flag threshold, so not a formal outlier this period, but it is the lowest-rate cluster two periods running)
• Primary Blocker: Abandoned without review, not CI failure or explicit rejection
• Confidence: Medium
• Evidence PRs: Refresh available container image versions and pins #66122, Fix ARC/DinD agent temporary directory staging #63045, Exclude agent scratch directory from unified artifacts #63013, Manage self-hosted runner enforcement with gh aw env #64268, Investigate campaign-query memory estimation gap #64583, Harden Windows CLI integration workflow #65614, Harden installer downloads and security ownership paths #66121, Document Copilot GPT Responses firewall compatibility #68719, Sync gh-aw-actions pre-release status with gh-aw #69367, Per-server HTTP MCP CA support requires gateway changes #69436, Document Gemini Vertex WIF adapter dependency #62488

Of the 11 closed PRs in this cluster, 6 (55%) were closed with zero comments and zero reviews — no human or bot reviewer weighed in before closure (e.g. #64583, #65614, #66121, #68719, #69367, #69436). Of the remaining 5: #66122 ("Refresh available container image versions and pins") drew 15 comments and a bot review flagging "AWF routing integration is incomplete, Grafana telemetry remains enabled" — the same PR and the same blocker language cited in the Oct 18 report, still unresolved three days later. #63013 shows a stalled-session pattern (a human comment "@copilot Please move this PR forward... currently shows no changed files" went unresolved before closure). #63045 drew favorable PR comments but was still closed, suggesting superseding work rather than rejection on merits.

Blocker Category	Count	Share	Notes
Abandoned, zero engagement	6	55%	Closed with no comments/reviews — duplicate, superseded, or deprioritized before review
Review friction (changes requested)	1	9%	#66122 — incomplete routing/telemetry contract, open since at least Oct 18
Stalled agent session	1	9%	#63013 — nudged by a human comment, never produced changes, closed
Superseded despite favorable comments	1	9%	#63045 — positive PR comments but closed, likely replaced by a follow-up PR
Scope/documentation-only	2	18%	#62488, #68719 — documentation-dependency tasks closed without action

Recommendations

1. Re-triage the recurring Refresh available container image versions and pins #66122 pattern: the same "AWF routing integration incomplete / Grafana telemetry enabled" blocker has now persisted across at least two reporting periods in this cluster — worth a direct look rather than relying on repeated auto-generated PR attempts. Exclude agent scratch directory from unified artifacts #63013's stalled, no-diff session is a separate signal worth a timeout/retry check.
2. Add a lightweight stale-close signal for Cluster 4: since 55% of this cluster's closures have zero review engagement, a bot comment confirming relevance within 24h would separate genuinely superseded work from PRs worth reviewing before they go stale.
3. Keep routing scoped, mechanical work (Cluster 2) at current volume: it has the best size-to-success ratio in the dataset and could absorb more of the backlog currently going to the weaker CLI/firewall cluster.
4. Investigate the Cluster 5 "Workflow Misc Part 2" recurrence: repeated auto-remediation PRs against the same integration job point to a flaky test or an under-provisioned CI lane rather than independent one-off failures — fixing the root cause would reduce this cluster's volume directly.
5. Outlier-cluster recommendation: For Cluster 4 specifically, require at least one human/bot review pass before a CLI/firewall-pin PR can be closed, converting silent abandonment into either a merge or a documented rejection reason.

---

Generated by Prompt Clustering Analysis (Run: 39579815515)

Generated by 📊 Copilot Agent Prompt Clustering Analysis · claude · agent · 150 AIC · ⌖ 1 AIC · ⊞ 7K · ◷

• expires on Oct 22, 2026, 2:09 AM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-10-22T10:09:24.703Z.

Closed by Workflow