[prompt-clustering] Prompt Clustering Analysis - 2026-09-29

[github-actions[bot]]

Summary

Analysis Period: Last 30 days (2026-08-30 → 2026-09-29)
Total Tasks Analyzed: 558 copilot-authored PRs
Clusters Identified: 3 (elbow method capped k at 3; a 4th–7th split was tried but added no separable structure beyond the dominant outlier)
Overall Success Rate: 76.3% (426 merged / 558 total)

Full Analysis Report

General Insights

• Most common task type: Cluster 1 — routine workflow/CI fixes, graders, and MCP-tooling patches (347 PRs, 62% of total).
• Highest success rate: Cluster 2 — larger feature work (Playwright, safe-outputs, firewall, Linear/Azure DevOps integrations) at 89.3% merged, despite being the heaviest-review cluster.
• Most complex tasks: Cluster 2 also carries the highest engagement load — 13.1 avg comments and 5.8 avg reviews per PR vs. 5.3/2.7 in Cluster 1.
• Outliers: Cluster 3 (61 PRs, 11% of total) is a severe negative outlier — only 1 of 61 merged (1.6%).

Cluster Analysis

Cluster 1: Routine fixes, graders & CI maintenance

• Size: 347 tasks (62.2% of total)
• Success Rate: 83.9% (291 merged, 53 closed unmerged, 3 open)
• Avg engagement: 5.3 comments, 2.7 reviews, 41.0 files changed
• Top keywords: workflow, github, job, model, fix, coverage, workflows, fixes
• Characteristics: Small-to-medium bug fixes, CI job repairs, dependency fixes, and "trajectory grader" additions for individual agent workflows. High volume, moderate review overhead.
• Example PRs: Emit grader results in OpenTelemetry spans #57015 (OpenTelemetry grader spans, merged), Fix missing Undici dependency in Copilot SDK workflows #57057 (Undici dependency fix, merged), Fix remote MCP authentication canary tool check #57081 (remote MCP auth canary fix, merged), Fix github-remote-mcp-auth-test: replace removed get_repository MCP tool with get_file_contents #57080 (related MCP tool-reference fix, closed unmerged)

---

Cluster 2: Substantial feature & infrastructure work

• Size: 150 tasks (26.9% of total)
• Success Rate: 89.3% (134 merged, 6 closed unmerged, 10 still open)
• Avg engagement: 13.1 comments, 5.8 reviews, 56.0 files changed
• Top keywords: firewall, network, Playwright, safe outputs, MCP, integration
• Characteristics: The highest-merge, highest-scrutiny cluster — new integrations (Linear, Azure DevOps), Playwright CLI overhauls, gh-aw-firewall upgrades. These get the most human review attention and still merge at the best rate, suggesting review investment here pays off rather than blocking.
• Example PRs: Integrate gh-aw-firewall v0.28.12 attested Cloud Hypervisor artifacts #57514 (firewall v0.28.12 integration), Add Azure DevOps work-item safe outputs #57796 (Azure DevOps safe outputs), Add native Linear Safe Outputs #57816/Add first-class Linear MCP tools #57874 (native Linear tools), Install Playwright OS dependencies before launching browsers #57951 (Playwright OS deps)

---

Cluster 3: Bulk auto-generated "operational-value grading" PRs — OUTLIER

• Size: 61 tasks (10.9% of total)
• Success Rate: 1.6% (1 merged, 60 closed unmerged, 0 open)
• Avg engagement: 0.4 comments, 0.2 reviews, 2.5 files changed
• Top keywords: operational, operational value, evidence, study, metric, grading
• Characteristics: A near-duplicate PR template ("Add operational-value grading for <workflow name>" / "Record blocked operational-value study for <workflow name>") repeated ~58 times, one per workflow in the repo. See root-cause section below.
• Example PRs: Add operational-value grading for Duplicate Code Detector #58452, Add operational-value grader for Issue Triage Agent #58453, Add operational-value grader for Daily Documentation Healer #58454, Add operational-value grading for A/B testing advisor #58464–58476 (and dozens more in the 58500s range)

Success Rate by Cluster

Cluster	Tasks	Merged	Closed unmerged	Open	Success Rate	Avg Comments	Avg Reviews	Top Keywords
1 — Routine fixes/graders	347	291	53	3	83.9%	5.3	2.7	workflow, github, job, fix, coverage
2 — Feature/infra work	150	134	6	10	89.3%	13.1	5.8	firewall, network, Playwright, MCP
3 — Bulk grading PRs (outlier)	61	1	60	0	1.6%	0.4	0.2	operational value, evidence, study

Full Data Table (sample)

Representative PRs per cluster (24 shown)

PR #	Title	Cluster	Outcome
57015	Emit grader results in OpenTelemetry spans	1	Merged
57057	Fix missing Undici dependency in Copilot SDK workflows	1	Merged
57080	Fix github-remote-mcp-auth-test tool reference	1	Merged
57088	Fix require-getexecoutput-exitcode-check false positive	1	Merged
57081	Fix remote MCP authentication canary tool check	1	Merged
57080	Fix github-remote-mcp-auth-test tool reference	1	Closed (unmerged)
57058	Remove Copilot SDK dependency from Documentation Noob Tester	1	Closed (unmerged)
57514	Integrate gh-aw-firewall v0.28.12 attested artifacts	2	Merged
57796	Add Azure DevOps work-item safe outputs	2	Merged
57816	Add native Linear Safe Outputs	2	Merged
57874	Add first-class Linear MCP tools	2	Merged
57951	Install Playwright OS dependencies before launching browsers	2	Merged
57521	Remove built-in Playwright MCP support	2	Merged
58452	Add operational-value grading for Duplicate Code Detector	3	Closed
58453	Add operational-value grader for Issue Triage Agent	3	Closed
58454	Add operational-value grader for Daily Documentation Healer	3	Closed
58465	Add ACE Editor operational-value grader	3	Closed
58466	Record blocked operational-value study for agent-job-health	3	Closed
58469	Add blocked operational-value study for agentic-token-trend-audit	3	Closed
58471	Record blocked operational-value study for API consumption report	3	Closed
58473	Add operational-value grading for Approach Validator	3	Closed
58474	Add deterministic operational-value grading for Archie	3	Closed
58476	Add operational-value grader for architecture-guardian workflow	3	Closed
58501	Record operational value study for archivx-agentic-workflows	3	Closed
58504	Add deterministic operational-value grading for auto-triage	3	Closed
58505	Add Avenger operational-value grader	3	Closed

Key Findings

1. The bulk-generated "operational-value grading" batch is the single biggest drag on the 30-day merge rate. Excluding Cluster 3 entirely, the merge rate across the other 497 PRs is 85.5%; including it drags the headline rate down to 76.3% — a 9.2-point difference from one batch event.
2. Heavier review does not mean lower success. Cluster 2 gets more than double the comments/reviews of Cluster 1 yet merges at a higher rate (89.3% vs 83.9%), suggesting review friction isn't the limiting factor for substantial feature work in this repo — task scoping is.
3. The merge rate has been trending down over the last two runs: 80.8% (Sep 25, n=733) → 78.4% (Sep 27, n=570) → 76.3% (Sep 29, n=558), as the Sep 4 bulk-closure batch has stayed inside the trailing 30-day window and newer routine-fix PRs haven't fully offset it yet. Expect the rate to recover once Sep 4 rolls out of the window (~Oct 4).

Lowest-Merge Cluster Root Cause

• Cluster: 3 — bulk "operational-value grading" PRs
• Merge Rate Gap: 1.6% vs 76.3% overall (gap: 74.7 points)
• Primary Blocker: Scope/complexity mismatch — bulk automated task generation, not individual PR quality
• Confidence: High
• Evidence PRs: Add operational-value grading for Duplicate Code Detector #58452, Add operational-value grader for Issue Triage Agent #58453, Record blocked operational-value study for agent-job-health #58466, Add blocked operational-value study for agentic-token-trend-audit #58469, Add operational-value grader for architecture-guardian workflow #58476

What happened: On 2026-09-04, between 10:42 and 15:32 UTC, ~58 near-identical PRs were opened — one per existing agentic workflow in the repo — each adding an "operational-value grader" or recording a "blocked operational-value study" for that workflow. Every one of these PRs:

• Was closed within 15–45 minutes of creation (far faster than any human review cycle observed elsewhere in the dataset).
• Received zero substantive human review (0 reviews on all 61; only 2 of 61 got any comment at all).
• The 2 commented PRs (Add operational-value grading for Duplicate Code Detector #58452, Add operational-value grader for Issue Triage Agent #58453) received an identical automated PR Triage Agent comment tagging them Action: batch_review | Batch: operational-value-grading — confirming these were triaged and closed as a batch by an automated agent, not rejected individually on their merits.

Blocker Category	Count	Share	Notes
Automated batch closure (PR Triage Agent, batch_review)	60	98%	Closed 15–45 min after opening, before any human could plausibly review
CI/test failure	0	0%	No evidence of CI runs blocking these
Review friction	0	0%	No human review comments found
Other/merged	1	2%	#57005 (a documentation PR, likely mis-clustered due to shared "study/design" vocabulary rather than part of the same batch)

This is not a quality problem with individual PRs — it's a single experimental/meta task (generating one probe PR per workflow to test "operational-value grading" instrumentation) that was, by design or by an automated triage policy, closed in bulk. It should be excluded from any headline "PR success rate" metric, since it represents one coordinated event rather than 58 independent task failures.

Recommendations

1. Exclude or separately track bulk/meta-experiment batches (like the Sep 4 operational-value-grading run) from the headline copilot-PR success metric — a single batch event should not be conflated with day-to-day coding task quality.
2. Cluster 2's pattern is worth reinforcing: tasks that land in richer integration/infrastructure work merge at the highest rate even with the most review — continue routing ambitious, well-scoped feature requests to the agent rather than assuming complexity hurts success.
3. For Cluster 1 (the largest bucket, 62% of volume), the 53 closed-unmerged PRs are worth a follow-up pass — investigate whether a subset are duplicate/superseded "trajectory grader" or CI-fix attempts that could be consolidated before creation.
4. Outlier-cluster-specific: If further batch/meta-experiments (one-PR-per-workflow style) are planned, consider running them as a single tracking issue or a dry-run report rather than opening dozens of live PRs that get auto-closed — this reduces PR-list noise and avoids skewing merge-rate metrics for a full 30-day window.

Data Limitations

• Workflow-run turn/duration/cost metrics (Phase 2) were unavailable this run — the gh-aw CLI binary was not present in this environment and gh is not authenticated, so aw_info.json turn counts could not be collected. This report is based on PR metadata (title, body, comments, reviews, file counts) only.
• All 558 PRs in the current 30-day window had cached full data available (comments/reviews/commits); no PRs were skipped for missing data.

---

Generated by Prompt Clustering Analysis (Run: 36551654081)

Generated by 📊 Copilot Agent Prompt Clustering Analysis · claude · agent · 167 AIC · ⌖ 10.3 AIC · ⊞ 6.6K · ◷

• expires on Sep 30, 2026, 2:08 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #64459.