[prompt-clustering] Prompt Clustering Analysis - 2026-09-30

[github-actions[bot]]

Summary

Analysis Period: Last 30 days (2026-08-31 → 2026-09-30)
Total Tasks Analyzed: 548 copilot-authored PRs
Clusters Identified: 3
Overall Merge Rate: 77.4% (424 / 548)

One outlier cluster stands out: a 60-PR batch of "operational-value grader/study" PRs has a 0% merge rate, dragging down the overall average. Root-cause below.

Full Analysis Report

General Insights

• Most common task type: General workflow/engine feature work (Cluster 3, 433 PRs, 79% of all tasks) — spans compiler changes, safe-output plumbing, audit tooling, and engine/model routing.
• Highest success rate: Cluster 1, network/firewall/domain-audit tasks, 90.9% merged (50/55).
• Outlier cluster: Cluster 2, "operational-value" grader/study PRs, 0% merged (0/60) — a 77-point gap below the overall rate.
• Data note: Turn counts / workflow-run durations were not available this run (gh-aw logs is not accessible from this sandbox); this analysis relies on PR body, comments, reviews, and file-change data only.

Cluster Analysis

Cluster 1: Network / firewall / audit tooling

• Size: 55 tasks (10% of total)
• Merge rate: 90.9%
• Avg comments: 11.2 · Avg reviews: 6.0 · Avg files changed: 56.3
• Top keywords: domain, details, firewall, summary, network, blocked, domains
• Characteristics: Firewall/egress domain-allowlisting, audit findings, MCP payload auditing, and related infra hygiene tasks. High engagement (comments/reviews) and the best merge rate in the dataset — well-scoped, reviewable changes.
• Example PRs: #64096, #63687, #63241, #63177, #63034

---

Cluster 2: Operational-value grader/study batch (outlier)

• Size: 60 tasks (11% of total)
• Merge rate: 0.0%
• Avg comments: 0.3 · Avg reviews: 0.2 · Avg files changed: 2.5
• Top keywords: operational, operational value, value, evidence, study, grader, adoption, metric
• Characteristics: A concentrated batch (PR Add operational-value grading for Duplicate Code Detector #58452–Fix operational-value grader created-at acquisition and preserve grader result contract in gh aw logs #60635, mostly created 2026-09-04) tasked with building per-workflow "operational-value graders" or recording "blocked" study outcomes under a .github/skills/operational-value-designer/SKILL.md framework. Nearly all were closed within 1 hour of creation with zero comments and zero reviews — i.e., closed in bulk rather than reviewed and rejected individually.
• Example PRs: #58559, #58557, #58555, #58554, #58476

---

Cluster 3: General workflow/engine feature work

• Size: 433 tasks (79% of total)
• Merge rate: 86.4%
• Avg comments: 7.4 · Avg reviews: 3.4 · Avg files changed: 47.4
• Top keywords: workflow, model, github, copilot, agent, job, workflows, fix
• Characteristics: The bulk of day-to-day engineering — compiler/CLI changes, engine model routing, safe-output/ledger features, CI fixes, audit-artifact plumbing. Above-average merge rate, normal review engagement.
• Example PRs: #64420, #64403, #64357, #64354, #64339

Success Rate by Cluster

Cluster	Tasks	Merge Rate	Avg Comments	Avg Reviews	Top Keywords
1 — Network/audit	55	90.9%	11.2	6.0	domain, firewall, network, blocked
2 — Op-value (outlier)	60	0.0%	0.3	0.2	operational value, evidence, study, grader
3 — Workflow/engine	433	86.4%	7.4	3.4	workflow, model, copilot, agent

Full Data Table (representative sample)

Sample of 20 PRs spanning all three clusters, most-recent-first per cluster; the full 548-PR dataset backs the cluster stats above.

PR #	Title	Cluster	Outcome	Comments	Reviews
#64420	Add replay projections to standalone ledgers	3 — Workflow/engine	Open	13	11
#64403	Support AWF task-level model routing in the Copilot engine	3 — Workflow/engine	Closed	0	0
#64359	Update agentic engine CLI pins and recompile workflows	3 — Workflow/engine	Merged	0	1
#64357	Filter generated files before assembling reviewer PR diffs	3 — Workflow/engine	Merged	6	5
#64356	Honor add-labels.max above 10 for per-call label limit	3 — Workflow/engine	Closed	0	0
#64354	Add standalone safe-output-backed ledger configuration	3 — Workflow/engine	Merged	26	21
#64339	Collect experiment and evals data in the usage artifact	3 — Workflow/engine	Merged	7	2
#64338	Attribute friction costs to audit sources	3 — Workflow/engine	Merged	10	5
#64096	Add Daily Ecosystem Explorer agentic workflow	1 — Network/audit	Merged	0	3
#63687	Report MCP payload sizes in audit and aggregate analysis	1 — Network/audit	Merged	16	6
#63241	Support dynamic checkout sets from GitHub Actions expressions	1 — Network/audit	Closed	43	25
#63187	[ci-coach] Speed up slow rate-limit sleep in cgo A-C unit-test	1 — Network/audit	Merged	5	4
#63177	Filter logs to agentic workflows	1 — Network/audit	Merged	13	10
#63034	Remove Docker sbx and gVisor sandbox runtimes	1 — Network/audit	Merged	6	6
#60635	Fix operational-value grader created-at acquisition	2 — Op-value (outlier)	Closed	12	12
#58559	Record blocked operational-value study for daily-choice-test	2 — Op-value (outlier)	Closed	0	0
#58557	Add operational-value grader for daily-caveman-optimizer	2 — Op-value (outlier)	Closed	0	0
#58555	Add operational-value grader for cache strategy remediation	2 — Op-value (outlier)	Closed	0	0
#58554	Record blocked operational-value study for daily-byok-ollama-test	2 — Op-value (outlier)	Closed	0	0
#58553	Record blocked operational-value study for AWF spec surfacing	2 — Op-value (outlier)	Closed	0	0

Key Findings

1. One batch initiative accounts for the entire outlier: All 60 zero-merge PRs come from a single "operational-value-designer" research/grading batch launched around 2026-09-04, not from scattered quality problems across the codebase.
2. Bulk-close, not review rejection: 54 of 60 outlier PRs (90%) were closed within 1 hour of creation with 0 comments and 0 reviews — consistent with an automated or maintainer-driven bulk cleanup rather than individual code review finding defects.
3. Everything else performs above the historical baseline: excluding the outlier batch, the remaining 488 PRs merge at 86.9% (424/488), well above the blended 77.4% headline figure — the "normal" pipeline of feature/fix work is healthy.
4. Highest-touch cluster still ships: Cluster 1 (network/firewall/audit) has the heaviest review load (avg 6.0 reviews/PR) yet still merges at 90.9%, suggesting review friction there is productive, not blocking.

Lowest-Merge Cluster Root Cause

• Cluster: Cluster 2 — "operational-value" grader/study batch
• Merge Rate Gap: 0.0% vs 77.4% overall (gap: 77.4 points)
• Primary Blocker: Scope/complexity mismatch (batch-generated exploratory PRs closed without review)
• Confidence: High
• Evidence PRs: #58559, #58530, #58501, #58476, #58452

Blocker Category	Count	Share	Notes
Scope/complexity mismatch	54	90%	Closed within ~10–70 min, 0 comments/0 reviews — bulk-closed exploratory research PRs
Review friction	5	8%	1–2 comments, 0 reviews, closed after 2–3.5 hrs — light pushback before close
CI/test failure	1	2%	#60635 had 12 comments/12 reviews over active back-and-forth before closing
Other/unknown	0	0%	—

Sampled 60/60 PRs in the cluster (100% coverage). No merged PRs exist in this cluster to compare against, so the blocker classification is based on close-timing and engagement patterns alone.

Recommendations

1. Treat "operational-value grader" tasks as a research spike, not a mergeable-PR workflow: since 90% are closed unreviewed within an hour, route this work through an issue/design-doc process instead of opening 60 individual PRs that are immediately discarded — it inflates PR volume without any merge value.
2. Keep doing what works in Cluster 1 and Cluster 3: both sustain >85% merge rates with healthy review engagement; no process change indicated there.
3. If the operational-value initiative is still wanted, add a pre-flight check: gate PR creation on a lightweight maintainer sign-off (e.g., an issue comment or label) before the agent opens a PR, so the grader/study output lands as a comment or a single consolidated PR instead of dozens of individually-closed ones.
4. Outlier-cluster-specific action: audit whether the operational-value-designer skill is still an active initiative — if abandoned, retire the prompt/skill so future runs don't keep regenerating this pattern.

---

Generated by Prompt Clustering Analysis (Run: §36698575430)

Generated by 📊 Copilot Agent Prompt Clustering Analysis · claude · agent · 184.5 AIC · ⌖ 9.09 AIC · ⊞ 6.6K · ◷

• expires on Oct 1, 2026, 2:11 AM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-10-01T10:11:16.570Z.

Closed by Workflow