[prompt-clustering] Prompt Clustering Analysis - 2026-09-06

[github-actions[bot]]

Summary

Analysis Period: Last 30 days (PRs created 2026-08-16 to 2026-09-06)
Total Tasks Analyzed: 1,000
Clusters Identified: 5
Overall Success Rate: 80.5% (805 merged / 186 closed / 9 open)

Full Analysis Report

General Insights

• Most Common Task Type: Cluster 4 — daily copilot workflow features, model routing, and MCP/agent configuration (377 PRs, 37.7% of total)
• Highest Success Rate: Cluster 2 — PR Sous Chef / config & schema features (94.0% merged)
• Most Complex Tasks: Cluster 4 also has the highest average diff size (716 additions / 44.6 files) and lowest review-comment density relative to size, consistent with broad multi-workflow changes
• Outliers: 1 outlier cluster identified — Cluster 0 (operational-value grader batch sweep) at 9.2% merge rate vs. 80.5% baseline, a 71-point gap

Data Quality Note

850 of 1,000 PRs had enriched full data (comments, reviews, review decision, diff stats) available in cache; the remaining 150 — nearly all belonging to the Cluster 0 batch sweep described below — were closed same-day and had no cached review/comment history to draw from. Workflow run logs (turn counts, cost, duration) were not available for this run, so turn/duration metrics are omitted; recommend enabling gh-aw logs collection for a future pass.

Cluster Analysis

Cluster 1: Operational-value grader batch sweep (outlier)

• Size: 65 tasks (6.5% of total)
• Success Rate: 9.2%
• Top Keywords: operational-value, evidence, grader, blocked, graders
• Characteristics: A single automated campaign generated one "add/record operational-value grader" PR per repository workflow (~65 workflows), all created within a 5-hour window on 2026-09-04, tagged pr-batch:operational-value-grading / pr-action:batch_review. 59 of 65 were closed the same day with no recorded review comments — consistent with a bulk policy-driven rejection rather than per-PR technical review.
• Example PRs: Record blocked operational-value study for daily-choice-test #58559, Add operational-value grader for daily-caveman-optimizer #58557, Add operational-value grader for cache strategy remediation #58555, Record blocked operational-value study for daily-byok-ollama-test #58554, Record blocked operational-value study for AWF spec surfacing #58553

---

Cluster 2: Safe-outputs job & PR-mutation fixes

• Size: 162 tasks (16.2% of total)
• Success Rate: 88.3%
• Average Diff: 378 additions / 26.8 files, 6.6 comments / 3.3 reviews per PR
• Top Keywords: safe outputs, job, output, actions
• Characteristics: Bug fixes and hardening to the safe-output collect/dispatch pipeline (handler registration, label/PR mutation correctness, conformance checks).
• Example PRs: Register replace_label handler in safe-output collect job dispatch map #58917, Fix Update Astro model and safe-output fallback #58909, Stage Linear and Jira smoke issue outputs #58786, Fix flaky SIGPIPE/pipefail false positives in safe-outputs conformance checker #58768, Reclassify policy-driven safe-output declines as skipped, not hard failures #58767

---

Cluster 3: PR Sous Chef / config & workflow schema

• Size: 67 tasks (6.7% of total)
• Success Rate: 94.0% (highest)
• Average Diff: 612 additions / 31.4 files, 13.0 comments / 5.5 reviews per PR — most heavily reviewed cluster
• Top Keywords: sous chef, config, schema extensions
• Characteristics: Larger, well-reviewed feature PRs extending aw.yml config, env-expression validation, and footer/templating conventions.
• Example PRs: Add deterministic body footer templates to safe outputs #58841, Extend aw.yml config with repository labels #58796, Validate top-level env expression contexts #58781, Restore the sandbox.agent: false opt-out #58693, Add payload-based grader execution command #58634

---

Cluster 4: Test coverage, error handling & schema hardening

• Size: 329 tasks (32.9% of total)
• Success Rate: 90.6%
• Average Diff: 261 additions / 11.8 files, 6.5 comments / 3.3 reviews per PR
• Top Keywords: coverage, test, pkg, error, schema
• Characteristics: The largest well-defined engineering cluster — focused, scoped bug fixes and test-coverage additions across Go packages. Small diffs and solid review engagement correlate with the second-highest success rate.
• Example PRs: (see cache for full listing)

---

Cluster 5: Daily copilot workflow features & model/MCP routing

• Size: 377 tasks (37.7% of total, largest cluster)
• Success Rate: 78.2%
• Average Diff: 716 additions / 44.6 files, 4.8 comments / 2.1 reviews per PR — largest diffs, lowest review density
• Top Keywords: copilot, daily, workflows, agent, mcp
• Characteristics: Broad changes spanning many daily-workflow definitions, Copilot/Codex model configuration, and MCP proxy routing. Largest average diff size combined with below-average review depth likely explains the somewhat lower (though not outlier-level) success rate.

Success Rate by Cluster

Cluster	Tasks	Success Rate	Avg Additions	Avg Files	Avg Comments	Avg Reviews	Top Keywords
1 — Operational-value batch sweep	65	9.2%	206	4.5	3.5	1.7	operational-value, grader, blocked
2 — Safe-outputs fixes	162	88.3%	378	26.8	6.6	3.3	safe outputs, job, output
3 — Sous Chef / config schema	67	94.0%	612	31.4	13.0	5.5	sous chef, config, schema
4 — Test coverage / error handling	329	90.6%	261	11.8	6.5	3.3	coverage, test, error, schema
5 — Daily workflows / model routing	377	78.2%	716	44.6	4.8	2.1	copilot, daily, workflows, mcp

Key Findings

1. A single batch campaign accounts for nearly all closed PRs in the dataset. The 65-PR operational-value grader sweep (Cluster 1) contributes 59 of the 186 total closed PRs (31.7%) despite being only 6.5% of all tasks — it is the dominant driver of the dataset's non-merge rate.
2. Review depth correlates with success, up to a point. Clusters 2 and 3 (highest success rates, 88.3%/94.0%) also have the highest average review/comment counts, while Cluster 1 (lowest success) has essentially zero recorded review engagement — it was rejected before individual review occurred.
3. Bigger diffs, thinner review — Cluster 5's pattern. The largest cluster by volume (377 PRs) also has the largest average diff (716 additions, 44.6 files) but the lowest review density (2.1 reviews/PR), and sits 12.3 points below the "well-reviewed" clusters despite not qualifying as a statistical outlier. Worth monitoring if this gap widens.

Lowest-Merge Cluster Root Cause

• Cluster: Cluster 1 — operational-value grader batch sweep
• Merge Rate Gap: 9.2% vs. 80.5% overall (gap: 71.3 points)
• Primary Blocker: Scope/complexity mismatch — a fleet-wide, one-PR-per-workflow campaign that was batch-reviewed and rejected as a class, not on individual technical merit
• Confidence: Medium — the classification rests on strong circumstantial/metadata evidence (uniform creation window, shared pr-batch:operational-value-grading/pr-action:batch_review labels, same-day closure) rather than quoted review comments, because none of the 59 closed PRs in this cluster had cached comment/review history available
• Evidence PRs: Record blocked operational-value study for daily-choice-test #58559, Add operational-value grader for daily-caveman-optimizer #58557, Add operational-value grader for cache strategy remediation #58555, Record blocked operational-value study for daily-byok-ollama-test #58554, Record blocked operational-value study for AWF spec surfacing #58553 (all created 2026-09-04T10:42–15:32Z, closed same day, zero recorded comments/reviews)

Blocker Category	Count	Share	Notes
Scope/complexity mismatch	~59	91%	Uniform batch labels + same-day mass closure with no individual review trail; consistent with a policy decision that a grader-per-workflow campaign was too broad/low-marginal-value to merge wholesale
CI/test failure	0	0%	No CI failure signal found in available metadata
Review friction	0	0%	No review comments were recorded for the closed PRs in this cluster
Other/unknown	~6	9%	6 PRs in the cluster (pre-dating the batch, e.g. #55445, #55794) merged normally as foundational infrastructure for the grader system — these are not part of the blocked pattern

Recommendations

Based on clustering analysis:

1. Avoid one-PR-per-target campaigns for exploratory/uncertain-value work. Cluster 1 shows that fanning out a single initiative into dozens of near-identical PRs (one per workflow) produces a near-total rejection rate when the underlying approach isn't yet validated. Prototype and validate the grader design against 2-3 workflows first, then batch the rollout only once the pattern is proven.
2. Preserve review/comment history even for batch-rejected PRs. 59 of 65 PRs in the outlier cluster had no comments/reviews cached, which limited root-cause confidence to "medium." If batch-review tooling closes PRs programmatically, consider leaving a closing comment stating the rejection rationale so future analyses (and other contributors) can distinguish "batch policy rejection" from "technical failure" without guessing from timestamps and labels.
3. Cluster 5 (daily workflows / model routing) is the highest-volume cluster with the largest diffs and thinnest review density. Consider capping scope-per-PR for these tasks (e.g. one workflow/model change per PR rather than fleet-wide sweeps) to keep review depth proportional to diff size, similar to the pattern seen in higher-success Clusters 2/3.
4. Outlier Cluster Recommendation: For any future "operational-value grader" or similar fleet-wide campaigns, gate the rollout behind an explicit go/no-go review of the design on a small sample before generating PRs for the full workflow inventory — this would likely have caught the scope mismatch before 65 PRs were opened and 59 closed in a single day.

---

Generated by Prompt Clustering Analysis (Run: 34025435215)

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• api.anthropic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.anthropic.com"

See Network Configuration for more information.

Generated by 📊 Copilot Agent Prompt Clustering Analysis · claude · agent · 107.3 AIC · ⌖ 10.9 AIC · ⊞ 16.2K · ◷

• expires on Sep 7, 2026, 2:09 AM UTC-08:00