[prompt-clustering] Prompt Clustering Analysis - 2026-10-30

[github-actions[bot]]

Summary

Analysis Period: Last 30 days (as of 2026-10-30)
Total Tasks Analyzed: 595 copilot-agent PRs
Clusters Identified: 7
Overall Success Rate: 76.6% (merged / all analyzed PRs, including 52 still-open); 84.0% among the 543 PRs with a final decision (merged vs. closed) — consistent with the prior cycle's 84.0%

Full Analysis Report

General Insights

• Most Common Task Type: "workflow / agent / output / safe-output" tasks — 185 PRs (31% of total), covering agentic-workflow compiler, safe-outputs, and job/run fixes.
• Highest Success Rate: "Sous Chef" automated-PR cluster — 89.5% (38 PRs), also the highest-engagement cluster (avg. 12.5 comments/PR).
• Most Complex Tasks: "model / routing / copilot / claude" cluster — highest average commit count per PR (6.2), consistent with multi-step model-routing logic changes.
• Watch item: "version / pins / gh aw" dependency-bump cluster sits at 67.1% raw success — the lowest of the 7 — but does not meet the outlier bar (≥15 PRs and ≥10 points below baseline; gap here is 9.5 points). See root-cause note below.
• Data limitation: gh-aw logs/turn-count data was not available in this run's environment, so iteration complexity is approximated using comment/review/commit counts per PR instead of agent turn counts (see Edge Cases note at end of report).

Cluster Analysis

Cluster 1: Agentic Workflow & Safe-Output Fixes

• Size: 185 tasks (31.1% of total)
• Success Rate: 73.0% (135/185 merged)
• Avg Engagement: 5.5 comments, 4.1 reviews, 4.4 commits per PR
• Top Keywords: workflow, agent, output, safe, job, pr, fixes, run
• Characteristics: Compiler/runtime fixes to the agentic-workflow engine itself — safe-output handling, job wiring, run fixes, diagnostics. Largest cluster and the single biggest lever for improving aggregate success rate given its size.
• Example PRs: Document common workflow compatibility fixes after upgrading #72930, Record safe job and script outcomes in unified sessions #72146, Record privacy-preserving safe-output outcomes in unified sessions #72138, Record memory save outcomes in unified sessions and conclusion data #72133, Enforce ubuntu-slim timeout limits and upgrade default runners for longer jobs #72131

---

Cluster 2: Test Coverage & Regression Fixes

• Size: 142 tasks (23.9% of total)
• Success Rate: 81.0% (115/142 merged)
• Avg Engagement: 5.0 comments, 3.4 reviews, 3.9 commits per PR
• Top Keywords: coverage, regression, regression coverage, fixes, add, test, errors, repository
• Characteristics: Narrowly-scoped "add test coverage" / "fix regression" tasks. Second-largest cluster and the best success rate among large (100+) clusters — a good template for well-scoped, low-ambiguity task prompts.

---

Cluster 3: Model Routing & Engine Support

• Size: 93 tasks (15.6% of total)
• Success Rate: 77.4% (72/93 merged)
• Avg Engagement: 6.6 comments, 4.9 reviews, 6.2 commits per PR
• Top Keywords: model, copilot, routing, claude, models, awf, codex, agent
• Characteristics: Model/engine routing logic, new model support (e.g. GPT, Claude engine additions). Highest commit count of any cluster — these tasks require the most iteration, though success rate holds near baseline.

---

Cluster 4: Dependency / Version Pin Bumps

• Size: 70 tasks (11.8% of total)
• Success Rate: 67.1% (47/70 merged) — lowest raw rate, closest to outlier status
• Avg Engagement: 6.0 comments, 3.5 reviews, 4.9 commits per PR
• Top Keywords: version, aw, gh aw, gh, pins, update, workflow, default
• Characteristics: Routine dependency/version-pin bump PRs (gh-aw-firewall, MCPG, CLI, action pins). 11 of 70 (15.7%) are still open, vs. 8.7% open-rate overall — once pending PRs are excluded, the decided-only success rate is 79.7%, close to the 84.0% baseline. See root-cause note below.

---

Cluster 5: Work Queue & Dispatch

• Size: 39 tasks (6.6% of total)
• Success Rate: 79.5% (31/39 merged)
• Avg Engagement: 6.9 comments, 4.7 reviews, 4.6 commits per PR
• Top Keywords: queue, work, issue, coverage, native, existing, pr, workflow
• Characteristics: Work-queue/dispatch policy and claim-handling logic for coordinating agent tasks.

---

Cluster 6: Sous Chef Automated PRs

• Size: 38 tasks (6.4% of total)
• Success Rate: 89.5% (34/38 merged) — highest of all clusters
• Avg Engagement: 12.5 comments, 6.1 reviews, 6.4 commits per PR — highest engagement of any cluster
• Top Keywords: em, aic, sub, pr sous, pr sous chef, sous, chef
• Characteristics: PRs generated via the "PR Sous Chef" automation path. High comment/review volume paired with the best success rate suggests iterative back-and-forth converges reliably for this task type.

---

Cluster 7: Detection & Regression Jobs

• Size: 28 tasks (4.7% of total)
• Success Rate: 78.6% (22/28 merged)
• Avg Engagement: 5.7 comments, 4.2 reviews, 4.4 commits per PR
• Top Keywords: detection, agent, jobs, coverage, job, regression, mode, fixes
• Characteristics: Detection-pipeline and evals-job fixes, overlapping thematically with Cluster 2 but centered on job/detection infrastructure rather than general test coverage.

Success Rate by Cluster

Cluster	Tasks	Success Rate	Avg Comments	Avg Reviews	Avg Commits	Top Keywords
1. Workflow/Safe-Output	185	73.0%	5.5	4.1	4.4	workflow, agent, output
2. Coverage/Regression	142	81.0%	5.0	3.4	3.9	coverage, regression, fixes
3. Model Routing	93	77.4%	6.6	4.9	6.2	model, copilot, routing
4. Version/Pin Bumps	70	67.1%	6.0	3.5	4.9	version, aw, pins
5. Work Queue/Dispatch	39	79.5%	6.9	4.7	4.6	queue, work, issue
6. Sous Chef Automation	38	89.5%	12.5	6.1	6.4	em, aic, sub, pr sous chef
7. Detection/Regression Jobs	28	78.6%	5.7	4.2	4.4	detection, agent, jobs

Trend vs. Previous Cycle (2026-10-28)

Theme	Prev. Size → Now	Prev. Success → Now	Change
Workflow/agent/safe-output	211 → 185	82.0% → 73.0%	↓ 9.0 pts
Model routing/copilot	93 → 93	86.0% → 77.4%	↓ 8.6 pts
Coverage/regression/ledger	71 → 142	82.0% → 81.0%	≈ stable, cluster grew
Version/pin bumps	60 → 70	86.7% → 67.1%	↓ 19.6 pts (raw); ↓ ~7 pts decided-only
Sous Chef automation	47 → 38	89.4% → 89.5%	≈ stable
Work queue/dispatch	37 → 39	83.8% → 79.5%	↓ 4.3 pts

The two largest swings (workflow/safe-output and version/pins) both coincide with a higher share of currently-open (undecided) PRs in this snapshot than in the prior cycle, which mechanically depresses the raw "merged / all" rate. This is noted as a measurement caveat, not necessarily a quality regression — worth re-checking once those PRs resolve in the next cycle.

Key Findings

1. No formal outlier cluster this cycle: the lowest-performing cluster (version/pin bumps, 67.1%) falls 9.5 points below the 76.6% baseline — just under the 10-point threshold used to flag an outlier (and ≥15 PRs). A full root-cause investigation was not triggered, but see below for a lighter-weight note given the size of the swing.
2. Largest cluster (workflow/safe-output, 185 PRs) is also below baseline (73.0% vs. 76.6%/84.0%) — because of its size, even a modest success-rate improvement here would move the aggregate rate more than fixes to any smaller cluster.
3. Sous Chef-originated PRs have the best outcomes and the most engagement (89.5% success, 12.5 avg comments) — suggests that the iterative review/comment pattern used in that automation path is worth studying as a template for other task types.
4. Model-routing tasks require the most commits per PR (6.2 avg) — these are inherently more complex/iterative, yet still land near baseline, suggesting the extra iteration is proving productive rather than wasted.
5. Coverage/regression tasks (142 PRs, grew from 71 last cycle) remain a strong, low-friction category (81.0% success, lowest avg commit/review count among large clusters) — narrowly-scoped "add test / fix regression" prompts continue to work well.

Lowest-Merge Cluster Note (Version/Pin Bumps)

• Cluster: Version/Pin Bumps (versions, gh aw, pins, firewall/MCPG/CLI bumps)
• Raw Gap: 67.1% vs. 76.6% overall (gap: 9.5 points) — just below the formal outlier threshold, so this is a lighter-weight note rather than a full Phase 4.5 investigation.
• Primary Driver: Elevated open-PR share, not rejection. 11/70 PRs (15.7%) are still open vs. 8.7% open-rate overall; excluding opens, the decided success rate is 79.7% (47 merged / 59 decided), only ~4 points below the 84.0% overall decided rate.
• Confidence: Medium — spot-checked 3 closed PRs in this cluster (Harden installer downloads and security ownership paths #66121, Disable uv Actions caching and isolate sandbox runtime paths #66958, Freeze compiler-generated Copilot SDK npm installs #71885): one had CHANGES_REQUESTED review status with resolved threads before closing (review friction), one had no review activity at all (likely superseded/stale), and one was an active multi-commit PR with heavy bot-driven comment activity. No single dominant blocker pattern emerged from this small sample.
• Recommendation: Re-evaluate this cluster next cycle once its ~11 open PRs resolve; if the raw rate stays depressed after that, escalate to a full root-cause pass.

Recommendations

1. Prioritize the workflow/safe-output cluster for prompt quality review: it is the largest cluster (31% of all tasks) and sits below both baselines. Since this cluster covers compiler/runtime internals, consider adding more specific file/function pointers in task prompts to reduce exploration overhead.
2. Use coverage/regression-style prompts as a template: this cluster's narrow scope ("add test for X", "fix regression in Y") correlates with high success and low iteration — favor this framing when writing new task prompts where possible.
3. Study the Sous Chef engagement pattern: its high comment/review volume paired with the best success rate suggests that building in an iterative review loop (rather than one-shot task prompts) may generalize to other clusters, particularly the model-routing cluster which already requires heavy iteration.
4. Re-check the version/pin-bump cluster next cycle: don't treat the current 67.1% as a quality signal yet — confirm whether it's a timing artifact (pending opens) or a genuine new friction pattern once more of its PRs are decided.

---

Full Data Table

Sample of analyzed PRs by cluster (up to 12 per cluster; 595 total analyzed)

PR #	Title	Cluster	Outcome
#72970	Retry MCP gateway startup and identify pre-agent che...	workflow, agent, output	open
#72968	Surface incomplete-result reasons in failure issues ...	workflow, agent, output	open
#72941	Clarify feature-branch dispatch and document safe pr...	workflow, agent, output	open
#72940	Preserve agent config permissions and restore only a...	workflow, agent, output	open
#72939	Reject Copilot discovery-only responses as successfu...	workflow, agent, output	open
#72938	Keep safe-output credentials out of agent configuration	workflow, agent, output	open
#72936	Fix compilation errors for compiler-generated setup ...	workflow, agent, output	open
#72930	Document common workflow compatibility fixes after u...	workflow, agent, output	merged
#72929	Auto-migrate literal safe-output appends in post-steps	workflow, agent, output	open
#72146	Record safe job and script outcomes in unified sessions	workflow, agent, output	merged
#72138	Record privacy-preserving safe-output outcomes in un...	workflow, agent, output	merged
#72133	Record memory save outcomes in unified sessions and ...	workflow, agent, output	merged
...	173 more PRs in this cluster (see Success Rate by Cluster table above)	workflow, agent, output	—
#73078	Include canonical issue IDs in duplicate close intents	coverage, regression, regression coverage	open
#72969	Surface final engine errors in crash and outage reports	coverage, regression, regression coverage	open
#72935	Honor disabled AI-credit budgets in runtime AWF config	coverage, regression, regression coverage	closed
#72933	Guard Copilot sandbox startup against RUNNER_TEMP loss	coverage, regression, regression coverage	merged
#72932	Wire minted-token env bindings and OIDC permissions ...	coverage, regression, regression coverage	open
#72466	Allow on.bots Apps to check out same-repository PR e...	coverage, regression, regression coverage	closed
#72405	Suppress startup-failure reports for cancelled agent...	coverage, regression, regression coverage	merged
#72403	Allow on.bots Apps to check out same-repository PRs ...	coverage, regression, regression coverage	merged
#72101	Narrow Chrome egress to browser-specific Google serv...	coverage, regression, regression coverage	closed
#72100	Use Gonum helpers for grader descriptive statistics	coverage, regression, regression coverage	merged
#72097	Deduplicate Copilot agent cleanup logic	coverage, regression, regression coverage	closed
#72074	Trust same-repository bot PRs triggered by labels	coverage, regression, regression coverage	merged
...	130 more PRs in this cluster (see Success Rate by Cluster table above)	coverage, regression, regression coverage	—
#73052	Support engine.model-routing for Claude on native An...	model, copilot, routing	open
#73034	Route Safe Output Health Monitor with model routing ...	model, copilot, routing	open
#73033	Route Agent Job Health Monitor with model routing (g...	model, copilot, routing	open
#73032	Route Claude Code User Documentation Review with mod...	model, copilot, routing	merged
#73031	Route Copilot Session Insights through model routing...	model, copilot, routing	merged
#72934	docs: clarify workflow-specific Copilot model availa...	model, copilot, routing	merged
#72273	Support Claude on Vertex AI with GCP WIF	model, copilot, routing	open
#71620	Replace unavailable Copilot model in video analyzer	model, copilot, routing	merged
#71608	Prevent arXiv researcher Codex startup failure	model, copilot, routing	merged
#71043	Attribute Pi main/sub-agent requests with x-awf-call...	model, copilot, routing	merged
#70952	Model routing: accept none/max for pinned Codex and ...	model, copilot, routing	open
#70945	Codex: stage native TOML sub-agents with resolved mo...	model, copilot, routing	merged
...	81 more PRs in this cluster (see Success Rate by Cluster table above)	model, copilot, routing	—
#72937	Fix pi alias model paths on arc-dind	version, aw, gh aw	closed
#72272	Honor newer cached action pins in runtime setup	version, aw, gh aw	merged
#72102	Separate Agy bootstrap downloads from runtime firewa...	version, aw, gh aw	closed
#72099	Upgrade Copilot CLI to 1.0.103 with reviewed checksu...	version, aw, gh aw	merged
#71957	Bump MCPG to v0.5.0 and firewall to v0.29.0	version, aw, gh aw	merged
#71934	Preserve harness retry telemetry in spans and usage ...	version, aw, gh aw	open
#71899	Update Pi CLI default to 1.4.0	version, aw, gh aw	merged
#71885	Freeze compiler-generated Copilot SDK npm installs	version, aw, gh aw	closed
#71879	Refresh Node image pins and enforce patched runtime ...	version, aw, gh aw	merged
#71616	Update CLI and scanner image versions	version, aw, gh aw	merged
#71451	docs: clarify directory restrictions for reusable ag...	version, aw, gh aw	merged
#71339	Fall back to pinned Copilot CLI when live compatibil...	version, aw, gh aw	merged
...	58 more PRs in this cluster (see Success Rate by Cluster table above)	version, aw, gh aw	—
#72632	Link work queue footers to creation and completion r...	queue, work, issue	merged
#72096	Enable readerstringconv in native and wasm CI	queue, work, issue	closed
#71898	Fix generated conclusion references and fetch warnin...	queue, work, issue	merged
#71665	Fix missing work-queue diagnostic guidance	queue, work, issue	merged
#71613	[Safe Outputs] Standardize machine-readable error codes	queue, work, issue	merged
#71607	Require item-level verification in daily queue reports	queue, work, issue	merged
#71495	Report missing work queue pools with actionable reco...	queue, work, issue	open
#71467	Verify queued work item status	queue, work, issue	closed
#71340	Add dual-role work queue smoke workflow	queue, work, issue	closed
#69987	Document claim delivery diagnostics	queue, work, issue	merged
#68715	Align MCP server CLI reference with help output	queue, work, issue	merged
#68582	Updating work queue protocol and related components	queue, work, issue	merged
...	27 more PRs in this cluster (see Success Rate by Cluster table above)	queue, work, issue	—
#67543	Deduplicate memory allowed-extension parsing	em, aic, sub	merged
#67035	Bump gh-aw-firewall to v0.28.50	em, aic, sub	merged
#65344	Normalize unified agent session metadata and AIC acc...	em, aic, sub	merged
#65115	Update lethal trifecta landing page copy	em, aic, sub	merged
#65069	Prevent base config restore from clobbering an exter...	em, aic, sub	merged
#65068	Fail add-comment when the triggering target cannot b...	em, aic, sub	merged
#65067	Handle transient stdin reads in the safe-output tran...	em, aic, sub	merged
#65066	Clarify PR branch target errors in scheduled runs	em, aic, sub	merged
#65062	Add Git-backed work queue operator commands	em, aic, sub	merged
#65030	Add deterministic dispatch coordinator replay core	em, aic, sub	merged
#64976	Require explicit scopes for GitHub App tokens at com...	em, aic, sub	merged
#64972	Standardize safe-output errors and staged asset prev...	em, aic, sub	merged
...	26 more PRs in this cluster (see Success Rate by Cluster table above)	em, aic, sub	—
#73046	Use topology-aware LLM gateway URL in detection and ...	detection, agent, jobs	merged
#73010	Route Avenger with model routing instead of pinning ...	detection, agent, jobs	merged
#72404	Exclude Copilot utility models from alias selection ...	detection, agent, jobs	closed
#71933	Prevent skipped threat detection from reporting inhe...	detection, agent, jobs	merged
#71897	Detect missing recovery in goroutines launched throu...	detection, agent, jobs	merged
#71896	Detect parenthesized Flush discards in uncheckedflus...	detection, agent, jobs	merged
#71880	Close custom-linter detection gaps and exempt safe o...	detection, agent, jobs	merged
#71624	Inherit agent API targets in matching-engine threat ...	detection, agent, jobs	merged
#71623	Allow validated status-only jobs after detection	detection, agent, jobs	merged
#71501	Pin threat detection to v0.5.3	detection, agent, jobs	merged
#70861	Resolve Claude detection model aliases before CLI la...	detection, agent, jobs	merged
#70325	Threat detection: apply the warn-mode safe-outputs p...	detection, agent, jobs	merged
...	16 more PRs in this cluster (see Success Rate by Cluster table above)	detection, agent, jobs	—

Edge Cases & Limitations

• Missing workflow-run metrics: gh-aw logs (turn counts, duration, cost) was not reachable from this sandbox environment, so per-task iteration complexity is approximated via comment/review/commit counts on each PR rather than agent turn counts.
• Open-PR caveat: 52 of 595 analyzed PRs (8.7%) are still open at analysis time; raw "success rate" treats these as not-yet-merged, which can understate true eventual success for clusters with many recent/pending PRs (see Version/Pin Bumps cluster above).
• No formal outlier this cycle: all 7 clusters had ≥15 PRs, but none crossed the ≥10-point-below-baseline threshold, so Phase 4.5's full root-cause workflow was not triggered; a lighter-weight note was produced instead for the lowest-performing cluster.

References:

• §41035439270

Generated by 📊 Copilot Agent Prompt Clustering Analysis · claude · agent · ~140 AIC · ⌖ ~1 AIC · ⊞ 5K · ◷

• expires on Oct 31, 2026, 2:06 AM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-10-31T10:06:28.607Z.

Closed by Workflow