[prompt-clustering] Prompt Clustering Analysis - 2026-10-29

[github-actions[bot]]

Summary

Analysis Period: Last 30 days (2026-09-29 to 2026-10-28)
Total Copilot PRs Analyzed: 597 (500 merged, 96 closed, 1 open)
Clusters Identified: 9
Overall Merge Rate: 83.8%

An outlier cluster was found: Cluster 7 (ledger / safe-output job plumbing), merging at 73.5% vs. the 83.8% baseline — a 10.3-point gap on 49 PRs. Root-cause findings are below.

Full Analysis Report

General Insights

• Most common task type: Cluster 5 — test coverage, validation and regression fixes (104 PRs, 17% of total), the largest and most generic engineering-hardening bucket.
• Highest success rate: Cluster 6 — "Sous Chef" assistant / small bot-driven fixes (89.1%) and Cluster 0 — linter/detection work (88.9%), tied.
• Most complex / highest-friction tasks: Cluster 7 — ledger and safe-output persistence plumbing, both the lowest merge rate and (per the root-cause review below) the heaviest concentration of design-gate and correctness pushback.
• Outliers: 1 outlier cluster identified (Cluster 7), meeting both thresholds (≥15 PRs, ≥10-point merge-rate gap vs. overall).
• Note on workflow metrics: Workflow run logs (turn counts, duration, cost) were not pulled for this run — PR-level data alone was sufficient to find and investigate the outlier cluster, and re-fetching 30 days of run logs for 597 PRs was out of scope for this pass. Turn-count columns below are omitted rather than estimated.

Cluster Analysis

Cluster 5: Test coverage, validation & regression fixes

• Size: 104 tasks (17.4% of total)
• Success Rate: 85.6%
• Top Keywords: coverage, add, validation, fixes, test, regression, checks
• Characteristics: General hardening work — adding tests, closing coverage gaps, fixing regressions, egress/permission guards. The catch-all "engineering hygiene" bucket.
• Example PRs: Use Gonum helpers for grader descriptive statistics #72100, Fix compile-progress file permission policy violation #71947, Harden fetch-script portability and structured lint reporting #71932, Unify GitHub Contents base64 decoding #71894, Guard indexing boundaries and prevent cron-step overflow #71889

---

Cluster 8: Agent job failure handling & session telemetry

• Size: 101 tasks (16.9%)
• Success Rate: 81.2%
• Top Keywords: failure, agent, job, artifact, step, prompt, run
• Characteristics: Recording/classifying agent job outcomes (startup failures, retries, secret-safe diagnostics) in unified sessions and artifacts.
• Example PRs: Suppress startup-failure reports for cancelled agent jobs #72405, Record privacy-preserving safe-output outcomes in unified sessions #72138, Record memory save outcomes in unified sessions and conclusion data #72133, Preserve harness retry telemetry in spans and usage artifacts #71934, Classify engine driver crashes with secret-safe stderr diagnostics #71883

---

Cluster 2: Repository/workflow trust & CI setup

• Size: 85 tasks (14.2%)
• Success Rate: 82.4%
• Top Keywords: repository, workflow, review, github, fixes, setup
• Characteristics: GitHub App / bot-PR trust rules, firewall egress scoping, CI security triage.
• Example PRs: Allow on.bots Apps to check out same-repository PRs on edited events #72403, Trust same-repository bot PRs triggered by labels #72074, Centralize missing permission scope expansion #71888, Triage setup-script security findings as false positives #71895, Separate Agy bootstrap downloads from runtime firewall egress #72102

---

Cluster 3: Model routing & engine configuration

• Size: 83 tasks (13.9%)
• Success Rate: 86.7%
• Top Keywords: model, copilot, routing, claude, models, smoke, routed
• Characteristics: Copilot/Claude/Codex model selection, routing fallbacks, startup-failure prevention for specific engines.
• Example PRs: Replace unavailable Copilot model in video analyzer #71620, Prevent arXiv researcher Codex startup failure #71608, Attribute Pi main/sub-agent requests with x-awf-caller and preserve caller metadata #71043, Model routing: accept none/max for pinned Codex and add routed Codex smoke #70952, Support Claude on Vertex AI with GCP WIF #72273

---

Cluster 4: CLI/action version pins & upgrades

• Size: 63 tasks (10.6%)
• Success Rate: 87.3%
• Top Keywords: version, aw, gh aw, pins, update, cli
• Characteristics: Dependency and version-pin bumps (Copilot CLI, MCPG, firewall, Node images) with checksum/pin hygiene.
• Example PRs: Honor newer cached action pins in runtime setup #72272, Upgrade Copilot CLI to 1.0.103 with reviewed checksum pin #72099, Bump MCPG to v0.5.0 and firewall to v0.29.0 #71957, Update Pi CLI default to 1.4.0 #71899, Refresh Node image pins and enforce patched runtime dependencies #71879

---

Cluster 7: Ledger & safe-output job plumbing — OUTLIER

• Size: 49 tasks (8.2%)
• Success Rate: 73.5% (10.3 points below the 83.8% overall rate)
• Top Keywords: ledger, safe, output, safe output, job, workflow, outputs, missing
• Characteristics: Internal persistence/plumbing for ledgers (map/set/table/counter ledgers, compaction, work-queue claims) and safe-output telemetry. This is an area under heavy, fast-moving architectural iteration (consolidation, migration to "Agentic Maintenance" jobs, repeated redesigns of ledger compaction).
• Example PRs: Record safe job and script outcomes in unified sessions #72146, Enforce ubuntu-slim timeout limits and upgrade default runners for longer jobs #72131, Read cached state before workflow analysis #71878 (merged); Propagate configured environments to persistence jobs #72098, Preserve safe-output telemetry and tighten terminal-output guidance #71884, Fix Windows agent runtime setup, dry-run configuration, and audit identities #71882 (not merged)

---

Cluster 6: "Sous Chef" assistant & small bot fixes

• Size: 46 tasks (7.7%)
• Success Rate: 89.1%
• Top Keywords: em, aic, sous chef, chef, run
• Characteristics: Small, scoped fixes routed through the PR assistant bot — memory parsing, config restore, transport edge cases.
• Example PRs: Deduplicate memory allowed-extension parsing #67543, Update lethal trifecta landing page copy #65115, Prevent base config restore from clobbering an external root checkout #65069, Fail add-comment when the triggering target cannot be resolved #65068, Handle transient stdin reads in the safe-output transport #65067

---

Cluster 1: Work-queue & safe-output policy/diagnostics

• Size: 39 tasks (6.5%)
• Success Rate: 79.5%
• Top Keywords: queue, work, issue, policy, coverage, native
• Characteristics: Work-queue footers/diagnostics and safe-output error-code standardization — adjacent in theme to Cluster 7 but did not cross the outlier size/gap threshold on its own.
• Example PRs: Link work queue footers to creation and completion revisions #72632, Fix generated conclusion references and fetch warnings; pin Redis #71898, Fix missing work-queue diagnostic guidance #71665, [Safe Outputs] Standardize machine-readable error codes #71613, Enable readerstringconv in native and wasm CI #72096

---

Cluster 0: Linter & static-detection fixes

• Size: 27 tasks (4.5%)
• Success Rate: 88.9%
• Top Keywords: detection, coverage, agent, regression, job
• Characteristics: New/fixed Go linters and static-analysis rules (goroutine recovery, flush-discard detection, token-leak detection).
• Example PRs: Prevent skipped threat detection from reporting inherited agent tokens #71933, Detect missing recovery in goroutines launched through local closures #71897, Detect parenthesized Flush discards in uncheckedflushreturn #71896, feat(linters): add readerstringconv analyzer #71891, Close custom-linter detection gaps and exempt safe os error checks #71880

Success Rate by Cluster

Cluster	Theme	Tasks	Success Rate	Top Keywords
5	Coverage/validation/regression	104	85.6%	coverage, add, validation
8	Agent job failure/telemetry	101	81.2%	failure, agent, job
2	Repo/workflow trust & CI	85	82.4%	repository, workflow, review
3	Model routing	83	86.7%	model, copilot, routing
4	Version pins/upgrades	63	87.3%	version, aw, gh aw
7	Ledger/safe-output plumbing	49	73.5%	ledger, safe, output
6	Sous Chef / small bot fixes	46	89.1%	em, aic, sous chef
1	Work-queue/safe-output policy	39	79.5%	queue, work, issue
0	Linter/static detection	27	88.9%	detection, coverage, agent

Full Data Table

50 most recent PRs (number, title, cluster, outcome, keywords)

PR #	Title	Cluster	Outcome	Keywords
#72632	Link work queue footers to creation and completion revisions	C1	Merged	queue, work, issue
#72466	Allow on.bots Apps to check out same-repository PR edits	C2	Closed	repository, workflow, review
#72405	Suppress startup-failure reports for cancelled agent jobs	C8	Merged	failure, agent, job
#72404	Exclude Copilot utility models from alias selection and restore fallbacks	C0	Closed	detection, coverage, agent
#72403	Allow on.bots Apps to check out same-repository PRs on edited events	C2	Merged	repository, workflow, review
#72273	Support Claude on Vertex AI with GCP WIF	C3	Open	model, copilot, routing
#72272	Honor newer cached action pins in runtime setup	C4	Merged	version, aw, gh aw
#72146	Record safe job and script outcomes in unified sessions	C7	Merged	ledger, safe, output
#72138	Record privacy-preserving safe-output outcomes in unified sessions	C8	Merged	failure, agent, job
#72133	Record memory save outcomes in unified sessions and conclusion data	C8	Merged	failure, agent, job
#72131	Enforce ubuntu-slim timeout limits and upgrade default runners	C7	Merged	ledger, safe, output
#72102	Separate Agy bootstrap downloads from runtime firewall egress	C2	Closed	repository, workflow, review
#72101	Narrow Chrome egress to browser-specific Google services	C5	Closed	coverage, add, validation
#72100	Use Gonum helpers for grader descriptive statistics	C5	Merged	coverage, add, validation
#72099	Upgrade Copilot CLI to 1.0.103 with reviewed checksum pin	C4	Merged	version, aw, gh aw
#72098	Propagate configured environments to persistence jobs	C7	Closed	ledger, safe, output
#72097	Deduplicate Copilot agent cleanup logic	C5	Closed	coverage, add, validation
#72096	Enable readerstringconv in native and wasm CI	C1	Closed	queue, work, issue
#72095	Correct map tool usage in built-in ledger smoke workflow	C8	Merged	failure, agent, job
#72074	Trust same-repository bot PRs triggered by labels	C2	Merged	repository, workflow, review
#71957	Bump MCPG to v0.5.0 and firewall to v0.29.0	C4	Merged	version, aw, gh aw
#71947	Fix compile-progress file permission policy violation	C5	Merged	coverage, add, validation
#71934	Preserve harness retry telemetry in spans and usage artifacts	C8	Merged	failure, agent, job
#71933	Prevent skipped threat detection from reporting inherited agent tokens	C0	Merged	detection, coverage, agent
#71932	Harden fetch-script portability and structured lint reporting	C5	Merged	coverage, add, validation
#71927	Separate workflow title parsing from file-I/O benchmark cost	C5	Merged	coverage, add, validation
#71926	Detect missing step references in Actions/AW expressions	C8	Merged	failure, agent, job
#71899	Update Pi CLI default to 1.4.0	C4	Merged	version, aw, gh aw
#71898	Fix generated conclusion references and fetch warnings; pin Redis	C1	Merged	queue, work, issue
#71897	Detect missing recovery in goroutines launched through local closures	C0	Merged	detection, coverage, agent
#71896	Detect parenthesized Flush discards in uncheckedflushreturn	C0	Merged	detection, coverage, agent
#71895	Triage setup-script security findings as false positives	C2	Closed	repository, workflow, review
#71894	Unify GitHub Contents base64 decoding	C5	Merged	coverage, add, validation
#71893	Consolidate public GitHub API GET fallback transport	C5	Merged	coverage, add, validation
#71892	Handle ignored errors and discarded results	C5	Closed	coverage, add, validation
#71891	feat(linters): add readerstringconv analyzer	C0	Merged	detection, coverage, agent
#71890	Refactor long workflow and CLI functions into focused helpers	C5	Closed	coverage, add, validation
#71889	Guard indexing boundaries and prevent cron-step overflow	C5	Merged	coverage, add, validation
#71888	Centralize missing permission scope expansion	C2	Merged	repository, workflow, review
#71887	Deduplicate AW import normalization in source policy lint rules	C5	Merged	coverage, add, validation
#71886	Return stage-aware partial results for MCP compile and audit	C5	Merged	coverage, add, validation
#71885	Freeze compiler-generated Copilot SDK npm installs	C4	Closed	version, aw, gh aw
#71884	Preserve safe-output telemetry and tighten terminal-output guidance	C7	Closed	ledger, safe, output
#71883	Classify engine driver crashes with secret-safe stderr diagnostics	C8	Merged	failure, agent, job
#71882	Fix Windows agent runtime setup, dry-run configuration, and audit identities	C7	Closed	ledger, safe, output
#71881	[WIP] Investigate recurring indexing and ignored-error findings	C2	Closed	repository, workflow, review
#71880	Close custom-linter detection gaps and exempt safe-os errors	C0	Merged	detection, coverage, agent
#71879	Refresh Node image pins and enforce patched runtime dependencies	C4	Merged	version, aw, gh aw
#71878	Read cached state before workflow analysis	C7	Merged	ledger, safe, output
#71850	Track credential handoffs through step outputs in secret-leak checks	C8	Merged	failure, agent, job

Full 597-row dataset was clustered; this table is a recency-ordered sample. Cache updated with all 597 PR numbers for incremental analysis next run.

Key Findings

1. Ledger/safe-output plumbing is the clear friction point. Cluster 7 (49 PRs) merges at 73.5% vs. 83.8% overall — the only cluster meeting both the size (≥15) and gap (≥10pt) thresholds for an outlier.
2. Infrastructure/CI hygiene work merges reliably. Version-pin bumps (C4, 87.3%), model-routing (C3, 86.7%), and linter/detection work (C0, 88.9%) are the most dependable task types — they're narrowly scoped and easy to review.
3. The largest cluster (C5, coverage/validation, 104 PRs) is a "junk drawer." Its generic top keywords (coverage, add, validation, fixes) span many unrelated small hardening tasks; it is not itself an outlier, but its size means a sub-cluster-level investigation could reveal narrower patterns in a future pass with a higher k.

Lowest-Merge Cluster Root Cause

• Cluster: 7 — ledger & safe-output job plumbing
• Merge Rate Gap: 73.5% vs. 83.8% overall (gap: 10.3 points)
• Primary Blocker: Scope/complexity mismatch — most non-merged PRs in this cluster were closed with zero review engagement (no comments, no review rounds), consistent with being superseded or abandoned mid-flight as the ledger/safe-output subsystem underwent rapid, overlapping redesigns (visible in neighboring PR titles: "Consolidate ledger compaction into two Agentic Maintenance jobs", "Move ledger compaction into Agentic Maintenance", "Add declarative ledger types for collection replay", "Remove work-pool ledger support" — four competing/iterative designs for the same subsystem within the period).
• Confidence: Medium — the "silent closure" pattern is strong circumstantial evidence of supersession/abandonment rather than explicit rejection, but no PR in the sample states a reason for closing without comment, so it cannot be confirmed from PR data alone.
• Evidence PRs: Propagate configured environments to persistence jobs #72098, Fix cached audit ledger schema fixture causing CI failure #69940, Make AI credits accounting resilient to unknown pricing #65563 (closed, zero comments/reviews, in an actively-churning area); Preserve safe-output telemetry and tighten terminal-output guidance #71884, Fix Windows agent runtime setup, dry-run configuration, and audit identities #71882, Add declarative ledger types for collection replay and latest-wins tables #64581 (explicit correctness/design pushback); Route Windows CLI integration failures through safe outputs #65402 (blocked: "safe-output submission denied" per automated reviewer)

All 13 non-merged Cluster 7 PRs were examined using cached full PR data (comments, reviews, review decisions):

Blocker Category	Count	Share	Notes
Scope/complexity mismatch (closed, no review engagement — likely superseded by competing ledger redesigns)	9	69%	#72098, #69940, #67889, #65616, #65126, #64496, #68555, #65682, #65563
Review friction (explicit correctness/design/privacy concerns raised)	3	23%	#71884 (secret-leak concern), #71882 (WSL path-corruption bug, CHANGES_REQUESTED), #64581 (ledger invariant bug)
CI/tooling failure (automated safe-output submission blocked)	1	8%	#65402

For comparison, the 10 merged PRs sampled from the same cluster averaged 7.5 comments and 7.2 review rounds each (vs. near-zero for most non-merged PRs) — merged PRs in this cluster typically went through multiple iterative fix-and-re-review cycles rather than being approved on the first pass, reinforcing that this is inherently high-friction, high-iteration subsystem work.

Recommendations

1. Split ledger/safe-output plumbing work into smaller, independently reviewable PRs. The cluster's merged PRs needed 7+ review rounds on average; large persistence/ledger changes should be pre-staged behind an agreed design (ADR) before implementation starts, rather than generating a full implementation that then needs a design pivot.
2. Investigate the "silent closure" pattern directly with the team. 9 of 13 non-merged Cluster 7 PRs closed with zero review engagement — confirm whether these were intentionally superseded by consolidation PRs (expected churn) or stalled/abandoned work that should be re-queued.
3. Keep CI/version-pin and linter-detection tasks as-is. These narrowly-scoped task types (Clusters 0, 3, 4) merge at 86–89% with low friction — a good template for well-specified Copilot tasks.
4. For the outlier cluster specifically: before dispatching new ledger/safe-output PRs, check for in-flight competing redesigns of the same subsystem (e.g. via open PR titles) to avoid assigning tasks that will be immediately superseded.

---

Generated by Prompt Clustering Analysis (Run: 40851798684)

Generated by 📊 Copilot Agent Prompt Clustering Analysis · claude · agent · ~112 AIC · ⌖ ~1 AIC · ⊞ 5K · ◷

• expires on Oct 30, 2026, 2:06 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #73163.