[prompt-clustering] Copilot Agent Prompt Clustering — Daily Analysis (2026-07-06)

[github-actions[bot]]

Summary

Analysis Period: Last 30 days (2026-06-16 → 2026-07-06)
Total Copilot PRs Analyzed: 999 (of 1000 fetched; 1 body too short to cluster)
Clusters Identified: 8 (K-means, k selected over 3–8)
Overall Success Rate: 80.0% merged (799 merged · 194 closed · 6 open)

Clustering was performed on the PR bodies of app/copilot-swe-agent PRs in github/gh-aw, TF‐IDF vectorized (uni+bigrams, min_df=3) and grouped with K‐means. Themes below are labeled from top TF‐IDF terms and representative PRs.

Key Findings

1. Two themes dominate ~44% of all agent work: Engine/schema config & docs (230, 23%) and CI jobs, steps & safe-outputs (213, 21%). These are the workhorse categories.
2. Documentation/config-heavy work has the lowest merge rate. The two lowest-success clusters — Engine/schema & docs (73.0%) and Workflow authoring & prompts (75.8%) — are the ones dominated by docs/prompt/guidance churn, where "correct" is subjective and PRs are more often superseded or closed.
3. Mechanical, well-scoped work merges best. Firewall/network & version bumps (97.6%) and Go linter/helper refactors (85.4%) top the chart — narrow, testable changes with clear acceptance criteria.
4. Data caveat surfaced by clustering itself: cluster C0 ("sous-chef/aic/generated pr") is defined by boilerplate footer text leaking from PR bodies rather than task semantics. Roughly 9% of PRs carry this template, which fragments an otherwise-semantic clustering.

Full Analysis Report

General Insights

• Most common task type: Engine/schema config & docs (23.0% of tasks)
• Highest success rate: Firewall/network & version bumps — 97.6%
• Lowest success rate: Engine/schema config & docs — 73.0%
• Outlier/smallest cohort: WIP CI-failure fixes (17 PRs) — all titled [WIP] Fix failing GitHub Actions job ..., a recurring auto-remediation pattern.

Cluster Analysis

C2 · Engine/schema config & docs — 230 tasks (23.0%), 73.0% merged

Top terms: engine, schema, added, updated, docs, copilot, runtime, new
Engine/runtime config, generated-schema updates, and Quick-Start/docs edits. Largest and lowest-success cluster — docs/config PRs are frequently reworked or closed.
Examples: #43743, #43715, #43697, #43766

C5 · CI jobs, steps & safe-outputs — 213 tasks (21.3%), 79.8% merged

Top terms: step, file, job, output, agent, changes, failure, safe
Workflow job wiring, CI drift checks, safe-output plumbing, PR-reviewer tooling.
Examples: #43747, #43730, #43658, #43716

C7 · Go linters/helpers & pkg refactors — 164 tasks (16.4%), 85.4% merged

Top terms: helper, package, function, path, changes, flagged, added, cases
ESLint-factory rules, AST helper consolidation, false-positive exemptions. Well-scoped, test-backed → high merge rate.
Examples: #43649, #43692, #43731, #43748

C1 · Workflow authoring & prompts — 161 tasks (16.1%), 75.8% merged

Top terms: workflow, workflows, prompt, guidance, issue, added, report, command
Prompt constraints, guidance docs, report/command tuning. Subjective acceptance → below-average merge rate.
Examples: #43744, #43699, #43681, #43659

C0 · Bot-generated fixes (sous-chef/aic) — 87 tasks (8.7%), 85.1% merged

Top terms: aic, sous, sous chef, chef, pr sous, generated pr, pr, generated
Cluster formed by shared boilerplate footer, not a single semantic theme (see caveat). Underlying changes are varied small fixes.
Examples: #43729, #43724, #43732, #43725

C3 · Engine model & sub-agent tuning — 86 tasks (8.6%), 83.7% merged

Top terms: model, workflow, agent, tool, sub, sub agent, copilot, output
Model selection, sub-agent dispatch limits, tool-permission tuning.
Examples: #43672, #43673, #43679, #43621

C6 · Firewall/network & version bumps — 41 tasks (4.1%), 97.6% merged

Top terms: domains, firewall, blocked, aic, workflow, smoke claude, claude, smoke
Firewall domain lists, dependency/CLI version bumps, DNS fixes. Narrow + verifiable → highest success.
Examples: #43664, #43358, #43172, #43056

C4 · WIP CI-failure fixes — 17 tasks (1.7%), 76.5% merged

Top terms: actions, job, fix, plan make, make progress
Auto-remediation of failing GitHub Actions jobs; all [WIP] Fix failing ....
Examples: #43733, #43306, #43305, #43304

Success Rate by Cluster

Cluster	Theme	Tasks	%	Merged	Success
C2	Engine/schema config & docs	230	23.0%	168	73.0%
C5	CI jobs, steps & safe-outputs	213	21.3%	170	79.8%
C7	Go linters/helpers & pkg refactors	164	16.4%	140	85.4%
C1	Workflow authoring & prompts	161	16.1%	122	75.8%
C0	Bot-generated fixes (sous-chef/aic)	87	8.7%	74	85.1%
C3	Engine model & sub-agent tuning	86	8.6%	72	83.7%
C6	Firewall/network & version bumps	41	4.1%	40	97.6%
C4	WIP CI-failure fixes	17	1.7%	13	76.5%

Data Table (25 most recent PRs)

PR #	Title	Cluster	Outcome
#43766	fix(mcp-server): bind HTTP server to 127.0.0.1; document validation escalation	Engine/schema & docs	OPEN
#43748	feat(no-json-stringify-error): scope-aware suggestions	Go linters/helpers	OPEN
#43747	feat: add CI drift check for workflow md vs lock files	CI jobs & safe-outputs	OPEN
#43744	Constrain list_code_scanning_alerts prompt usage	Workflow authoring	OPEN
#43743	cleanup: replace deprecated needs.activation.outputs.*	Engine/schema & docs	MERGED
#43733	[WIP] Fix failing GitHub Actions job: Integration Misc P2	WIP CI-fix	MERGED
#43732	fix: populate engine_id from lock file metadata	Bot-gen fixes	CLOSED
#43731	fix: exempt github_api_helpers.cjs from SEC-004	Go linters/helpers	MERGED
#43730	Add bounded diff pre-fetch to PR Code Quality Reviewer	CI jobs & safe-outputs	MERGED
#43729	fix: honor GH_HOST for --trigger-context issue URLs	Bot-gen fixes	MERGED
#43725	Fix docs dev server network isolation + preload warnings	Bot-gen fixes	CLOSED
#43724	eslint-factory: extend no-json-stringify-error	Bot-gen fixes	MERGED
#43716	fix(security): replace curl|sh installer pipes	CI jobs & safe-outputs	OPEN
#43715	docs: "Why a separate token?" for COPILOT_GITHUB_TOKEN	Engine/schema & docs	CLOSED
#43700	compiler: add evals job for BinEval binary evaluations	Bot-gen fixes	OPEN
#43699	docs: add "When to use" column to CLI commands table	Workflow authoring	MERGED
#43697	docs: reorder Quick Start "What's next?"	Engine/schema & docs	CLOSED
#43696	Clarify Quick Start frontmatter terminology	Engine/schema & docs	MERGED
#43695	docs: consolidate Quick Start API key notes	Engine/schema & docs	MERGED
#43693	feat: add daily-doc-cli-tab-scanner workflow	Engine/schema & docs	CLOSED
#43692	fix(execcommandwithoutcontext): stop scope walk at boundaries	Go linters/helpers	MERGED
#43681	feat: show only avatars in Community Contributions	Workflow authoring	CLOSED
#43679	fix(code-simplifier): allow all bash commands	Model & sub-agent	MERGED
#43673	Constrain Daily Compiler Quality safe-output reports	Model & sub-agent	MERGED
#43672	q: enforce small model for contribution-checker subagent	Model & sub-agent	MERGED

Recommendations

1. Tighten acceptance criteria for docs/config prompts (C2, C1). These two clusters (~39% of volume) sit ~5–7 pts below the mean merge rate. Prompts here should name the exact file/section, the desired end state, and a done-condition, to cut the rework-and-close churn.
2. Strip boilerplate footers before clustering / before agent context. The "sous-chef/aic/generated pr" template dominates C0 and pollutes term space. Filtering it would sharpen clustering and reduce token noise in agent prompts.
3. Lean into the high-success pattern for autonomy. Narrow, verifiable tasks (version bumps, firewall/domain edits, lint-rule refactors) merge at 85–98%. These are the safest candidates to expand agent coverage on.
4. Instrument turn-count/cost. Per-PR turn and cost metrics were unavailable this run (see limitations); wiring them in would let us correlate task type with effort, not just outcome.

Limitations

• Workflow metrics (turns, duration, cost) not available. The pre-fetched pr-full-data cache (PRs 30577–34390) does not overlap the current 30-day window (PRs 39651–43766) — it is stale from a prior period — and these are copilot-swe-agent PRs, not gh-aw engine runs, so aw_info.json turn counts don't apply. Analysis uses PR bodies + outcomes only.
• "Prompt" == PR body. No START COPILOT markers were present; the agent's PR description is used as the task proxy.
• Silhouette scores are low (~0.03). Expected for sparse TF‐IDF text; clusters are semantically coherent by inspection but boundaries are soft. Treat cluster sizes as approximate.

---

Generated by Copilot Agent Prompt Clustering Analysis — Run §28787472646

Generated by 📊 Copilot Agent Prompt Clustering Analysis · 153.4 AIC · ⌖ 19.2 AIC · ⊞ 13.2K · ◷

• expires on Jul 7, 2026, 3:42 AM UTC-08:00