[prompt-clustering] Prompt Clustering Analysis - 2026-08-13

[github-actions[bot]]

Summary

Analysis Period: Last 30 days (2026-07-28 to 2026-08-13)
Total Tasks Analyzed: 1,011 copilot-agent PRs
Clusters Identified: 7
Overall Merge Rate: 73.8% (746 merged / 249 closed unmerged / 16 open)

Limitation: Workflow-run logs (aw_info.json turn/duration/cost data) did not finish downloading within this run's time budget, so this analysis is based on PR body/comment/review/file-change data only — no turn-count or cost metrics.

Full Analysis Report

General Insights

• Most common task type: Cluster 6, "Tests, Lint & Conformance Hardening" (316 PRs, 31% of all tasks) — watchdogs, spec/conformance tests, linters, safe-output plumbing fixes.
• Highest success rate: Cluster 5, "Feature Adds (Sous-Chef groomed)" at 90.9% — these PRs carry a footer showing they were touched by the repo's "PR Sous Chef" grooming bot, and they cluster around clean, self-contained engine/CLI features.
• Most files touched on average: Cluster 5 again (83.6 files/PR avg) and Cluster 0 (32.8 files/PR avg) — both touch generated/lock files broadly.
• Outliers: 2 clusters fall ≥10 points below the 73.8% baseline merge rate with ≥15 PRs — Cluster 0 (52.1%) and Cluster 2 (54.7%).

Cluster Analysis

Cluster 6: Tests, Lint & Conformance Hardening

• Size: 316 tasks (31.3% of total)
• Success Rate: 80.1%
• Avg comments/reviews: 6.1 / 3.0
• Top Keywords: test, fix, safe, cjs, coverage, tests, outputs, changes
• Characteristics: Bug fixes and hardening work around test coverage, linters, safe-outputs plumbing, and watchdogs. High volume, high merge rate — the bread-and-butter maintenance category.
• Example PRs: Restore safe-outputs MCP startup by staging data_schema_normalizer.cjs #48496, Strengthen actionpins spec tests for cache keys, fallback auditing, and container digest validation #48497, fix: add post-result watchdog to codex_harness to prevent CLI hang-on-exit #48499, linters: add bytescomparezero analyzer for bytes.Compare(... ) == 0 equality checks #48500, Add formal conformance suite for AWF config source contract and DriftRecord invariants #48503

---

Cluster 3: Agent/Workflow Behavior Tuning

• Size: 282 tasks (27.9% of total)
• Success Rate: 75.2%
• Avg comments/reviews: 4.6 / 2.4
• Top Keywords: agent, daily, mcp, engine, prompt, model, lock, claude
• Characteristics: Tuning the daily agentic workflows themselves — persona guidance, model routing, prompt-bloat reduction, lock-file regeneration. This is the project's own dogfooding category (workflows editing workflows).
• Example PRs: Reduce ambient prompt bloat in daily workflow specs (script extraction, DIFC chart consolidation, proxy mode) #48483, Strengthen persona workflow guidance for path scoping, escalation dedup, and toolset selection #48502, Stabilize Daily Agent of the Day sub-agent model routing for Copilot chat endpoint compatibility #48504, Restore compat-aware Copilot toolcache fallback for explicit pins #48507, Stabilize smoke-copilot-auto lock output by skipping dynamic auto pricing injection #48516

---

Cluster 1: Go Refactoring & Cleanup

• Size: 141 tasks (13.9% of total)
• Success Rate: 70.9%
• Avg comments/reviews: 7.2 / 3.3 (highest review engagement outside Cluster 5)
• Top Keywords: cli, refactor, helpers, error, custom, string, command, helper
• Characteristics: Internal Go code-quality work — string building, error propagation, context threading, lint-monster diagnostics. Draws the most review discussion per PR, consistent with refactors needing more scrutiny.
• Example PRs: refactor: replace string += in loops with strings.Builder (41 findings) #48484, refactor: introduce named type LLMProvider string to replace bare string threading #48485, refactor: replace duplicate experimentStateJSON/experimentRunRecord with exported ExperimentState/ExperimentRunRecord #48486, fix: propagate download errors through StartDockerImageDownload join function #48487, refactor: thread context.Context through ExecGH/RunGH/RunGHCombined wrappers #48488

---

Cluster 4: Docs & CLI Polish

• Size: 82 tasks (8.1% of total)
• Success Rate: 72.0%
• Avg comments/reviews: 2.6 / 1.4
• Top Keywords: docs, ai, request, agent, agentic, txt, site, reference
• Characteristics: Documentation site and CLI-consistency fixes (engine reference docs, robots.txt/llms.txt, emoji rendering).
• Example PRs: Add GitHub-style emoji rendering to the docs renderer #48601, fix: CLI consistency — secrets bootstrap engine list, ghes flag punctuation, docs description #48659, fix(docs): add antigravity to engines reference doc + add engine-doc sync test #48678, [WIP] Add llms.txt for docs site for improved GEO score #48691, feat(docs): add robots.txt with AI-bot Allow rules for docs site #48697

---

Cluster 0: Container/Dependency Vuln Fixes — OUTLIER

• Size: 71 tasks (7.0% of total)
• Success Rate: 52.1% (−21.7 points vs. baseline)
• Avg comments/reviews: 3.0 / 1.4
• Top Keywords: image, container, pin, mcp, scan, high, json, 27
• Characteristics: Daily container/dependency vulnerability-scan remediations — bumping or removing pinned container images and packages flagged for CVEs. See root-cause section below.
• Example PRs: [WIP] Fix vulnerabilities in ghcr.io/github/gh-aw-firewall/api-proxy #48498, fix: update brace-expansion to 5.0.8 and upgrade Alpine base image to 3.22 #48521, Remove vulnerable gh-aw-firewall/squid:0.27.41 pin from embedded container pin catalogs #48534, fix: remove ghcr.io/chopratejas/headroom due to 67 CVEs and 131 license violations #48550, fix: remove docker.io/mcp/brave-search — Critical/High CVEs, no upstream fix available #48561

---

Cluster 2: WIP Draft Fixes — OUTLIER

• Size: 64 tasks (6.3% of total)
• Success Rate: 54.7% (−19.1 points vs. baseline)
• Avg comments/reviews: 2.3 / 1.3
• Top Keywords: wip, wip fix, fix, failing, job, validation, make, tests
• Characteristics: 100% of this cluster's PR titles are literally prefixed [WIP] — in-progress exploratory fixes for smoke-test failures, agent status, backlog consolidation. By definition these are draft/checkpoint PRs rather than finished work.
• Example PRs: [WIP] Fix fork-backed create_pull_request duplicate Authorization error #48501, [WIP] Fix sub agent status in smoke tests #48512, [WIP] Consolidate duplicate function-length backlog issues into one #48692, [WIP] Update agent persona exploration findings #48702, [WIP] Fix add-labels to allow opt-out of pull-requests write access #48912

---

Cluster 5: Feature Adds (Sous-Chef groomed)

• Size: 55 tasks (5.4% of total)
• Success Rate: 90.9% (best in the set)
• Avg comments/reviews: 13.9 / 5.7 (most engagement of any cluster)
• Top Keywords: sous, sous chef, chef, em, sub, aic, souschef, workflow_id sous
• Characteristics: Small, well-scoped engine/CLI features (model aliases, watchdog retries, network-domain aliasing) that were run through the repo's "PR Sous Chef" auto-grooming workflow before merge — heavier review engagement correlates with the highest success rate in the dataset.
• Example PRs: Retry Copilot startup no-output watchdog failures in scheduled/push runs #48514, feat: honor engine.version for copilot; add version tests for copilot and gemini #48519, fix: support network.allowed-domains as alias for network.allowed #48522, feat: error when agent-job step outputs are referenced in the prompt body #48610, feat: add auto as a proper builtin model alias for Copilot #48655

Success Rate by Cluster

Cluster	Theme	Tasks	Success Rate	Avg Comments	Avg Reviews	Top Keywords
6	Tests, Lint & Conformance	316	80.1%	6.10	3.00	test, fix, coverage
3	Agent/Workflow Tuning	282	75.2%	4.62	2.37	agent, daily, mcp, engine
1	Go Refactoring	141	70.9%	7.16	3.29	cli, refactor, helpers
4	Docs & CLI Polish	82	72.0%	2.59	1.39	docs, ai, agentic
0	Container/Dep Vuln Fixes	71	52.1%	2.97	1.42	image, container, pin, scan
2	WIP Draft Fixes	64	54.7%	2.33	1.33	wip, failing, job
5	Feature Adds (Sous-Chef)	55	90.9%	13.87	5.67	sous chef, aic

Full Data Table (representative sample, 6 per cluster)

PR #	Title	Cluster	Outcome
#48498	[WIP] Fix vulnerabilities in ghcr.io/github/gh-aw-firewall/api-proxy	Container/Dependency Vuln Fixes	closed
#48521	fix: update brace-expansion to 5.0.8 and upgrade Alpine base image	Container/Dependency Vuln Fixes	merged
#48534	Remove vulnerable gh-aw-firewall/squid:0.27.41 pin	Container/Dependency Vuln Fixes	closed
#48550	fix: remove ghcr.io/chopratejas/headroom due to 67 CVEs	Container/Dependency Vuln Fixes	merged
#48561	fix: remove docker.io/mcp/brave-search — Critical/High CVEs	Container/Dependency Vuln Fixes	merged
#48562	Fix serena-mcp-server container: rebuild on python:3.13-slim	Container/Dependency Vuln Fixes	closed
#48484	refactor: replace string += in loops with strings.Builder	Go Refactoring & Cleanup	closed
#48485	refactor: introduce named type LLMProvider string	Go Refactoring & Cleanup	merged
#48486	refactor: replace duplicate experimentStateJSON/experimentRunRecord	Go Refactoring & Cleanup	merged
#48487	fix: propagate download errors through StartDockerImageDownload	Go Refactoring & Cleanup	merged
#48488	refactor: thread context.Context through ExecGH/RunGH wrappers	Go Refactoring & Cleanup	closed
#48520	fix: resolve three targeted lint-monster diagnostics	Go Refactoring & Cleanup	merged
#48501	[WIP] Fix fork-backed create_pull_request duplicate Authorization error	WIP Draft Fixes	closed
#48512	[WIP] Fix sub agent status in smoke tests	WIP Draft Fixes	closed
#48692	[WIP] Consolidate duplicate function-length backlog issues into one	WIP Draft Fixes	closed
#48702	[WIP] Update agent persona exploration findings	WIP Draft Fixes	closed
#48912	[WIP] Fix add-labels to allow opt-out of pull-requests write access	WIP Draft Fixes	closed
#48927	[WIP] Fix premature-closure regression of workflow-health issues	WIP Draft Fixes	closed
#48483	Reduce ambient prompt bloat in daily workflow specs	Agent/Workflow Behavior Tuning	closed
#48502	Strengthen persona workflow guidance for path scoping	Agent/Workflow Behavior Tuning	merged
#48504	Stabilize Daily Agent of the Day sub-agent model routing	Agent/Workflow Behavior Tuning	closed
#48507	Restore compat-aware Copilot toolcache fallback for explicit pins	Agent/Workflow Behavior Tuning	closed
#48516	Stabilize smoke-copilot-auto lock output	Agent/Workflow Behavior Tuning	merged
#48551	fix: constrain list_code_scanning_alerts with state:open	Agent/Workflow Behavior Tuning	merged
#48601	Add GitHub-style emoji rendering to the docs renderer	Docs & CLI Polish	closed
#48659	fix: CLI consistency — secrets bootstrap engine list	Docs & CLI Polish	merged
#48678	fix(docs): add antigravity to engines reference doc	Docs & CLI Polish	closed
#48691	[WIP] Add llms.txt for docs site for improved GEO score	Docs & CLI Polish	closed
#48697	feat(docs): add robots.txt with AI-bot Allow rules	Docs & CLI Polish	closed
#48736	docs: explain why WASM builds no-op repository feature validation	Docs & CLI Polish	merged
#48514	Retry Copilot startup no-output watchdog failures	Feature Adds (Sous-Chef groomed)	merged
#48519	feat: honor engine.version for copilot	Feature Adds (Sous-Chef groomed)	merged
#48522	fix: support network.allowed-domains as alias	Feature Adds (Sous-Chef groomed)	closed
#48610	feat: error when agent-job step outputs are referenced	Feature Adds (Sous-Chef groomed)	merged
#48655	feat: add auto as a proper builtin model alias for Copilot	Feature Adds (Sous-Chef groomed)	merged
#48657	Add ESLint rule for duplicate constant values	Feature Adds (Sous-Chef groomed)	merged
#48496	Restore safe-outputs MCP startup by staging data_schema_normalizer.cjs	Tests, Lint & Conformance Hardening	merged
#48497	Strengthen actionpins spec tests for cache keys	Tests, Lint & Conformance Hardening	merged
#48499	fix: add post-result watchdog to codex_harness	Tests, Lint & Conformance Hardening	merged
#48500	linters: add bytescomparezero analyzer	Tests, Lint & Conformance Hardening	closed
#48503	Add formal conformance suite for AWF config source contract	Tests, Lint & Conformance Hardening	closed
#48505	Add missing godoc for validateToolConfiguration	Tests, Lint & Conformance Hardening	closed

Full dataset covers 1,011 PRs; this table shows 6 representative rows per cluster (42 of 1,011).

Key Findings

1. The two lowest-merge clusters are both process artifacts, not quality problems. Cluster 2 is literally 100% [WIP]-titled draft PRs — by construction these are checkpoints, not finished submissions. Cluster 0 (container/dependency vuln fixes) is a daily-cadence automated scan that frequently supersedes its own prior-day PR before anyone reviews it.
2. Review engagement correlates strongly with merge rate. Cluster 5 (90.9% merged) averages 13.9 comments and 5.7 reviews per PR — far above any other cluster — while the two outlier clusters average under 3 comments and under 1.5 reviews. More eyes on a PR track with (not against) getting it merged here.
3. Nearly a third of all agent work (Cluster 6, 316 PRs) is test/lint/conformance hardening, and it merges at a healthy 80.1% — suggesting this is the safest, highest-throughput category of task to hand to the agent.
4. The project's own workflow-tuning loop (Cluster 3, 282 PRs) is the second-largest category — the agent spends significant time adjusting its own daily agentic workflows, at a 75.2% merge rate.

Lowest-Merge Cluster Root Cause

• Cluster: 0 — Container/Dependency Vuln Fixes
• Merge Rate Gap: 52.1% vs 73.8% overall (gap: 21.7 points)
• Primary Blocker: Abandonment without review (superseded by later automated runs), not review rejection or CI failure
• Confidence: High (79.4% of this cluster's non-merged PRs)
• Evidence PRs: [WIP] Fix vulnerabilities in ghcr.io/github/gh-aw-firewall/api-proxy #48498, Remove vulnerable gh-aw-firewall/squid:0.27.41 pin from embedded container pin catalogs #48534 (closed with 0 comments and 0 reviews — never touched by a human)

Blocker Category	Count	Share of non-merged	Notes
Abandoned, no review	27	79.4%	Zero comments and zero reviews before close — consistent with being superseded by the next day's vulnerability scan
Closed, other	5	14.7%	Some human interaction but no clear rejection reason captured
Review friction	1	2.9%	Explicit "changes requested" / security-concern language
CI/test failure	1	2.9%	Explicit failing-check language in comments

The daily container-scan workflow appears to open a fresh remediation PR each run; if the prior day's PR (frequently [WIP]-titled, 13 of 71 in this cluster) hasn't merged yet, it gets closed unreviewed once the newer scan supersedes it. This is a cadence/dedup issue in the automation, not a code-quality or reviewer-availability problem.

Recommendations

1. De-duplicate daily container-scan PRs before opening new ones. Have the vulnerability-scan workflow check for an existing open PR on the same image/pin before creating another, or auto-close-with-comment (not silent close) referencing the successor PR, so the 21.7-point merge-rate gap in Cluster 0 reflects genuine review outcomes rather than churn.
2. Don't count [WIP]-titled PRs (Cluster 2) in success-rate dashboards as if they were finished submissions — track them as a separate "in-progress" funnel stage with its own completion metric (e.g., % that later reopen as a non-WIP merged PR), rather than penalizing overall agent performance.
3. Route more of the daily agentic-workflow-tuning backlog (Cluster 3) through extra review passes, similar to what Cluster 5 gets — the strong correlation between review volume and merge rate suggests lightweight additional review (not necessarily human) could lift merge rates in the 70-75% clusters.
4. Outlier-cluster mitigation for Cluster 0: (a) add supersede-detection to the container-scan workflow, (b) post an explicit close-reason comment instead of silent closure, (c) track the "abandoned without review" rate directly as a workflow health metric.

---

Generated by Prompt Clustering Analysis (Run: 31689329524)

Generated by 📊 Copilot Agent Prompt Clustering Analysis · agent · 127.9 AIC · ⌖ 50.2 AIC · ⊞ 14.5K · ◷

• expires on Aug 14, 2026, 2:33 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #52691.