[prompt-clustering] Prompt Clustering Analysis - 2026-08-05

[github-actions[bot]]

Summary

Analysis Period: 2026-07-16 to 2026-08-05 (last ~20 days of available PR history)
Total Copilot PRs Analyzed: 1,110 (489 with sufficient body text for clustering)
Clusters Identified: 4
Overall Merge Rate: 80.0%

Full Analysis Report

General Insights

• Most common task type: broad gh-aw tooling development (workflow/copilot/lint/CI plumbing) — Cluster 3, 70.8% of all analyzed PRs.
• Highest success rate: "suffix fixes" / CLI-consistency cleanup work — Cluster 2, 93.5% merged.
• Most distinctive automated category: [WIP] Fix failing GitHub Actions job ... auto-remediation PRs — Cluster 0, a small (3.7%) but very coherent group with the lowest merge rate (72.2%) and smallest diffs (avg 1.9 files changed).
• Non-issue-driven origin: this repository dogfoods gh-aw itself — the majority of PRs are opened by automated workflows, not by a human filing an issue for Copilot to pick up. See "PR Origin Breakdown" below.
• Operational signal: 34 PRs (7.0%) had at least one outbound network request blocked by the sandbox firewall during the agent run, visible via an auto-injected "Firewall blocked N domain(s)" warning in the PR body.

PR Origin Breakdown

Roughly 4 in 10 analyzed PRs carry an automated "Generated by [Workflow Name]" attribution footer, revealing which internal agentic workflow opened them — this is a repo that dogfoods its own automation heavily.

Source	Count	%	Merge Rate	Avg Comments
(no attribution — direct/issue-driven task)	292	59.7%	82.9%	5.3
PR Sous Chef	157	32.1%	93.6%	14.0
Daily Container Image Security Scan	11	2.2%	0.0%	0.0
Squad Plan	8	1.6%	0.0%	0.0
Workflow Normalizer	4	0.8%	0.0%	0.2
GEO Optimizer Daily Audit	4	0.8%	0.0%	0.0
Deep Report	3	0.6%	0.0%	0.7
others (≤2 each)	10	2.0%	mixed	—

PR Sous Chef — an automated nudge/fix-up workflow — is responsible for nearly a third of all PRs and has the highest merge rate of any large group (93.6%, well above the 82.9% baseline for direct tasks), suggesting its narrowly-scoped, mechanical fixes are unusually reliable. In contrast, several report/scan-style workflows (Container Security Scan, Squad Plan, GEO Optimizer) show a 0% merge rate — consistent with them opening informational/report PRs that aren't intended to be merged as code changes rather than representing agent failures.

Cluster Analysis

Cluster 3: General gh-aw Tooling & Workflow Development

• Size: 346 tasks (70.8% of total)
• Merge Rate: 76.6%
• Avg reviews/commits/files: 3.1 / 3.9 / 27.8
• Top Keywords: workflow, aw, github, md, copilot, model, workflows, add
• Characteristics: The catch-all bucket for day-to-day feature work and bug fixes across the gh-aw codebase — new workflow features, engine/model config, linting, docs. Wide diffs, moderate review overhead.
• Example PRs: feat: surface failed checks in pr-sous-chef nudge comments #45957, fix(evals): parse YES/NO answers from Pi JSONL engine log format #45960, fix: pin actionlint to v1.7.12 and eliminate version drift across all references #45962, Refactor smoke workflow imports with shared smoke-base bundle #45963, eslint-factory: consolidate fs-sync try/catch rules and unify ESM/CJS fs binding detection #45964

---

Cluster 2: Suffix / Consistency Fixes

• Size: 93 tasks (19.0%)
• Merge Rate: 93.5% (highest of any sizeable cluster)
• Avg reviews/commits/files: 5.5 / 6.4 / 31.0
• Top Keywords: suffix fixes, fixes, run, branch, pr, details, url
• Characteristics: Targeted correctness/consistency clean-ups — CLI audit follow-ups, config-parsing edge cases, refactors of specific modules. Despite the highest review/comment volume, these convert to merges at the highest rate, suggesting well-scoped, verifiable changes that survive iteration.
• Example PRs: Disable checkout by default for pull_request_target workflows #46771, Treat expression-backed engine.model as omitted when empty #46777, Copilot: make MCP CLI prompt list reflect actual mounted servers under --disable-builtin-mcps #46778, fix(cli): address 16 CLI consistency issues from 2026-07-20 audit #46854, Refactor bootstrap profile runner into focused modules #46855

---

Cluster 1: Meta / Exploratory Agent Tasks

• Size: 32 tasks (6.5%)
• Merge Rate: 81.2%
• Avg files changed: 43.1 (largest average diff of any cluster)
• Top Keywords: pull request, created, ai agent, configuration
• Characteristics: Open-ended asks — "create a PR for the current branch," analyzing generation issues, checking squad/skill files. Broad scope leads to the largest diffs.
• Example PRs: Updating selection screen and reusing Astro starlight #46782, Fixing model name normalizer logic #49001, Analyzing agentic_maintenance.yml generation issues #50107, Configuring frontmatter for bundled artifact folders #50108, Checking squad files and skills presence for agent #50124

---

Cluster 0: CI-Failure Auto-Remediation

• Size: 18 tasks (3.7%, smallest cluster)
• Merge Rate: 72.2% (lowest of any cluster)
• Avg files changed: 1.9 (smallest diffs of any cluster)
• Top Keywords: job, actions, fix failing, github actions
• Characteristics: Highly formulaic [WIP] Fix failing GitHub Actions job ... PRs, almost certainly auto-opened in response to a red CI check. Small, targeted diffs but the lowest merge rate — consistent with these being first-pass attempts at flaky/hard CI failures that sometimes need a follow-up or get superseded.
• Example PRs: [WIP] Fix failing GitHub Actions job agent #45935, [WIP] Fix failing GitHub Actions job Integration: CLI Add & List Commands #46160, [WIP] Fix failing GitHub Actions job 'agent' #46766, [WIP] Fix failing GitHub Actions job for CLI completion #46859, [WIP] Fix failing GitHub Actions job Integration: Workflow Misc Part 2 #47266

Success Rate by Cluster

Cluster	Tasks	% of Total	Merge Rate	Avg Files Changed	Top Keywords
3 — General tooling	346	70.8%	76.6%	27.8	workflow, aw, github, copilot
2 — Suffix/consistency fixes	93	19.0%	93.5%	31.0	suffix fixes, run, branch, pr
1 — Meta/exploratory	32	6.5%	81.2%	43.1	pull request, ai agent, configuration
0 — CI auto-remediation	18	3.7%	72.2%	1.9	fix failing, github actions

Key Findings

1. Narrow, mechanical fixes outperform broad feature work. Cluster 2 (consistency/suffix fixes) merges at 93.5% vs. 76.6% for general tooling work in Cluster 3, despite similar diff sizes — task scoping correlates more with success than diff size.
2. PR Sous Chef is the dominant automated contributor and its most reliable one. At 32.1% of all PRs and a 93.6% merge rate, this nudge-style workflow outperforms the "no attribution" baseline (direct/issue-driven tasks, 82.9%) by over 10 points.
3. Report/scan-style workflows (Container Security Scan, Squad Plan, GEO Optimizer, Deep Report) show 0% merge rates — by design these appear to open informational PRs rather than mergeable code changes, so a 0% rate here is not itself a red flag, but is worth confirming matches intent.
4. CI auto-remediation is the weakest-performing category (72.2% merge rate, smallest diffs) — these are reactive, single-issue fixes for a failing check, and the lower success rate suggests some fraction of flaky/hard-to-diagnose CI failures aren't resolved on the first attempt.
5. 7.0% of analyzed PRs hit a sandboxed network firewall block during the agent run — a useful operational metric for tracking egress-allowlist gaps as workflows evolve.

Recommendations

1. Investigate Cluster 0 (CI auto-remediation) failures — pull the 5 closed-but-unmerged examples from this cluster and check whether they're flaky-test false positives, genuine hard failures needing human triage, or duplicate/superseded attempts.
2. Study what makes PR Sous Chef / Cluster 2 style tasks succeed — their prompts are narrow and mechanical; consider applying the same scoping discipline to broader Cluster 3 tooling tasks where feasible (e.g., splitting large feature PRs into smaller consistency-fix-shaped units).
3. Confirm 0%-merge automated report workflows are working as intended — if Container Security Scan / Squad Plan / GEO Optimizer PRs are meant to be closed after review rather than merged, consider whether "PR" is the right output type for them (an issue or discussion may better reflect that they're reports, not code changes).
4. Track the firewall-block rate over time as a leading indicator — a rising trend across future runs would suggest workflows need updated network.allowed entries rather than one-off fixes.

Limitations

• Workflow run metrics (turn counts, duration, token cost) could not be retrieved: the agenticworkflows logs lookup for the last 30 days timed out (60s context deadline) and was not retried further to stay within this run's time budget. This report is based on PR/GitHub metadata only (comments, reviews, commits, files changed) as a proxy for iteration effort.
• 621 of 1,110 PRs (56%) had bodies too short/generic to include in clustering (e.g., single-line auto-generated titles with no body); these are included in the origin-attribution counts but not the 4-cluster breakdown.
• k=4 was chosen via an elbow heuristic on inertia; cluster boundaries are approximate and some overlap exists between Clusters 1 and 3.

---

Generated by Prompt Clustering Analysis (Run: 30997823120)

Generated by 📊 Copilot Agent Prompt Clustering Analysis · agent · 110 AIC · ⌖ 43.1 AIC · ⊞ 13.7K · ◷

• expires on Aug 6, 2026, 2:59 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #50830.