[prompt-clustering] Prompt Clustering Analysis - 2026-09-15

[github-actions[bot]]

Summary

Analysis Period: Last ~30 days (PRs created 2026-08-18 → 2026-09-15)
Total Tasks Analyzed: 1000 copilot-agent PRs
Clusters Identified: 6
Overall Merge Rate: 81.6% (816 merged / 180 closed / 4 open)

Full Analysis Report

General Insights

• Most Common Task Type: Cluster 3 — general daily workflow fixes, docs, and safe-output tweaks (606 PRs, 61% of all tasks).
• Highest Success Rate: Cluster 0 — internal Go refactors/hardening (93.5% merged, 124 PRs).
• Most Engagement-Heavy: Cluster 4 — firewall/network/security hardening averages 16.3 comments and 5.9 reviews per PR, the most scrutinized category.
• Outlier: Cluster 5 — "operational-value grading" scaffolding tasks, 60 PRs, only 1.7% merged. This is a 79.9-point gap below the overall rate and is investigated in detail below.
• Workflow turn/duration metrics were unavailable this run (gh-aw logs CLI was not present in the sandbox), so this analysis relies on PR/comment/review signals only, not agent turn counts.

Cluster Analysis

Cluster 3: Daily workflow fixes, docs & safe-output tweaks

• Size: 606 tasks (60.6% of total)
• Merge Rate: 85.3%
• Avg Comments / Reviews: 5.1 / 2.7
• Avg Files Changed: 29.8 (avg +445 lines)
• Top Keywords: workflow, github, workflows, fixes, coverage, safe, daily, md
• Characteristics: The catch-all bucket — day-to-day bug fixes, docs redirects, safe-output/schema tweaks, and workflow config corrections. High volume, solidly above-average merge rate.
• Example PRs: Add next step to PR triage failure message #53798 (PR triage failure message), Fix GEO optimizer project-site AI discovery false negatives #53800 (GEO optimizer false negatives), Fix JSON-array expressions in approve workflow run safe outputs #53812 (JSON-array safe-output expressions), Redirect Astro wizard docs to hosted wizard #53815 (Astro docs redirect), Remove hidden README and docs navigation text #53816 (remove hidden nav text)

---

Cluster 0: Internal refactors & test hardening

• Size: 124 tasks (12.4%)
• Merge Rate: 93.5% (highest)
• Avg Comments / Reviews: 14.2 / 5.7 (heavily reviewed despite high merge rate)
• Avg Files Changed: 33.3 (avg +876 lines — the biggest diffs of any cluster)
• Top Keywords: (noisy — see note below) — titles show test-file splitting, safe-output accessor refactors, parser hardening
• Characteristics: Large, deliberate refactors of Go internals (parser, safe-outputs, caching). Despite being the biggest diffs, they merge at the highest rate — heavy review clearly converges rather than blocks.
• Example PRs: Split oversized test files compiler_jobs_test.go and compiler_safe_outputs_config_test.go #53818 (split oversized test files), Refactor safe-output repo target accessors #53838 (refactor safe-output accessors), Extract shared safe-output target parsing #53841 (extract shared target parsing), Harden post-agent cache Git commits #53879 (harden post-agent cache commits), Refactor parser import/path helpers to eliminate wasm/native drift and duplicate logic #53895 (parser import/path refactor)
• Note: TF-IDF top terms for this cluster (em, sub, aic, chef, sous) are boilerplate leaking in from an authoring-tool footer, not real topical keywords — cluster theme was derived from titles instead.

---

Cluster 2: Automated CI-failure fixes ("[WIP] Fix failing GitHub Actions job ...")

• Size: 30 tasks (3.0%)
• Merge Rate: 93.3%
• Avg Comments / Reviews: 1.3 / 1.2 (lowest engagement among merged clusters — these are narrow, templated fixes)
• Avg Files Changed: 12.1 (avg +70 lines — smallest diffs)
• Top Keywords: actions job, job, actions, github actions, fix, failing
• Characteristics: A dedicated auto-triggered task type that reacts to a single failing CI job and patches it. Small, focused, fast to review, merges reliably.
• Example PRs: [WIP] Fix failing GitHub Actions job Integration: Workflow Compiler #54008, [WIP] Fix failing GitHub Actions job Integration: Workflow Infra #54032, [WIP] Fix failing GitHub Actions job Integration: Workflow Misc Part 2 #54088, [WIP] Fix failing GitHub Actions job lint-go #54089, [WIP] Fix failing GitHub Actions job Integration: Workflow Misc Part 2 #54109

---

Cluster 4: Firewall, network & security hardening

• Size: 87 tasks (8.7%)
• Merge Rate: 92.0%
• Avg Comments / Reviews: 16.3 / 5.9 (most-discussed cluster)
• Avg Files Changed: 64.6 (avg +917 lines — largest average footprint)
• Top Keywords: firewall, network, details, summary, blocked
• Characteristics: Security-sensitive changes (domain allowlists, permission modes, signed-commit fallbacks). Large diffs and heavy review, but still merges at a strong rate.
• Example PRs: Centralize engine default domain sets #55461 (centralize engine domain sets), Write MCP CLI wrapper scripts with owner-only permissions (0o700) instead of 0o755 #55587 (owner-only MCP wrapper permissions), Make engine domain sets explicit opt-in #56447 (explicit opt-in domain sets), Document Copilot org-billing tip opt-out #56564 (org-billing docs), Fall back to unsigned push instead of failing when a rebase hits a genuine merge conflict in pushSignedCommits #56568 (unsigned-push fallback)

---

Cluster 1: Engine/model parity (Copilot, Codex, OpenAI)

• Size: 93 tasks (9.3%)
• Merge Rate: 79.6% (roughly baseline)
• Avg Comments / Reviews: 5.0 / 2.1
• Avg Files Changed: 41.2 (avg +717 lines)
• Top Keywords: model, codex, copilot, engine, openai, gpt, workflow, agent
• Characteristics: Cross-engine parity work — docs and code keeping Copilot/Codex/custom-engine behavior aligned. Roughly average outcomes, moderate diff size.
• Example PRs: docs: close non-Copilot engine parity gaps for gh aw init and custom engine frontmatter #53834, docs: default Quick Start auth section to Copilot, name the curl fallback trigger #53959, Surface Codex CLI's internal log output when it exits non-zero with no console error text #54298, Refactor Copilot guardrails into shared baseline import across workflows #54300, Remove Copilot ripgrep install dependency from generated workflow contract #54305

---

Cluster 5: "Operational-value grading" scaffolding — OUTLIER

• Size: 60 tasks (6.0%)
• Merge Rate: 1.7% (1 merged / 59 closed)
• Avg Comments / Reviews: 0.2 / 0.02 — essentially no engagement
• Avg Files Changed: 2.3 (avg +221 lines — small, templated diffs)
• Top Keywords: operational, operational value, study, evidence, metric
• Characteristics: A large fan-out of near-identical PRs, each adding an "operational-value grader" or "blocked study record" for a different target workflow (Duplicate Code Detector, Issue Triage Agent, ACE Editor, agent-job-health, ...). Full root-cause analysis below.
• Example PRs: Rename aw-value skill to operational-value-designer and update grader guidance references #55829 (the one that merged — a skill rename, not a grader), Add operational-value grading for Duplicate Code Detector #58452, Add operational-value grader for Issue Triage Agent #58453, Add operational-value grader for Daily Documentation Healer #58454, Add operational-value grading for A/B testing advisor #58464

Success Rate by Cluster

Cluster	Theme	Tasks	Success Rate	Avg Comments	Avg Files	Top Keywords
0	Internal refactors/hardening	124	93.5%	14.2	33.3	refactor, parser, safe-output
1	Engine/model parity	93	79.6%	5.0	41.2	model, codex, copilot, openai
2	Automated CI-failure fixes	30	93.3%	1.3	12.1	actions job, failing
3	Daily workflow fixes/docs	606	85.3%	5.1	29.8	workflow, github, coverage
4	Firewall/network hardening	87	92.0%	16.3	64.6	firewall, network, blocked
5	Operational-value grading (outlier)	60	1.7%	0.2	2.3	operational value, study, evidence

Full Data Table (representative sample, 6 per cluster — 1000 PRs analyzed in total)

Sample PR Table (36 of 1000 rows)

PR #	Title	Cluster	Outcome	Comments	Reviews	Files
53818	Split oversized test files compiler_jobs_test.go...	C0	merged	23	5	28
53838	Refactor safe-output repo target accessors	C0	merged	44	9	3
53841	Extract shared safe-output target parsing	C0	closed	14	6	13
53879	Harden post-agent cache Git commits	C0	merged	24	5	2
53895	Refactor parser import/path helpers...	C0	merged	29	10	10
53896	Fix package-level mutable slice/map reassignment...	C0	merged	29	3	4
53834	docs: close non-Copilot engine parity gaps...	C1	merged	2	2	3
53959	docs: default Quick Start auth section to Copilot...	C1	closed	0	0	1
54298	Surface Codex CLI's internal log output...	C1	merged	21	7	265
54300	Refactor Copilot guardrails into shared baseline...	C1	closed	1	0	45
54305	Remove Copilot ripgrep install dependency...	C1	merged	2	1	2
54306	Treat provider invalid_request_error as terminal...	C1	merged	10	4	2
54008	[WIP] Fix failing GitHub Actions job Integration: Workflow Compiler	C2	merged	1	1	1
54032	[WIP] Fix failing GitHub Actions job Integration: Workflow Infra	C2	merged	3	1	1
54088	[WIP] Fix failing GitHub Actions job Integration: Workflow Misc Part 2	C2	merged	1	1	2
54089	[WIP] Fix failing GitHub Actions job lint-go	C2	merged	1	1	1
54109	[WIP] Fix failing GitHub Actions job Integration: Workflow Misc Part 2	C2	merged	1	1	2
54588	[WIP] Fix failing GitHub Actions job lint-go-custom	C2	merged	1	1	5
53798	Add next step to PR triage failure message	C3	merged	1	1	2
53800	Fix GEO optimizer project-site AI discovery false negatives	C3	merged	1	1	2
53812	Fix JSON-array expressions in approve workflow run safe outputs	C3	merged	11	8	8
53815	Redirect Astro wizard docs to hosted wizard	C3	merged	1	1	11
53816	Remove hidden README and docs navigation text	C3	merged	3	1	3
53817	Fix chronic Smoke Copilot AOAI dispatch failure...	C3	merged	3	1	5
55461	Centralize engine default domain sets	C4	merged	22	15	6
55587	Write MCP CLI wrapper scripts with owner-only permissions (0o700)...	C4	merged	43	11	2
56447	Make engine domain sets explicit opt-in	C4	merged	15	2	303
56564	Document Copilot org-billing tip opt-out	C4	merged	54	5	16
56568	Fall back to unsigned push instead of failing...	C4	open	64	7	5
56572	Clarify fallback issue paths for manual PR creation	C4	merged	46	12	4
55829	Rename aw-value skill to operational-value-designer...	C5	merged	4	1	9
58452	Add operational-value grading for Duplicate Code Detector	C5	closed	2	0	3
58453	Add operational-value grader for Issue Triage Agent	C5	closed	2	0	3
58454	Add operational-value grader for Daily Documentation Healer	C5	closed	2	0	3
58464	Add operational-value grading for A/B testing advisor	C5	closed	1	0	4
58465	Add ACE Editor operational-value grader	C5	closed	0	0	4

Key Findings

1. The pipeline is healthy overall: 81.6% merge rate across 1000 PRs, holding steady in the 77–85% band over the last ~18 days (see trend chart) — no sign of quality drift.
2. Big, well-scoped refactors outperform small ad-hoc changes: Cluster 0 (large internal refactors, avg +876 lines) merges at 93.5%, the highest of any cluster — heavy review (avg 5.7 reviews) converges rather than stalls.
3. Narrow, single-purpose automation merges reliably: The "[WIP] Fix failing GitHub Actions job" cluster (C2) merges at 93.3% with minimal review overhead (avg 1.2 reviews) — a good template for low-friction automated fixes.
4. One initiative accounts for nearly a third of all closed PRs: Cluster 5's 59 closed PRs are 33% of all 180 closed PRs in the dataset, despite being only 6% of total volume — a single systemic pattern, not diffuse quality issues.

Lowest-Merge Cluster Root Cause

• Cluster: 5 — "Operational-value grading" scaffolding (graders/study records fanned out across dozens of different target workflows: Duplicate Code Detector, Issue Triage Agent, ACE Editor, agent-job-health, agent-persona-explorer, A/B testing advisor, etc.)
• Merge Rate Gap: 1.7% vs 81.6% overall (gap: 79.9 points)
• Primary Blocker: Scope/redundancy — bulk closure of a mass fan-out, not CI failure or review rejection
• Confidence: Medium (inferred from behavioral signal; the dataset has no explicit "closed as duplicate/superseded" reason field, so intent isn't directly confirmed)
• Evidence PRs: Add operational-value grading for Duplicate Code Detector #58452, Add operational-value grader for Issue Triage Agent #58453, Add operational-value grader for Daily Documentation Healer #58454, Add ACE Editor operational-value grader #58465, Record blocked operational-value study for agent-job-health #58466, Add operational-value grading for agent performance analysis #58467, Record blocked operational-value study for agent-persona-explorer #58468, Add blocked operational-value study for agentic-token-trend-audit #58469, Record blocked operational-value study for AI Moderator #58470, Record blocked operational-value study for API consumption report #58471

Evidence supporting this conclusion, from inspecting all 60 PRs in the cluster:

• 54 of 60 (90%) received zero comments and zero reviews before being closed — no CI failure output, no reviewer pushback, no bot triage note explaining rejection.
• Median time from open to close was ~15 minutes (range: 2.9 min to 205 min) — far too fast for a human code review cycle, consistent with a bulk sweep/cleanup rather than case-by-case evaluation.
• Titles are templated across dozens of unrelated target workflows created in rapid succession (e.g., PRs Add ACE Editor operational-value grader #58465–Record blocked operational-value study for API consumption report #58471 opened 6–14 seconds apart), indicating a single generator task spun out one PR per workflow rather than a coordinated batch.
• The one PR that did merge (Rename aw-value skill to operational-value-designer and update grader guidance references #55829, a skill rename) is qualitatively different — it received real back-and-forth review (4 comments including a maintainer request: "@copilot also update the instructions file in .github/aw/graders") and was a documentation/naming change, not a new grader scaffold.
• No comments in the cluster mention CI failures, test failures, or explicit security/compliance concerns — ruling out "CI/test failure" and "review friction" as the driver for the other 59 PRs.

Blocker Category	Count	Share	Notes
CI/test failure	0	0%	No failing-check references found in any comment/review
Review friction	1	2%	Only #55829 had substantive review discussion (and it merged)
Scope/redundancy (bulk closure, near-zero engagement)	54	90%	Zero comments/reviews, closed within minutes, templated fan-out
Other/unknown	5	8%	Minor bot triage comments present but no reviewer engagement; same fast-close pattern

Recommendations

1. Consolidate fan-out initiatives into fewer, coordinated PRs. Rather than generating one PR per target workflow for a new automation pattern (like operational-value graders), batch related workflows into a single PR or a small tracked series with an issue for scope sign-off before implementation work begins.
2. Gate large PR fan-outs with an upfront design/scope check. A short issue or proposal validating that a pattern (e.g., "add an operational-value grader") is wanted for a given target before generating the implementation PR would avoid the 59-PR waste seen in Cluster 5.
3. Keep leaning on narrow, single-purpose automation for CI fixes (Cluster 2) and large, deliberate refactors with heavy review (Cluster 0) — both patterns are working well (93%+ merge rates) and are worth using as templates for other task types.
4. Outlier-cluster-specific: For any future "operational-value grading" or similar meta-tooling work, require a closing comment stating the reason (duplicate/superseded/out-of-scope) when bulk-closing related PRs — this analysis had to infer intent from timing and silence, and an explicit reason would make future audits (and course-correction) much faster.

---

Generated by Prompt Clustering Analysis (Run: 34954435293)

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• api.anthropic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.anthropic.com"

See Network Configuration for more information.

Generated by 📊 Copilot Agent Prompt Clustering Analysis · claude · agent · 186.9 AIC · ⌖ 10.6 AIC · ⊞ 16.2K · ◷

• expires on Sep 16, 2026, 2:15 AM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-09-16T10:15:27.254Z.

Closed by Workflow