[prompt-clustering] Prompt Clustering Analysis - 2026-09-25

[github-actions[bot]]

Summary

Analysis Period: 2026-08-25 to 2026-09-25 (last 30 days)
Total Tasks Analyzed: 733 PRs authored by the Copilot coding agent
Clusters Identified: 7
Overall Success Rate: 80.8% (592 merged / 733 total)

Full Analysis Report

General Insights

• Most Common Task Type: Cluster 3 — "Workflow & Shared Infrastructure Maintenance" (314 tasks, 42.8% of all PRs)
• Highest Success Rate: Cluster 2 — "Model/Engine Pinning" at 94.4% merge rate
• Most Complex Tasks: Cluster 1 — "CLI / gh-aw Tooling" averages 71.4 files changed per PR, by far the largest footprint of any cluster
• Outliers: 1 outlier cluster identified (Cluster 0), with a merge rate 77.5 points below the overall baseline

Cluster Analysis

Cluster 1: Workflow & Shared Infrastructure Maintenance

• Size: 314 tasks (42.8% of total)
• Success Rate: 88.5%
• Avg Comments / Reviews: 7.3 / 3.7
• Avg Files Changed: 26.4
• Top Keywords: workflow, github, workflows, shared, coverage, run, existing, daily
• Characteristics: Routine maintenance and enhancement of .github/workflows/*.md agentic workflow definitions — adding network presets, hardening existing workflows, refactoring shared logic.
• Example PRs: Fix github-discussion-query payload scaling by eliminating jq argument growth #55560, Add node ecosystem preset to network.allowed for Daily Documentation Diagram #55561, Codex harness: add working-set rebuild circuit breaker to stop runaway context loops #55562, Fix Daily Team Evolution title date to derive from window_start, not run date #55573, Stabilize Daily Performance Summary 90-day metrics #55574, Pin deterministic git-scoped cloc invocation in Daily Code Metrics workflow #55575

Representative example: "Add node ecosystem preset to network.allowed for Daily Documentation Diagram" (#55561)

---

Cluster 2: CLI / gh-aw Tooling

• Size: 140 tasks (19.1% of total)
• Success Rate: 82.9%
• Avg Comments / Reviews: 7.6 / 3.0
• Avg Files Changed: 71.4 (highest of any cluster)
• Top Keywords: cli, aw, gh, v0, copilot, gh aw, awf, workflow
• Characteristics: Changes to the gh-aw CLI itself and its supporting Go code — the widest-reaching, most structurally complex category.
• Example PRs: Write MCP CLI wrapper scripts with owner-only permissions (0o700) instead of 0o755 #55587, Add Playwright readability checks to Archivx diagrams #55639, Bump default CLI/MCP versions: Claude Code, Codex, Pi, GitHub MCP Server #55676, Fix swapped openai/anthropic ports in AWF api-proxy gateway constants #55693, Fix false-positive gate failure in Daily Container Image Security Scan #55695, Expose Copilot web_fetch when tools.web-fetch is enabled #55696

Representative example: "Add Playwright readability checks to Archivx diagrams" (#55639)

---

Cluster 3: PR Sous Chef / Automated PR Assistance

• Size: 71 tasks (9.7% of total)
• Success Rate: 93.0% (highest volume + high success combination)
• Avg Comments / Reviews: 12.0 / 6.4 (highest interaction volume of any cluster)
• Avg Files Changed: 52.5
• Top Keywords: sous, sous chef, chef, pr sous
• Characteristics: Tasks driven by the "PR Sous Chef" assistant workflow — heavily reviewed, iterative PRs with the most back-and-forth of any cluster, yet still merge at a high rate.
• Example PRs: Remove bounded-queries support (replaced by enclaves) #55839, Harden compat.json blocked-version enforcement #55869, feat(audit): add cluster and pattern analysis to cross-run audit #56072, Always configure OTLP from enterprise default variables and secrets #56353, [WIP] Fix failing GitHub Actions job for misc part 2 #56689, Fix Playwright MCP TOML sandbox configuration #56800

Representative example: "Harden compat.json blocked-version enforcement" (#55869)

---

Cluster 4: Safe Outputs & Workflow Failure Handling

• Size: 70 tasks (9.5% of total)
• Success Rate: 82.9%
• Avg Comments / Reviews: 6.8 / 3.2
• Avg Files Changed: 35.2
• Top Keywords: safe, outputs, safe output, safe outputs, output, issue, workflow, failure
• Characteristics: Work on the safe-outputs subsystem (issue/PR/comment creation guardrails) and workflow failure diagnostics.
• Example PRs: Update Archivx to publish diagrams through pull requests #55563, Post a closure comment when the pre-created steering PR is discarded after a failed run #55564, Honor ignore-missing-branch-failure in the safe-outputs MCP server when a PR branch is deleted mid-run #55571, Refactor pull request steering to use issues #55572, [WIP] Fix daily code debt cleanup issue with no safe outputs #55624, Add PDF-aware preliminary investigation to arXiv research #55644

Representative example: "Refactor pull request steering to use issues" (#55572)

---

Cluster 5: Operational-Value Grading Batch Initiative ⚠️ OUTLIER

• Size: 61 tasks (8.3% of total)
• Success Rate: 3.3% (2 merged / 61) — 77.5 points below the 80.8% baseline
• Avg Comments / Reviews: 0.2 / 0.03 (near-zero interaction)
• Avg Files Changed: 2.3
• Top Keywords: operational, operational value, value, evidence, study, adoption, metric, record
• Characteristics: A single bulk batch of near-identical PRs, one per repository workflow, each attempting to add a deterministic "operational-value grader" or record a "value study." See root-cause analysis below.
• Example PRs: Rename aw-value skill to operational-value-designer and update grader guidance references #55829, Document intent-driven workflow design #57005, Add operational-value grading for Duplicate Code Detector #58452, Add operational-value grader for Issue Triage Agent #58453, Add operational-value grader for Daily Documentation Healer #58454, Add operational-value grading for A/B testing advisor #58464

---

Cluster 6: Model/Engine Pinning

• Size: 54 tasks (7.4% of total)
• Success Rate: 94.4% (highest of any cluster)
• Avg Comments / Reviews: 5.1 / 2.9
• Avg Files Changed: 24.6
• Top Keywords: codex, model, copilot, gpt, openai, workflow, claude, daily
• Characteristics: Small, well-scoped PRs pinning workflows to a specific engine/model (Codex, Copilot, GPT, Claude) — narrow diffs, low review overhead, near-universal acceptance.
• Example PRs: Fix AI Moderator Codex crash: seed CODEX_HOME into cloud-hypervisor filesystem allow-list #55711, Pin Linter Miner to a supported Copilot model #55837, docs: clarify Pi provider-prefixed model configuration #55865, Add daily dashboard language specification review #55867, Fix pricing discrepancies for gpt-5.6-sol and gemini-3.6-flash in models.json #56272, [aw] Route Daily Evals Feature Report through Copilot model path #56327

Representative example: "Pin Linter Miner to a supported Copilot model" (#55837)

---

Cluster 7: GitHub Actions Job Fixes

• Size: 23 tasks (3.1% of total)
• Success Rate: 91.3%
• Avg Comments / Reviews: 1.6 / 1.3
• Avg Files Changed: 15.3
• Top Keywords: actions, job, github actions, fix, failing, github, work, check
• Characteristics: Targeted fixes to failing GitHub Actions jobs/checks — small, high-confidence patches.
• Example PRs: [WIP] Fix failing GitHub Actions job Integration: Workflow Features #55638, Treat GitHub Actions bot PRs as trusted contributions #55645, [WIP] Fix failing lint-go-golangci GitHub Actions job #55648, [WIP] Fix failing GitHub Actions job 'Unit tests (A-C)' #56345, [WIP] Fix failing GitHub Actions job 'Unit tests (S-Z and other)' #56346, [WIP] Fix failing GitHub Actions job Integration: Workflow Misc Part 2 #56430

Representative example: "Treat GitHub Actions bot PRs as trusted contributions" (#55645)

Success Rate by Cluster

Cluster	Tasks	Success Rate	Avg Comments	Avg Reviews	Avg Files Changed	Top Keywords
1 — Workflow & Infra Maintenance	314	88.5%	7.3	3.7	26.4	workflow, github, shared, coverage
2 — CLI / gh-aw Tooling	140	82.9%	7.6	3.0	71.4	cli, aw, gh, v0, copilot
3 — PR Sous Chef	71	93.0%	12.0	6.4	52.5	sous chef, chef
4 — Safe Outputs & Failure Handling	70	82.9%	6.8	3.2	35.2	safe outputs, issue, failure
5 — Operational-Value Grading (outlier)	61	3.3%	0.2	0.03	2.3	operational value, evidence, study
6 — Model/Engine Pinning	54	94.4%	5.1	2.9	24.6	codex, model, copilot, gpt
7 — GitHub Actions Job Fixes	23	91.3%	1.6	1.3	15.3	actions, job, fix, failing

Full Data Table (sample)

Representative PRs per cluster (full dataset: 733 PRs, see cache for complete assignments)

PR #	Title	Cluster	Outcome
55561	Add node ecosystem preset to network.allowed for Daily Documentation Diagram	1	Merged
55639	Add Playwright readability checks to Archivx diagrams	2	Merged
55869	Harden compat.json blocked-version enforcement	3	Merged
55572	Refactor pull request steering to use issues	4	Merged
58452	Add operational-value grading for Duplicate Code Detector	5	Closed
55837	Pin Linter Miner to a supported Copilot model	6	Merged
55645	Treat GitHub Actions bot PRs as trusted contributions	7	Merged

Key Findings

1. Routine workflow maintenance dominates and succeeds: Nearly half of all agent PRs (Cluster 1, 314 PRs) are incremental maintenance of existing agentic workflow files, merging at a healthy 88.5%.
2. Small, narrowly-scoped tasks merge best: The two highest success-rate clusters — model/engine pinning (94.4%) and GitHub Actions job fixes (91.3%) — are also among the smallest in file-change footprint (24.6 and 15.3 files avg), reinforcing that narrow, well-defined tasks are the agent's sweet spot.
3. High review volume doesn't hurt success when the task is well-defined: PR Sous Chef tasks (Cluster 3) get the most comments/reviews of any cluster (12.0/6.4 avg) yet still merge at 93%, showing iteration itself isn't the problem — task framing is.
4. One batch-generated initiative accounts for nearly all failures: Cluster 5 (Operational-Value Grading) alone contributes 59 of the ~141 closed-without-merge PRs in the entire 30-day window — roughly 42% of all failures came from a single batch initiative representing just 8.3% of total volume.

Lowest-Merge Cluster Root Cause

• Cluster: Cluster 5 — Operational-Value Grading Batch Initiative
• Merge Rate Gap: 3.3% vs 80.8% overall (gap: 77.5 points)
• Primary Blocker: Scope/complexity mismatch, compounded by unexplained bulk process closure
• Confidence: Medium-high
• Evidence PRs: Add operational-value grading for Duplicate Code Detector #58452, Add operational-value grader for Issue Triage Agent #58453, Add operational-value grader for Daily Documentation Healer #58454, Add ACE Editor operational-value grader #58465, Record blocked operational-value study for agent-job-health #58466, [WIP] Add operational-value grader for blog-auditor workflow #58507, Record blocked operational-value design for bot detection #58508, [WIP] Design deterministic operational-value grader for breaking-change-checker #58509, [WIP] Add operational-value grader for code scanning fixer #58521, [WIP] Design operational-value grader for codex-github-remote-mcp-test #58525

What happened: On 2026-09-04, a single automated initiative opened ~59 nearly-identical PRs in a ~5-hour window (10:42–15:32 UTC), each attempting to add a deterministic "operational-value grader" or record a "value study" for one specific repository workflow (one PR per workflow, e.g. Duplicate Code Detector, Issue Triage Agent, Daily Documentation Healer, etc.). Of these:

• 27 PRs (44%) were explicitly titled "Record blocked operational-value study for X" — the agent itself reported it could not complete the grading task, most likely due to insufficient operational/instrumentation data for that workflow.
• 5 PRs (8%) were titled "[WIP]," indicating incomplete implementations.
• 29 PRs (48%) were substantive "Add operational-value grader for X" attempts with no explicit blocker in the title.
• 54 of 61 PRs (89%) received zero comments, and only 1 PR received a substantive review — which was positive ("This looks ready for review. No actionable feedback needed"). Every PR in this batch was closed within 10–95 minutes of creation.
• The only 2 merged PRs in this cluster were meta/infrastructure PRs about the initiative itself (renaming the grading skill, documenting workflow-design intent) — not per-workflow grader implementations.

This pattern — near-zero review engagement, rapid uniform closure, and a large self-reported "blocked" rate — indicates the initiative was closed in bulk shortly after creation, rather than rejected through normal code review. There is no evidence of CI failures or reviewer pushback driving the closures.

Blocker Category	Count	Share	Notes
CI/test failure	0	0%	No evidence of failing checks in comments/reviews
Review friction	0	0%	Only 1 substantive review existed, and it was positive
Scope/complexity mismatch	32	52%	27 self-reported "blocked" + 5 "WIP" — task infeasible for many target workflows (insufficient instrumentation/data)
Other/unknown	29	48%	Zero-comment closures with no stated rationale — consistent with bulk/process closure rather than individual review

Recommendations

Based on clustering analysis:

1. Keep investing in narrow, well-scoped task generation: Model/engine pinning and GitHub Actions job fixes show that small, targeted diffs merge at 90%+ — prefer decomposing large initiatives into this shape.
2. Don't discourage heavy review iteration: PR Sous Chef shows that high comment/review volume correlates with success, not failure, when the underlying task is well-defined — iteration is healthy when it's substantive.
3. Watch Cluster 2 (CLI/tooling) file-change footprint: At 71.4 files/PR average, this cluster carries the highest structural risk; consider whether large CLI PRs could be split for easier review, even though current merge rate (82.9%) is still solid.
4. Outlier Cluster Recommendation: Before fanning a batch initiative out across the entire workflow inventory, run a small pilot (5–10 workflows) to validate feasibility first. Add an automated pre-check that verifies a workflow has sufficient operational/instrumentation data before opening a grader PR, so PRs destined to be self-reported "blocked" are never created. When an initiative is abandoned mid-batch, close remaining PRs via a single tracking issue with an explanatory comment rather than silently closing dozens of individual PRs — this preserves audit trail and avoids wasted agent/CI minutes on the ~48% of PRs closed with no stated reason.

---

Generated by Prompt Clustering Analysis (Run: 36120322349)

Generated by 📊 Copilot Agent Prompt Clustering Analysis · claude · agent · 132 AIC · ⌖ 10.3 AIC · ⊞ 16.3K · ◷

• expires on Sep 26, 2026, 2:05 AM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-09-26T10:05:25.933Z.

Closed by Workflow