[prompt-clustering] Prompt Clustering Analysis - 2026-09-10

[github-actions[bot]]

Summary

Analysis Period: 2026-08-17 to 2026-09-10 (last 1,000 copilot-authored PRs, ~24 days — the search API cap was hit before reaching a full 30 days back)
Total Tasks Analyzed: 1,000
Clusters Identified: 6
Overall Success Rate: 81.0% (810 merged / 188 closed / 2 open)

Full Analysis Report

General Insights

• Most Common Task Type: Cluster 0 — general workflow/GitHub Actions fixes (598 PRs, 59.8% of all tasks)
• Highest Success Rate: Cluster 1 — "fix failing GitHub Actions job" tasks, 100% merged (23/23)
• Most Complex Tasks: Cluster 4 (firewall/network/audit logging) and Cluster 2 ("PR Sous Chef" automation) — both average 33–54 changed files and 845–859 additions per PR, roughly double the corpus average
• Outliers: 1 outlier cluster flagged (Cluster 3, 61 PRs, 77.7-point merge-rate gap below overall)
• Data note: Workflow run logs (turn counts, duration, cost) were not available in cache for this run, so cluster complexity is approximated using comment/review/file-change counts instead of agent turns.

Cluster Analysis

Cluster 0: General workflow & GitHub Actions fixes

• Size: 598 tasks (59.8% of total)
• Success Rate: 84.8%
• Avg Comments / Reviews: 4.2 / 2.5 — Avg Files Changed: 27.2 — Avg Additions: 372
• Top Keywords: workflow, github, workflows, fixes, pkg, safe, coverage, daily
• Characteristics: The catch-all bucket of day-to-day bug fixes and small feature work across the gh-aw workflow engine — logs command, safe-outputs, enclave/MCP plumbing.
• Example PRs: Fix static GitHub agent enclave startup #59878, Fix local cross-repository workflow resolution in gh aw logs #59859, Improve logs command cache diagnostics and Drain3 training #59823, Clarify workflow source 404s in log consumers #59820, Fix invalid empty GitHub server guard policy for enclave-only GitHub tools #59816

---

Cluster 1: Failing CI job auto-fixes

• Size: 23 tasks (2.3% of total)
• Success Rate: 100%
• Avg Comments / Reviews: 0.9 / 1.0 — Avg Files Changed: 2.2 — Avg Additions: 20
• Top Keywords: job, actions, fix, failing, github, id, check, root
• Characteristics: Small, narrowly-scoped [WIP] Fix failing GitHub Actions job <name> PRs — targeted, single-purpose, and merge cleanly every time.
• Example PRs: [WIP] Fix failing GitHub Actions job build-wasm #59724, [WIP] Fix failing GitHub Actions job Integration: CLI Completion & Other #59539, [WIP] Fix failing GitHub Actions job Check workflow markdown/lock drift #58385, [WIP] Fix failing GitHub Actions job impacted-go-tests #58325, [WIP] Fix failing GitHub Actions job for unit tests (D-G) #58049

---

Cluster 2: "PR Sous Chef" bot automation

• Size: 143 tasks (14.3% of total)
• Success Rate: 92.3%
• Avg Comments / Reviews: 13.3 / 5.2 — Avg Files Changed: 33.8 — Avg Additions: 845
• Top Keywords: sous chef, run, provenance, dispatch
• Characteristics: Larger PRs generated/iterated via the PR Sous Chef re-run loop — high comment/review counts reflect multiple /souschef iterations before merge.
• Example PRs: Preserve valid GitHub App checkout token permissions when permissions.contents: none skips default checkout #59743, Retry transient slash-command provenance checks #59313, Cache GitHub Actions job metadata in logs output #59039, Register replace_label handler in safe-output collect job dispatch map #58917, Add deterministic body footer templates to safe outputs #58841

---

Cluster 3: Operational-value grader sweep (OUTLIER)

• Size: 61 tasks (6.1% of total)
• Success Rate: 3.3% (2/61 merged)
• Avg Comments / Reviews: 0.2 / 0.03 — Avg Files Changed: 2.3 — Avg Additions: 217
• Top Keywords: operational value, study, evidence, metric, grader, blocked
• Characteristics: A batch sweep (branch prefix copilot/operational-value-study-paper-v1*) tasking the agent to design an "operational-value grader" for dozens of other workflows one PR at a time. See root-cause section below.
• Example PRs: Record blocked operational-value study for daily-choice-test #58559, Add operational-value grader for daily-caveman-optimizer #58557, Add operational-value grader for cache strategy remediation #58555, Record blocked operational-value study for daily-byok-ollama-test #58554, Record blocked operational-value study for AWF spec surfacing #58553

---

Cluster 4: Firewall / network / audit logging

• Size: 71 tasks (7.1% of total)
• Success Rate: 94.4%
• Avg Comments / Reviews: 11.6 / 5.7 — Avg Files Changed: 53.6 — Avg Additions: 859
• Top Keywords: firewall, network, domain, summary, audit
• Characteristics: The largest/most complex PRs in the corpus by files-changed — firewall domain decision logging, audit schema exposure, MCP tool-call visibility in logs JSON.
• Example PRs: Classify trusted AWF max-ai-credits HTTP 403 as budget exhaustion and suppress futile Copilot retries #59745, Expose JSON schemas for audit and logs output #59720, Ignore Dependabot PRs in PR Sous Chef #59698, Refresh cached logs JSON in place #59690, Expose identifiable MCP tool calls in logs JSON #59579

---

Cluster 5: Model/engine inventory & version management

• Size: 104 tasks (10.4% of total)
• Success Rate: 76.0%
• Avg Comments / Reviews: 4.0 / 1.8 — Avg Files Changed: 32.2 — Avg Additions: 672
• Top Keywords: copilot, model, codex, engine, claude, pricing
• Characteristics: Keeping the agentic-engine CLI/model inventory current (new model IDs, pricing tables, SDK migrations) — success rate is 5 points below overall, likely reflecting churn as vendor model catalogs change faster than review can keep up.
• Example PRs: Add AI Credits pricing for Gemini 3.8 Flash #59748, Add GPT-6 Astra model inventory support #59711, Fix Copilot SDK model inventory collection #59703, Migrate 80% of Codex workflows to Pi #59695, Update agentic engine CLI versions #59694

Success Rate by Cluster

Cluster	Tasks	Success Rate	Avg Comments	Avg Files	Top Keywords
0	598	84.8%	4.2	27.2	workflow, github, fixes
1	23	100.0%	0.9	2.2	fix failing job
2	143	92.3%	13.3	33.8	sous chef, run
3	61	3.3%	0.2	2.3	operational value, study
4	71	94.4%	11.6	53.6	firewall, network, audit
5	104	76.0%	4.0	32.2	model, copilot, codex

Full Data Table (sample of 30 most recent PRs)

PR #	Title	Cluster	Outcome
59878	Fix static GitHub agent enclave startup	0	Open
59859	Fix local cross-repository workflow resolution in gh aw logs	0	Merged
59823	Improve logs command cache diagnostics and Drain3 training	0	Merged
59820	Clarify workflow source 404s in log consumers	0	Merged
59816	Fix invalid empty GitHub server guard policy for enclave-only GitHub tools	0	Merged
59748	Add AI Credits pricing for Gemini 3.8 Flash	5	Merged
59745	Classify trusted AWF max-ai-credits HTTP 403 as budget exhaustion	4	Merged
59743	Preserve valid GitHub App checkout token permissions	2	Merged
59724	[WIP] Fix failing GitHub Actions job build-wasm	1	Merged
59720	Expose JSON schemas for audit and logs output	4	Merged
59711	Add GPT-6 Astra model inventory support	5	Merged
59703	Fix Copilot SDK model inventory collection	5	Merged
59698	Ignore Dependabot PRs in PR Sous Chef	4	Merged
59695	Migrate 80% of Codex workflows to Pi	5	Merged
59694	Update agentic engine CLI versions	5	Merged
59690	Refresh cached logs JSON in place	4	Merged
59579	Expose identifiable MCP tool calls in logs JSON	4	Merged
59539	[WIP] Fix failing GitHub Actions job Integration: CLI Completion & Other	1	Merged
59313	Retry transient slash-command provenance checks	2	Merged
59039	Cache GitHub Actions job metadata in logs output	2	Merged
58917	Register replace_label handler in safe-output collect job dispatch map	2	Merged
58841	Add deterministic body footer templates to safe outputs	2	Merged
58559	Record blocked operational-value study for daily-choice-test	3	Closed
58557	Add operational-value grader for daily-caveman-optimizer	3	Closed
58555	Add operational-value grader for cache strategy remediation	3	Closed
58385	[WIP] Fix failing GitHub Actions job Check workflow markdown/lock drift	1	Merged
58325	[WIP] Fix failing GitHub Actions job impacted-go-tests	1	Merged
58049	[WIP] Fix failing GitHub Actions job for unit tests (D-G)	1	Merged
57005	Document intent-driven workflow design	3	Merged
55829	Rename aw-value skill to operational-value-designer	3	Merged

(Full 1,000-row dataset was processed for clustering; this table shows a representative sample. See cluster-summary.json in cache for the complete per-cluster PR lists.)

Key Findings

1. Narrow, single-purpose tasks merge almost universally. Cluster 1 ("fix this one failing CI job") hits 100% merge with near-zero review overhead — small blast radius and an objective pass/fail signal (the job goes green) make these the easiest wins in the corpus.
2. The one true outlier is a self-terminating experiment, not a review bottleneck. Cluster 3's 3.3% merge rate is driven by a deliberate batch sweep asking the agent to design an "operational-value grader" for ~60 different workflows — see root-cause below.
3. Bigger PRs correlate with more review cycles but not necessarily lower success. Clusters 2 and 4 have the highest avg comments/files/additions in the corpus yet still merge at 92–94%, suggesting the PR Sous Chef re-run loop and firewall/audit logging work are well-supported by review, not just larger.
4. Model/engine inventory upkeep (Cluster 5) is the weakest "real" cluster. At 76.0% (5 points below overall) it's not flagged as a statistical outlier (gap < 10 pts), but it's the lowest success rate among clusters with genuine human review activity — likely because vendor model catalogs (new IDs, pricing, deprecations) change faster than these PRs can be validated before going stale.

Lowest-Merge Cluster Root Cause

• Cluster: 3 — Operational-value grader sweep
• Merge Rate Gap: 3.3% vs 81.0% overall (gap: 77.7 points)
• Primary Blocker: Scope/complexity mismatch (self-assessed non-viability), not CI failure or review friction
• Confidence: High
• Evidence PRs: Record blocked operational-value study for daily-choice-test #58559, operational-value-study: block copilot-opt (no defensible direct metric) #58535, Block operational-value grader design for copilot-centralization-optimizer #58532, Record operational value study for archivx-agentic-workflows-analyzer #58501, Add deterministic operational-value grading for Archie #58474

59 of 61 PRs in this cluster have zero review comments and zero formal reviews — they were closed without any human ever weighing in. Titles are explicit about the outcome: "Record blocked operational-value study for X (no defensible direct metric)", "Block operational-value grader design for Y". Branch names (copilot/operational-value-study-paper-v1[-again]) confirm this is one coordinated sweep, not organic task variety: the agent was dispatched once per target workflow to attempt designing a deterministic "operational value" grading metric, and in the large majority of cases concluded the metric wasn't definable and self-closed the PR rather than merging speculative grading logic. Only 2 PRs escaped this pattern, and neither is a grader itself — one documents workflow design intent (#57005), the other renames the skill driving the sweep (#55829).

Blocker Category	Count	Share	Notes
Scope/complexity mismatch (self-closed, "no defensible metric")	54	88.5%	Zero comments, zero reviews — closed by agent's own judgment
Other/unknown (closed, 1-2 comments, no formal review)	5	8.2%	Same theme, minor human touch but still closed
Merged (not a grader; meta/docs work)	2	3.3%	#57005, #55829
CI/test failure	0	0%	No evidence of CI-driven closures in this cluster
Review friction	0	0%	No formal reviews recorded on any closed PR in cluster

Recommendations

1. Treat Cluster 1 as the template for high-yield tasks. Single-file, single-purpose "fix this one CI failure" prompts merge at 100% with minimal review overhead — favor this framing when task scope can be narrowed that way.
2. Watch Cluster 5 (model/engine inventory) for staleness. Consider a lighter-weight, higher-cadence update path (e.g. a scheduled diff-only PR) rather than full engine-version PRs, to reduce the 24% closure rate likely caused by catalogs moving before merge.
3. For large multi-file work (Clusters 2 and 4), current review support is working — don't change it. High comment/review counts here correlate with high merge rates, not friction; the process is functioning as intended.
4. Outlier Cluster Recommendation: Before launching another broad "design a grader/metric for N workflows" sweep, add an explicit low-cost precheck step (e.g., a single scoping issue or a criteria checklist) so the ~88% of cases without a defensible metric are filtered out before opening a PR, rather than opening and immediately self-closing 54 PRs.

---

Generated by Prompt Clustering Analysis (Run: 34462352293)

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• api.anthropic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.anthropic.com"

See Network Configuration for more information.

Generated by 📊 Copilot Agent Prompt Clustering Analysis · claude · agent · 132.8 AIC · ⌖ 11.5 AIC · ⊞ 16.2K · ◷

• expires on Sep 11, 2026, 2:09 AM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-09-11T10:09:39.955Z.

Closed by Workflow