[prompt-clustering] Prompt Clustering Analysis - 2026-10-02

[github-actions[bot]]

Summary

Analysis Period: Last 30 days (2026-09-02 to 2026-10-02)
Total Tasks Analyzed: 589 copilot-agent PRs (448 merged, 134 closed, 7 open)
Clusters Identified: 7
Overall Success Rate (merged): 76%

Headline finding: one cluster — "operational-value grader generation" (60 PRs, 10% of all tasks) — has a 0% merge rate, a 76-point gap below the overall rate, and is the single largest driver of closed-without-merge PRs in the period. See root-cause section below.

Full Analysis Report

General Insights

• Most Common Task Type: Cluster 4 — security/safe-outputs/compiler guardrails (217 PRs, 37% of all tasks)
• Highest Success Rate: Cluster 0 — agentic workflow features & firewall/sandbox infra (93%)
• Most Review-Intensive Tasks: Cluster 0 (avg 13.0 comments, 5.1 reviews/PR) and Cluster 2 — compiler internals/ledger (avg 11.4 comments, 5.8 reviews/PR)
• Outliers: 1 outlier cluster identified (Cluster 3, operational-value grader generation) meeting both size (≥15 PRs) and merge-gap (≥10 points) thresholds
• Data limitation: Workflow-run logs (turn counts, duration, token cost) could not be enriched this run — the GitHub API hit an installation-level rate limit (gh-aw logs --engine copilot). Analysis below uses PR metadata, comments, and reviews only.

Cluster Analysis

Cluster 4: Security & Safe-Outputs Guardrails

• Size: 217 tasks (37% of total)
• Success Rate: 82%
• Top Keywords: workflow, safe, coverage, github, agent, fixes, usage, output
• Characteristics: Compiler-enforced security boundaries, safe-output schema/validation work, prompt/output hardening. Largest and broadly healthy cluster.
• Example PRs: Require explicit scopes for GitHub App tokens at compile time #64976, Embed the bootstrap registration template safely #64974, Document SEC-005 boundary for mention resolution #64973, Standardize safe-output errors and staged asset previews #64972, Claude: default to strict MCP configuration #64970

---

Cluster 0: Agentic Workflow Features & Firewall/Sandbox Infra

• Size: 90 tasks (15% of total)
• Success Rate: 93% (highest of all clusters)
• Top Keywords: firewall, network, blocked, summary, details
• Characteristics: New workflow features, firewall/egress domain handling, sandbox runtime changes. Most reviewed cluster (avg 13.0 comments/PR) but still highest success rate — reviews here are collaborative rather than blocking.
• Example PRs: Add Daily Ecosystem Explorer agentic workflow #64096, Prevent false blocked-domain warnings for successful firewall requests #63246, Filter logs to agentic workflows #63177, Allow add-labels max to set per-call label limit #63044, Route upgrade compilation through full post-processing #63043

---

Cluster 2: Compiler Internals & Dispatch/Ledger

• Size: 74 tasks (13% of total)
• Success Rate: 89%
• Top Keywords: model ledger, TLA+, claims ledger, dispatch coordinator, git commands
• Characteristics: Core internals — formal modeling (TLA+), ledger/claim arbitration, CLI plumbing. Second most review-intensive cluster.
• Example PRs: Model ledger types and persistence in TLA+ #64957, Add a built-in claims ledger with immutable voting #64884, Make setup-cli version optional and support major-version channels #64860, Use argument arrays for PR Git commands #64856, Define deterministic Dispatch Coordinator Claim arbitration #64851

---

Cluster 1: Model/Engine Routing & Copilot CLI

• Size: 72 tasks (12% of total)
• Success Rate: 83%
• Top Keywords: model, copilot, cli, engine, detection
• Characteristics: Model catalog updates, engine resolution/routing, pricing tables, SDK version bumps.
• Example PRs: Honor wire API overrides in Copilot SDK provider resolution #64969, Refresh default agent CLI and Copilot SDK versions #64872, Add GPT-6.1 Sol support to Copilot workflows #64871, Clarify Copilot model rejection diagnostics #64870, Route model classification from workflow task text #64859

---

Cluster 5: Infra, Versions & Dependencies

• Size: 54 tasks (9% of total)
• Success Rate: 74% (near overall baseline)
• Top Keywords: gh aw, workflow, logs, repository, target
• Characteristics: Dependency/version bumps (firewall, mcpg), runner management, integration-test plumbing.
• Example PRs: Bump gh-aw-firewall and gh-aw-mcpg versions #64917, Investigate campaign-query memory estimation gap #64583, Write version output to stdout #64336, Manage self-hosted runner enforcement with gh aw env #64268, Update gh-aw-mcpg to v0.4.26 and gh-aw-firewall to v0.28.25 #63206

---

Cluster 6: CI Failure Fixes

• Size: 22 tasks (4% of total)
• Success Rate: 91%
• Top Keywords: actions, job, fix, failure, run
• Characteristics: Reactive [WIP] Fix failing GitHub Actions job ... tasks auto-filed from CI failures. Small, well-scoped, fast to merge.
• Example PRs: Consolidate ledger compaction into two Agentic Maintenance jobs #64709, [WIP] Fix failing GitHub Actions job Integration: Workflow Misc Part 2 #64076, [WIP] Fix failing GitHub Actions job 'build' #64053, [WIP] Fix failing GitHub Actions job Integration: Workflow Misc Part 2 #63235, Make dev-mode actions folder checkout shallow #63035

---

Cluster 3: Operational-Value Grader Generation — OUTLIER

• Size: 60 tasks (10% of total)
• Success Rate: 0%
• Top Keywords: operational, operational value, evidence, study, metric
• Characteristics: Automated creation/update of operational-value graders and "blocked study" records for individual daily workflows (e.g. daily-caveman-optimizer, daily-byok-ollama-test, cache strategy remediation). See root-cause analysis below — this entire cluster closes without merging.
• Example PRs: Fix operational-value grader created-at acquisition and preserve grader result contract in gh aw logs #60635, Record blocked operational-value study for daily-choice-test #58559, Add operational-value grader for daily-caveman-optimizer #58557, Add operational-value grader for cache strategy remediation #58555, Record blocked operational-value study for daily-byok-ollama-test #58554

Success Rate by Cluster

Cluster	Theme	Tasks	Success Rate	Gap vs Overall (76%)	Avg Comments	Avg Reviews
4	Security & safe-outputs guardrails	217	82%	-6 pts	6.0	3.2
0	Workflow features & firewall/sandbox	90	93%	-17 pts	13.0	5.1
2	Compiler internals & ledger	74	89%	-13 pts	11.4	5.8
1	Model/engine routing & Copilot CLI	72	83%	-7 pts	3.9	1.9
5	Infra, versions & dependencies	54	74%	+2 pts	5.4	2.6
3	Operational-value grader generation	60	0%	+76 pts	0.3	0.2
6	CI failure fixes	22	91%	-15 pts	2.1	1.6

("Gap" is overall-minus-cluster, so positive = worse than baseline.)

Key Findings

1. One cluster accounts for the entire merge-rate drag from otherwise-healthy work. Excluding Cluster 3, the remaining 529 PRs merge at ~89%. Cluster 3 alone pulls the blended rate down to 76%.
2. Engineering-feature clusters (0, 2, 6) are the healthiest, with 89–93% success — concrete, scoped, reviewable changes (firewall behavior, ledger/dispatch internals, CI job fixes) land reliably even with heavy review.
3. Cluster 3 fails almost instantly and almost always without review (see root cause below) — this is a structurally different failure mode from normal review friction or CI breakage seen elsewhere.

Lowest-Merge Cluster Root Cause

• Cluster: 3 — Operational-value grader generation
• Merge Rate Gap: 0% vs 76% overall (gap: 76 points)
• Primary Blocker: Other — systemic bulk-closure without individual review (not CI failure, not substantive review rejection)
• Confidence: High
• Evidence PRs: Record blocked operational-value study for daily-choice-test #58559, Add operational-value grader for daily-caveman-optimizer #58557, Add operational-value grader for cache strategy remediation #58555, Add operational-value grader for Daily Documentation Healer #58454, Fix operational-value grader created-at acquisition and preserve grader result contract in gh aw logs #60635

Supporting evidence (all 60 cluster-3 PRs classified using comments, reviews, labels, and close timing from cached PR data):

Blocker Category	Count	Share	Notes
No review engagement (0 comments, 0 reviews)	54	90%	Closed with no human or bot interaction at all
Triage-bot "batch_review" tag only, then closed	3	5%	Labeled pr-batch:operational-value-grading, pr-action:batch_review by the repo's PR Triage Agent, closed without further discussion
Positive bot acknowledgment, then closed anyway	2	3%	Bot comment praised the contribution ("Trusted team contribution") but PR was still closed
Genuine review iteration, then closed	1	2%	#60635 received 12 review comments requesting fixes (grader conclusion-job wiring) but was closed, not merged

Timing confirms this is systemic rather than case-by-case: median time from open to close was 15 minutes, and 53 of 60 PRs (88%) closed in under 1 hour — far too fast for substantive individual review. This pattern is consistent with this repository's own governance rule for operational-value graders (AGENTS.md: "Do not edit an enabled evaluator independently of its workflow... evaluator-only corrections are prospective"): these PRs look like individually-generated, per-workflow grader additions that get superseded/bulk-closed under the contract process rather than merged one at a time.

---

Recommendations

1. Stop generating per-workflow operational-value grader PRs as individual one-off tasks. Given 90% close with zero review and a 15-minute median lifetime, the current prompt pattern produces work that is structurally never merged — treat this as a batch/contract-driven process (per operational-value-designer skill) rather than tasking the agent with one PR per workflow.
2. For the one cluster-3 PR that did get real review (Fix operational-value grader created-at acquisition and preserve grader result contract in gh aw logs #60635), the root issue was data-plumbing (grader created-at acquisition, result-contract wiring in gh aw logs) — if grader PRs continue, fold that feedback into the generation template so each PR arrives contract-compliant rather than being closed for missing wiring.
3. Keep investing in Clusters 0/2/6-style tasks (firewall/sandbox features, compiler/ledger internals, CI-failure fixes) — these show the highest success rates (89–93%) even under heavy review, suggesting the prompt style and scoping used there generalizes well.
4. Re-run workflow-metrics enrichment (turns/duration/cost via gh-aw logs) once the installation's GitHub API rate limit resets, to add iteration-count context to this clustering — it was unavailable this run.

---

Generated by Prompt Clustering Analysis (Run: §36991894499)

Generated by 📊 Copilot Agent Prompt Clustering Analysis · claude · agent · 140.2 AIC · ⌖ 12.7 AIC · ⊞ 7K · ◷

• expires on Oct 3, 2026, 2:08 AM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-10-03T10:08:39.970Z.

Closed by Workflow