[prompt-clustering] Prompt Clustering Analysis - 2026-09-07

[github-actions[bot]]

Summary

Analysis Period: 2026-08-11 to 2026-09-07 (last 30 days)
Total Tasks Analyzed: 1226 copilot-authored PRs (945 merged, 224 closed, 57 open)
Clusters Identified: 4
Overall Success Rate: 77.1% (945 / 1226)

Full Analysis Report

General Insights

• Most Common Task Type: Cluster 0 — general "workflow / MCP / copilot / daily" feature and maintenance work (731 PRs, 60% of all tasks).
• Highest Success Rate: Cluster 2 — "sous chef"-style refactor/consolidation tasks (87.5% merged), though see data-quality caveat below.
• Most Complex Tasks: Cluster 2 also has the largest footprint (avg 740 additions / 37.5 files / 13.2 comments / 5.0 reviews per PR) — these are large, heavily-reviewed structural changes.
• Outliers: 1 cluster flagged (Cluster 1, "operational-value grader" tasks) — 65 PRs at a 9.2% merge rate vs. 77.1% overall (a 67.9-point gap), by far the largest and most severe outlier seen in the last several days of this analysis.

Data-quality note: cluster 2's top TF-IDF terms ("em", "sub", "aic", "sous", "chef") come from a recurring bot-generated PR-body footer ("PR Sous Chef", AIC scoring) that leaked into the vectorized text alongside the real task description. The cleaning step strips the footer text but not fully; cluster 2 should be read as "large refactor/consolidation PRs that happen to include this footer," not a distinct task type. This is a known limitation carried over from prior runs and worth fixing in the prompt-cleaning regex.

Workflow metrics limitation: agenticworkflows logs (turn counts, duration, cost) did not return within the time budget available for this run, so turn/cost columns are omitted below. Analysis relies on PR-level signals (comments, reviews, review decision, size) only.

Cluster Analysis

Cluster 0: General workflow/agent feature & maintenance work

• Size: 731 tasks (59.6% of total)
• Success Rate: 77.6%
• Top Keywords: workflow, add, workflows, mcp, copilot, daily, agent, coverage
• Characteristics: The catch-all bucket for day-to-day feature additions, new daily workflows, MCP integration work, and coverage/testing improvements. Roughly tracks the overall merge rate — no unusual friction.
• Example PRs: Allow repository_dispatch user rate limits #52101, Add formal RS-05a checkout gate coverage #52103, Add Agent Job Health Monitor to own fleet-wide agent-job failure rate #52108, [WIP] Add AI-crawler robots.txt allow rules and llms.txt for the docs site #52115, Address Persona Explorer findings: trigger ambiguity, digest grouping, compliance example #52121

---

Cluster 1: Operational-value grader tasks (outlier)

• Size: 65 tasks (5.3% of total)
• Success Rate: 9.2%
• Top Keywords: operational value, operational, blocked, grader, add, daily, copilot
• Characteristics: A batch initiative to add a per-workflow "operational-value grader" (one PR per existing daily workflow, e.g. duplicate-code-detector, issue-triage-agent, doc-healer, etc.). Smallest average diff of any cluster (avg 224 additions, 2.6 files) but by far the worst outcome.
• Example PRs: Introduce typed enums for workflow fixed-value fields #54803, Harden operational-value grader execution and make evaluator timeouts configurable #55445, Rename aw-value skill to operational-value-designer and update grader guidance references #55829, Add YAML dashboard language specification #55841, Lint shell scripts declared in workflow frontmatter #56150

---

Cluster 2: Large refactor / consolidation PRs ("sous chef" footer artifact)

• Size: 303 tasks (24.7% of total)
• Success Rate: 87.5%
• Top Keywords: em, sub, aic, sous, chef, run (bot-footer artifact — see data-quality note)
• Characteristics: The largest, most heavily-reviewed PRs (avg 740 additions, 37.5 files, 13.2 comments, 5.0 reviews) — structural refactors, compiler/type consolidation, spec tightening. High review investment correlates with the highest merge rate.
• Example PRs: Give the four log-entry structs a shared LogEntry interface #52107, Split pkg/workflow/compiler_types.go by lifecycle: options, mutators, types #52109, Tighten SPDD compliance specs and fixtures #52111, Add preview Cloud Hypervisor runtime support for sandbox.agent.runtime with docker-sbx parity #52119, Consolidate JobStep/JobStepData identical structs in pkg/cli #52120

---

Cluster 3: CI / job / Actions failure fixes

• Size: 127 tasks (10.4% of total)
• Success Rate: 84.3%
• Top Keywords: fix, job, actions, check, failure, github actions, daily
• Characteristics: Targeted bug fixes for failing CI jobs, lint/compliance errors, and GitHub Actions workflow failures. Smaller, focused diffs (avg 211 additions) with solid merge rate.
• Example PRs: Fix Linter Miner crash: unsupported model: inherited on inline sub-agents #52133, Fix error message compliance in runs_on_validation, frontmatter_extraction_metadata, reactions, call_workflow_validation, and ambient_folders #52177, Fix error message compliance in 5 pkg/workflow files #52180, Fix coverage.findProfile path matching so perf-linter coverage gating actually applies #52566, Fix Design Decision Gate false failures caused by the LLM invocation cap #52836

Success Rate by Cluster

Cluster	Tasks	Success Rate	Avg Additions	Avg Files	Avg Reviews	Top Keywords
0 — General workflow/agent work	731	77.6%	371.4	22.5	2.15	workflow, mcp, copilot, daily
1 — Operational-value grader (outlier)	65	9.2%	223.8	2.6	0.18	operational value, blocked, grader
2 — Large refactor/consolidation	303	87.5%	740.1	37.5	5.01	sous chef footer artifact, refactor
3 — CI/Actions failure fixes	127	84.3%	210.7	18.6	1.92	fix, actions, failure, check

Trend (overall merge rate, last 8 analyzed windows)

Date	Overall Merge Rate	Total PRs
2026-08-29	79.9%	—
2026-08-30	84.0%	—
2026-08-31	80.8%	—
2026-09-02	81.1%	—
2026-09-03	81.1%	—
2026-09-04	84.5%	—
2026-09-05	77.1%	1197
2026-09-06	80.5%	1000
2026-09-07 (today)	77.1%	1226

The overall rate has drifted down slightly over the last week (mid-84% → high-70s%). The operational-value grader outlier cluster has appeared in this same shape (~65 PRs, ~9% merge rate) across multiple recent runs — it is a persistent, not one-off, drag on the overall rate.

Full Data Table (representative sample)

Sample PRs from the outlier cluster (Cluster 1)

PR #	Title	Outcome	Reviews	Comments
58452	Add operational-value grading for Duplicate Code Detector	Closed	0	2
58453	Add operational-value grader for Issue Triage Agent	Closed	0	2
58454	Add operational-value grader for Daily Documentation Healer	Closed	0	2
58464	Add operational-value grading for A/B testing advisor	Closed	0	1
58476	Add operational-value grader for architecture-guardian workflow	Closed	0	1
55445	Harden operational-value grader execution, configurable evaluator timeouts	Merged	4	9
54803	Introduce typed enums for workflow fixed-value fields	Merged	4	10
55829	Rename aw-value skill to operational-value-designer	Merged	1	3

Key Findings

1. The operational-value grader initiative is systematically failing, not just underperforming. 65 PRs at 9.2% merge (vs. 77.1% overall) is not noise — it's the same pattern seen across at least the last 6 days of this analysis, meaning the batch task template itself, not any individual PR, is the problem.
2. Closed PRs in the outlier cluster received essentially zero human engagement. Of the 59 closed/unmerged PRs in Cluster 1, 100% had zero code reviews and 92% (54/59) had zero comments at all — they were not rejected on technical merit, they were never reviewed.
3. A third of the outlier cluster is self-admittedly incomplete work. 33 of 65 PR titles explicitly say "blocked," "[WIP]," or reference "no defensible [direct] metric" — the agent itself concluded it couldn't produce a usable grader for that workflow, and the PR exists only to record that finding.
4. Large, heavily-reviewed refactors (Cluster 2) succeed at the highest rate (87.5%), suggesting that investment in review cycles (avg 5 reviews/PR) — not smaller diffs — is what correlates with merging, contrary to the intuition that big PRs are riskier.

Lowest-Merge Cluster Root Cause

• Cluster: Cluster 1 — "operational-value grader" batch tasks
• Merge Rate Gap: 9.2% vs. 77.1% overall (gap: 67.9 points)
• Primary Blocker: Scope/complexity mismatch combined with an unreviewed backlog — the task template asks the agent to design a "defensible direct metric" grader for every existing daily workflow, one PR each; a large share of attempts are structurally infeasible (agent self-reports "blocked"), and even the completed attempts pile up in a batch_review queue that isn't being worked down.
• Confidence: Medium — no CI/check-run data was available in the fetched PR records, so this conclusion is based on review/comment engagement and title-text evidence rather than confirmed CI logs.
• Evidence PRs: Add operational-value grading for Duplicate Code Detector #58452, Add operational-value grader for Issue Triage Agent #58453, Add operational-value grader for Daily Documentation Healer #58454, Harden operational-value grader execution and make evaluator timeouts configurable #55445 (merged comparison), Introduce typed enums for workflow fixed-value fields #54803 (merged comparison)

Blocker Category	Count	Share	Notes
Scope/complexity mismatch (self-reported blocked / WIP / "no defensible metric")	33	56%	Title explicitly signals the agent could not complete the task as specified
Other/unknown (unreviewed backlog — 0 reviews, 0 comments, closed without engagement)	26	44%	Structurally complete "Add grader" PRs auto-tagged batch_review by the triage bot, then closed with no human review ever recorded
CI/test failure	0	0%	No CI/check-run data available in this run to confirm or rule out
Review friction (explicit human pushback)	0	0%	None of the 59 closed PRs in this cluster received a single human review

Recommendations

1. Pause or redesign the operational-value grader batch initiative. At 9.2% merge across 65 PRs it is consuming agent runs for near-zero output; either narrow it to workflows where a defensible metric is plausible, or convert it into a single upfront feasibility-scoring pass before generating any PRs.
2. Add a pre-flight feasibility check to the grader task prompt. Since a third of attempts self-report "blocked / no defensible metric," have the agent assess feasibility first and skip PR creation entirely for infeasible workflows, rather than opening a PR just to record the blocker.
3. Clear or triage the batch_review backlog. 26 structurally-complete grader PRs were closed with zero reviews — either assign a reviewer explicitly for this batch or stop generating more PRs than can realistically be reviewed.
4. Continue investing review cycles in large refactor PRs (Cluster 2). These have the best merge rate at 87.5% despite being the biggest diffs; the correlation with review count (not diff size) suggests review bandwidth, not risk aversion, is the lever that matters here.
5. Fix the prompt-cleaning regex for the recurring "PR Sous Chef" bot footer so future clustering runs separate genuine task themes from bot-footer noise in Cluster 2.

---

Generated by Prompt Clustering Analysis (Run: 34108014399)

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• api.anthropic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.anthropic.com"

See Network Configuration for more information.

Generated by 📊 Copilot Agent Prompt Clustering Analysis · claude · agent · 170.9 AIC · ⌖ 11.9 AIC · ⊞ 16.2K · ◷

• expires on Sep 8, 2026, 2:18 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #59421.