[prompt-clustering] Prompt Clustering Analysis - 2026-09-09

[github-actions[bot]]

Summary

Analysis Period: Last 30 days (2026-08-17 to 2026-09-09)
Total Tasks Analyzed: 1000 copilot-authored PRs (808 merged, 186 closed, 6 open)
Clusters Identified: 6
Overall Merge Rate: 80.8%

One cluster stands out sharply: Cluster 3 ("operational-value grading") — 61 PRs at a 3.3% merge rate, 77.5 points below the overall baseline. This is a stable pattern, not a one-off: the same theme showed up in the prior run (2026-09-08) at 63 PRs / 11% merge rate.

Full Analysis Report

General Insights

• Most Common Task Type: Cluster 0 — general gh-aw workflow/tooling engineering (615 PRs, 61% of all tasks), merging at 84%, roughly in line with baseline.
• Highest Success Rate: Cluster 4 — CI/Actions job-failure fixes (23 PRs, 100% merged).
• Most Complex Tasks: Cluster 2 — firewall/network policy work (67 PRs), highest average footprint at ~56 files / ~779 additions per PR, and the most review activity (5.9 reviews/PR average).
• Outlier: Cluster 3 — "operational-value grading" batch, 61 PRs, 3.3% merge rate (vs. 80.8% overall).

Turn-count / duration metrics from workflow logs (gh-aw logs) were not available in this run's environment (gh CLI unauthenticated), so complexity is proxied here using comments, reviews, changed files, and line additions instead.

Cluster Analysis

Cluster 0: General workflow & tooling engineering

• Size: 615 tasks (61.5% of total)
• Merge Rate: 84.2%
• Avg Comments / Reviews: 4.1 / 2.4
• Avg Files Changed: 25.4 (avg +368 additions)
• Top Keywords: github, workflows, safe, daily, test, md, gh, aw
• Characteristics: The catch-all bucket for day-to-day gh-aw feature work — logs tooling, safe-output handling, MCP policy, workflow config.
• Example PRs: Bump bundled MCP gateway to v0.4.20 and refresh pinned workflow image references #59602, Include firewall domain decisions in logs JSON #59575, Generate complete MCP policy and lifetime wiring for dynamic enclaves #59524, Resolve remote workflow names for logs targets #59454, Add oldest-run pruning mode to logs downloads #59453

---

Cluster 1: Sous Chef / triage-driven changes

• Size: 146 tasks (14.6%)
• Merge Rate: 92.5%
• Avg Comments / Reviews: 13.3 / 5.1 (highest comment volume)
• Avg Files Changed: 31.3 (avg +811 additions — largest average diff)
• Top Keywords: sous chef, pr sous, run, retry (bot-footer terms from the "PR Sous Chef" triage tool leaked into TF-IDF here — a known data-cleaning limitation, see note below)
• Characteristics: Larger, heavily-reviewed changes that went through the Sous Chef/triage-comment pipeline before merge.
• Example PRs: Retry transient slash-command provenance checks #59313, Cache GitHub Actions job metadata in logs output #59039, Register replace_label handler in safe-output collect job dispatch map #58917, Add deterministic body footer templates to safe outputs #58841, Extend aw.yml config with repository labels #58796

---

Cluster 2: Firewall & network policy

• Size: 67 tasks (6.7%)
• Merge Rate: 94.0% (highest among substantial clusters)
• Avg Comments / Reviews: 11.8 / 5.9
• Avg Files Changed: 56.2 (avg +779 additions)
• Top Keywords: firewall, network, domain, summary
• Characteristics: Highest per-PR review burden but also highest quality bar / merge rate — sustained review pays off here.
• Example PRs: Expose identifiable MCP tool calls in logs JSON #59579, Add cached JSON reuse to the logs command #59572, Fix logs workflow resolution in large repositories #59502, Prune logs cache during concurrent downloads #59338, Improve logs cache size diagnostics #59336

---

Cluster 3: Operational-value grading (outlier)

• Size: 61 tasks (6.1%)
• Merge Rate: 3.3% ⚠️
• Avg Comments / Reviews: 0.2 / 0.03 (almost no engagement)
• Avg Files Changed: 2.3 (avg +217 additions)
• Top Keywords: operational, operational value, study, evidence, metric
• Characteristics: A bulk, one-PR-per-workflow initiative to add a deterministic "operational-value grader" to each daily workflow, following .github/skills/operational-value-designer/SKILL.md. See root-cause analysis below.
• Example PRs: Record blocked operational-value study for daily-choice-test #58559, Add operational-value grader for daily-caveman-optimizer #58557, Add operational-value grader for cache strategy remediation #58555, Record blocked operational-value study for daily-byok-ollama-test #58554, Record blocked operational-value study for AWF spec surfacing #58553 (all closed)

---

Cluster 4: CI / Actions job failures

• Size: 23 tasks (2.3%)
• Merge Rate: 100%
• Avg Comments / Reviews: 0.9 / 1.0
• Avg Files Changed: 2.2 (avg +22 additions) — smallest, most surgical fixes
• Top Keywords: job, actions, fix, failing, check, root
• Characteristics: Small, well-scoped CI fix-up PRs — narrowest diff, but merges every time.
• Example PRs (from cluster data): job/actions-failure fixes, root-cause style patches.

---

Cluster 5: Model/engine & agent runtime

• Size: 88 tasks (8.8%)
• Merge Rate: 76.1% (below baseline, but not an outlier by the size/gap threshold)
• Avg Comments / Reviews: 4.5 / 2.0
• Avg Files Changed: 33.4 (avg +693 additions)
• Top Keywords: model, codex, copilot, engine, claude, agent, runtime
• Characteristics: Cross-engine work (Copilot/Codex/Claude runtime support) — moderately complex, moderately risky.
• Example PRs: drawn from engine/runtime configuration and model-support changes.

Success Rate by Cluster

Cluster	Theme	Tasks	Merge Rate	Gap vs. Overall	Avg Files	Avg Additions
4	CI/Actions fixes	23	100.0%	+19.2 pts	2.2	22
2	Firewall/network	67	94.0%	+13.2 pts	56.2	779
1	Sous Chef/triage	146	92.5%	+11.7 pts	31.3	811
0	General workflow eng.	615	84.2%	+3.4 pts	25.4	368
5	Model/engine runtime	88	76.1%	−4.7 pts	33.4	693
3	Operational-value grading	61	3.3%	−77.5 pts	2.3	217

(Gap = overall rate 80.8% minus cluster rate; positive = better than baseline, negative = worse.)

Key Findings

1. The "operational-value grading" batch is the dominant blocker signal this period. 61 PRs (6.1% of all copilot activity) target a single initiative — adding deterministic value-graders to individual daily workflows — and only 2 merged. Both merges were meta-level changes (a skill rename and a design-doc PR), not per-workflow grader instances. Every single per-workflow grader attempt was closed.
2. Small, narrowly-scoped fixes merge reliably; broad batch-generated proposals don't. Cluster 4 (CI fixes, avg 2.2 files) merges 100% of the time, while Cluster 3 (also small, avg 2.3 files) merges 3.3% of the time — file/diff size alone doesn't predict merge outcome. The difference is task framing: reactive bug fixes vs. speculative batch proposals.
3. Review engagement correlates with merge success outside the outlier cluster. Clusters 1 and 2, which see the heaviest comment/review activity (13.3 and 11.8 comments/PR respectively), also merge at 92–94%. Cluster 3 sees essentially zero engagement (0.2 comments/PR) and merges at 3.3% — the absence of review, not negative review, is the story.
4. This pattern is persistent, not transient. Comparing to the cached 2026-09-08 run: the same "operational-value" cluster showed 63 PRs at an 11% merge rate then; it's now 61 PRs at 3.3%. The initiative keeps generating largely-unmerged PRs day over day.

Lowest-Merge Cluster Root Cause

• Cluster: 3 — "operational-value grading" batch initiative
• Merge Rate Gap: 3.3% vs. 80.8% overall (gap: 77.5 points)
• Primary Blocker: Scope/complexity mismatch (bulk task generation, not CI or review rejection)
• Confidence: High
• Evidence PRs: Record blocked operational-value study for daily-choice-test #58559, Add operational-value grader for daily-caveman-optimizer #58557, Add operational-value grader for cache strategy remediation #58555, operational-value-study: block copilot-opt (no defensible direct metric) #58535, Add operational-value grader for architecture-guardian workflow #58476, Add operational-value grading for A/B testing advisor #58464

Classification of all 61 PRs by how they failed to merge (based on comments, reviews, reviewDecision, and body content):

Blocker Category	Count	Share	Notes
Self-reported "blocked / no defensible metric"	27	44%	PR body explicitly concludes the workflow has no valid direct operational metric (e.g. "no defensible direct metric", "not a valid substitute") and is closed by design.
Closed with zero review engagement	27	44%	Real "Add operational-value grader for X" implementation attempts that received no comments and no reviews before being closed — apparently swept up/abandoned alongside the self-reported-blocked PRs rather than individually triaged.
Bot triage comment only, still closed	5	8%	Received an automated PR-triage-agent comment (tagged Batch: operational-value-grading) but no human review, and were closed anyway.
Merged	2	3%	Both are meta/infrastructure changes (renaming the operational-value-designer skill, documenting workflow design) — not per-workflow grader instances.

There is no evidence of CI failures or review rejections driving this cluster's outcome: reviewDecision is empty for all 61 PRs, and only 2 ever received a formal review (both merged). The dominant pattern is a large batch of narrow, single-workflow PRs generated by an "operational-value-designer" initiative, where ~89% (54/61) receive no meaningful human engagement at all before being closed — whether the PR self-reports a negative/blocked finding or contains a real implementation attempt.

Recommendations

1. Consolidate the operational-value-grading initiative. Rather than opening one narrowly-scoped PR per daily workflow, batch related graders into fewer, larger PRs (e.g., per workflow category) so reviewers engage once instead of the current 1-PR-per-workflow pattern that's going unreviewed.
2. Add an explicit review/triage gate before opening these PRs, or route "blocked" outcomes (44% of the cluster) to an issue/log rather than a PR — a PR that self-concludes "no defensible metric" doesn't need a pull request lifecycle at all.
3. Keep investing in the CI-fix and firewall/network patterns (Clusters 4 and 2) — both show strong merge rates with proportionate review effort; no changes needed there.
4. Cluster 3 mitigation, specifically: (a) stop auto-generating individual grader PRs for workflows where an adoption-time scan can already tell there's no durable evidence signal, (b) merge the 27 zero-engagement "Add grader" PRs in bulk review sessions before they go stale, (c) track this initiative's merge rate week over week — it degraded from 11% to 3.3% between the last two runs, which suggests it's getting worse, not better.

---

Generated by Prompt Clustering Analysis (Run: §34336437769)

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• api.anthropic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.anthropic.com"

See Network Configuration for more information.

Generated by 📊 Copilot Agent Prompt Clustering Analysis · claude · agent · 200.7 AIC · ⌖ 11.5 AIC · ⊞ 16.2K · ◷

• expires on Sep 10, 2026, 2:12 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #59939.