[prompt-clustering] Prompt Clustering Analysis - 2026-09-24

[github-actions[bot]]

Summary

Analysis Period: Last 30 days (2026-08-25 to 2026-09-24)
Total Tasks Analyzed: 722 copilot-agent PRs (usable prompt text for all 722)
Clusters Identified: 7 (k chosen via elbow method on TF-IDF features)
Overall Merge Rate: 80.7% (583 merged / 135 closed / 4 open)

Data limitation: workflow run logs (gh-aw logs) were not available in this run's cache, so turn counts, duration, and cost metrics could not be joined to PRs. All findings below are based on PR metadata, bodies, comments, and reviews only.

Full Analysis Report

General Insights

• Most Common Task Type: Cluster 3 — general workflow bug fixes & feature additions (311 PRs, 43.1% of all tasks)
• Highest Success Rate: Cluster 6 — core engineering / internals hardening (93.5% merged)
• Most Complex Tasks: Cluster 6 also has the highest average review/comment load (13.6 comments, 5.7 reviews per PR) and largest average diff (49.8 files changed), consistent with harness-internals work needing more scrutiny
• Outliers: 1 cluster (Cluster 0) is a severe outlier — 76.0 points below the overall merge rate, and it is a recurring pattern also flagged in the 2026-09-15 analysis (then: 60 PRs, 1.7% merge rate)

Cluster Analysis

Cluster 3: General Workflow Fixes & Feature Additions

• Size: 311 tasks (43.1% of total)
• Merge Rate: 86.8%
• Avg Reviews / Comments: 3.1 / 5.6
• Top Keywords: workflow, fixes, workflows, added, github, changes, existing, add
• Characteristics: The bulk of day-to-day maintenance — targeted fixes and small feature additions across the many .github/workflows/*.md agentic workflows in this repo. Healthy merge rate, typical of well-scoped, incremental changes.
• Example PRs: Ensure a blank line follows closing </details> tags in generated markdown #63069, Make dev-mode actions folder checkout shallow #63035, Harden docker-sbx install: remove curl|sudo sh root install #62991, Add engine-independent tool profile configuration #62951, Report gateway steering events in audit output #62943

---

Cluster 6: Core Engineering / Internals Hardening

• Size: 154 tasks (21.3% of total)
• Merge Rate: 93.5% (highest of all clusters)
• Avg Reviews / Comments: 5.7 / 13.6
• Top Keywords: (noisy — footer/branding terms from the "PR Sous Chef" bot dominate TF-IDF); underlying titles show harness internals: circuit breakers, permission hardening, type-safety conventions, compliance coverage
• Characteristics: Larger, more scrutinized changes to Go/TypeScript source (compiler, constants, scopes) rather than workflow markdown. Despite being the most reviewed cluster, it has the best merge rate — reviewers engage heavily but changes ultimately land.
• Example PRs: Codex harness: add working-set rebuild circuit breaker to stop runaway context loops #55562, Write MCP CLI wrapper scripts with owner-only permissions (0o700) instead of 0o755 #55587, Expose Copilot web_fetch when tools.web-fetch is enabled #55696, Extend semantic-type convention: add ArtifactName/Filename/FilePath to pkg/constants #55787, Give GitHubReposScope a concrete type #55793

---

Cluster 4: Copilot/Codex Model & Engine Configuration

• Size: 75 tasks (10.4% of total)
• Merge Rate: 85.3%
• Avg Reviews / Comments: 1.9 / 3.8
• Top Keywords: model, copilot, codex, workflow, engine, configuration, cli, runtime
• Characteristics: Model/engine pinning, port fixes, sandbox allow-lists, model picker tuning. Lower review overhead than other clusters — often narrow, mechanical fixes.
• Example PRs: Skip duplicate generated issues after unmerged Copilot PRs #55627, Use mai-code-1-flash-picker model in pr-triage-agent #55643, Fix swapped openai/anthropic ports in AWF api-proxy gateway constants #55693, Fix AI Moderator Codex crash: seed CODEX_HOME into cloud-hypervisor filesystem allow-list #55711, Split add command implementation into focused modules #55785

---

Cluster 0: Operational-Value Grading Initiative — OUTLIER

• Size: 63 tasks (8.7% of total)
• Merge Rate: 4.8% (3 merged / 60 closed)
• Avg Reviews / Comments: 0.2 / 0.5 (near-zero human interaction)
• Top Keywords: operational, operational value, evidence, value, study, adoption, grader, metric
• Characteristics: Auto-generated one-workflow-at-a-time PRs titled "Add operational-value grader for <workflow>" or "Record blocked operational-value study for <workflow>". See root-cause section below — this is a recurring, previously-flagged pattern (also an outlier in the 2026-09-15 report).
• Example PRs: Fix operational-value grader created-at acquisition and preserve grader result contract in gh aw logs #60635, Record blocked operational-value study for daily-choice-test #58559, Add operational-value grader for daily-caveman-optimizer #58557, Add operational-value grader for cache strategy remediation #58555, Record blocked operational-value study for daily-byok-ollama-test #58554

---

Cluster 5: Safe-Outputs Pipeline & Workflow Features

• Size: 63 tasks (8.7% of total)
• Merge Rate: 85.7%
• Avg Reviews / Comments: 2.9 / 5.5
• Top Keywords: safe, output, safe output, outputs, safe outputs, workflow, issue, agent
• Characteristics: Feature work on the safe-outputs MCP server and its consumers (diagnostics, ignore-missing-branch handling, steering-issue refactors). Healthy, typical merge rate.
• Example PRs: Update Archivx to publish diagrams through pull requests #55563, Honor ignore-missing-branch-failure in the safe-outputs MCP server when a PR branch is deleted mid-run #55571, Refactor pull request steering to use issues #55572, [WIP] Fix daily code debt cleanup issue with no safe outputs #55624, Add PDF-aware preliminary investigation to arXiv research #55644

---

Cluster 2: Infra & Tooling Utilities

• Size: 34 tasks (4.7% of total)
• Merge Rate: 85.3%
• Avg Reviews / Comments: 4.1 / 7.5
• Top Keywords: target, cache, logs, shared, api, run, runs, preserve
• Characteristics: Cross-cutting infra changes — govulncheck CI gate, pkg/console wasm/native parity, rate-limit reporting, target-only checkout. Smallest average diff size among healthy clusters but highest review/comment density, suggesting careful infra review.
• Example PRs: Add govulncheck gate to Go CI #55889, Fix pkg/console native/wasm API parity #56089, Add issue-only permission controls to remove-labels #56270, Support target-only checkout for MultiRepoOps sidecar workflows #56504, Disable OTLP export when authorization secrets are empty #58312

---

Cluster 1: CI-Failure Auto-Fix Tasks

• Size: 22 tasks (3.0% of total)
• Merge Rate: 86.4%
• Avg Reviews / Comments: 1.2 / 2.8
• Top Keywords: actions, job, fix, failing, github, check, logs, root
• Characteristics: Nearly every title is literally "[WIP] Fix failing GitHub Actions job <name>" — the automated CI-failure-response workflow. Small cluster but reliably merges, indicating this auto-triage flow is working as intended.
• Example PRs: [WIP] Fix failing GitHub Actions job Integration: Workflow Features #55638, [WIP] Fix failing lint-go-golangci GitHub Actions job #55648, [WIP] Fix failing GitHub Actions job 'Unit tests (A-C)' #56345, [WIP] Fix failing GitHub Actions job 'Unit tests (S-Z and other)' #56346, [WIP] Fix failing GitHub Actions job Integration: Workflow Misc Part 2 #56430

Success Rate by Cluster

Cluster	Theme	Tasks	Share	Merge Rate	Avg Reviews	Avg Comments	Top Keywords
3	Workflow fixes & features	311	43.1%	86.8%	3.1	5.6	workflow, fixes, added
6	Core engineering hardening	154	21.3%	93.5%	5.7	13.6	(internals: types, permissions, circuit breakers)
4	Model/engine configuration	75	10.4%	85.3%	1.9	3.8	model, copilot, codex, engine
0	Operational-value grading (outlier)	63	8.7%	4.8%	0.2	0.5	operational value, study, grader
5	Safe-outputs pipeline	63	8.7%	85.7%	2.9	5.5	safe output(s), workflow, issue
2	Infra & tooling utilities	34	4.7%	85.3%	4.1	7.5	target, cache, logs, api
1	CI-failure auto-fix	22	3.0%	86.4%	1.2	2.8	actions job, fix, failing

Full Data Table (sample of 20 most recent PRs per cluster, largest clusters abbreviated)

PR #	Title	Cluster	Outcome
63069	Ensure a blank line follows closing </details> tags in generated markdown	3	Closed
63048	Improve detection and diagnostics for missing Copilot custom agent	6	Merged
62957	Support native web-search on the Copilot engine	4	Merged
60635	Fix operational-value grader created-at acquisition and preserve grader result contract in gh aw logs	0	Closed
63045	Fix ARC/DinD agent temporary directory staging	5	Open
62492	Remove failing shared agentic logs cache restore	2	Merged
62716	[WIP] Fix failing GitHub Actions job Integration: CLI Logs Live Cache	1	Merged

Full per-PR cluster assignments (722 rows) were computed but are omitted here for length; ask if a CSV export is wanted.

Key Findings

1. Workflow maintenance dominates the workload: Nearly 65% of all copilot-agent tasks (clusters 3 + 6 combined) are either day-to-day workflow fixes or core engineering hardening, and both have merge rates at or above the 80.7% baseline — the core agentic-workflow authoring loop is healthy.
2. CI-failure auto-fix and infra work are reliable niches: Clusters 1 and 2, though small (3–5% of volume each), merge at 85%+ with modest review overhead — these narrowly-scoped automated triggers are producing consistently mergeable output.
3. One initiative accounts for nearly all "wasted" agent runs: Cluster 0 (operational-value grading) is 8.7% of all copilot PR volume in the last 30 days but only 4.8% ever merge — meaning roughly 60 agent runs per month are being generated and then discarded with almost no human engagement.

Lowest-Merge Cluster Root Cause

• Cluster: 0 — "Operational-value grading initiative" (add/record grader or study PRs, one per target workflow)
• Merge Rate Gap: 4.8% vs 80.7% overall (gap: 76.0 points)
• Primary Blocker: Bulk administrative closure, not organic per-PR rejection
• Confidence: High
• Evidence PRs: Add operational-value grading for Duplicate Code Detector #58452, Add operational-value grader for Issue Triage Agent #58453, Add operational-value grader for Daily Documentation Healer #58454, Add operational-value grading for A/B testing advisor #58464, Add ACE Editor operational-value grader #58465, Document blocked Craft operational-value grader design #58542, Record blocked operational-value design for setup security audit #58543, Add operational-value grader for daily-ambient-context-optimizer #58546, Record blocked operational-value study for architecture diagram workflow #58547, Record blocked operational-value design for daily arXiv researcher #58548

Supporting evidence:

• Of the 60 closed PRs in this cluster, 54 (90%) have zero comments and zero reviews — no human ever engaged with them before closure.
• 59 of the 60 closures happened on the same single day (2026-09-04), with a median time-to-close of 18 minutes (min 3 minutes, max 3.4 hours) — far too fast for organic code review, and consistent with a scripted bulk-close rather than individual triage.
• The few PRs with any comment carry an automated PR Triage Agent tag: Batch: operational-value-grading, Action: batch_review — confirming these were pre-flagged as a single batch for disposition together, not reviewed individually on merit.
• Even the one PR with a substantive bot assessment (Add operational-value grading for A/B testing advisor #58464, "A/B testing advisor" grader) was rated positively ("well-scoped... implementation... as requested") and still closed — reinforcing that closure was tied to the batch/initiative, not per-PR quality.
• This is a recurring pattern: the 2026-09-15 clustering run flagged the same theme (60 PRs, 1.7% merge rate, top keywords "operational, operational value, value, study") — this is not a new or transient issue, it has persisted for at least the last two analysis windows.
• Context clue: PR Rename aw-value skill to operational-value-designer and update grader guidance references #55829 ("Rename aw-value skill to operational-value-designer...") did merge — suggesting the per-workflow grader-PR approach was superseded by a redesigned, consolidated skill, and the individually-generated PRs were pruned as no-longer-needed rather than rejected for defects.

Blocker Category	Count	Share	Notes
CI/test failure	0	0%	No CI-failure comments observed; closures happened too fast (median 18 min) for CI to be the driver
Review friction	1	2%	#58464 received a substantive (positive) bot review yet was still closed
Scope/complexity mismatch	0	0%	No evidence of scope pushback in comments
Other/unknown (bulk administrative closure)	59	98%	Same-day, near-simultaneous closures with no per-PR review; consistent with a superseded initiative being pruned in bulk

Recommendations

1. Stop generating one-PR-per-workflow "operational-value grader" tasks until the underlying operational-value-designer skill redesign (per Rename aw-value skill to operational-value-designer and update grader guidance references #55829) is finalized — this is currently the single largest source of wasted agent runs (~60/month, ~8.7% of all copilot volume) with a 4.8% success rate.
2. Add a pre-flight check to the operational-value workflow that verifies whether the redesigned skill already covers a target before generating a new grader PR, so agent runs aren't spent on work likely to be superseded.
3. Keep investing in the CI-failure auto-fix (Cluster 1) and infra-utility (Cluster 2) niches — both are small but consistently high-merge, low-friction categories; consider expanding their scope since the pattern is working.
4. Operational-value grading cluster: given the bulk-closure evidence, the fix is not about prompt quality (these PRs were never reviewed for quality) — it's a workflow-trigger/scheduling fix: pause or gate the task-generation source until the initiative's design has stabilized, then re-enable per-workflow generation against the finalized design.

---

Generated by Prompt Clustering Analysis (Run: 35983267147)

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• api.anthropic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.anthropic.com"

See Network Configuration for more information.

Generated by 📊 Copilot Agent Prompt Clustering Analysis · claude · agent · 127 AIC · ⌖ 10.4 AIC · ⊞ 16.3K · ◷

• expires on Sep 25, 2026, 2:08 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #63411.