[prompt-analysis] Copilot PR Prompt Analysis - 2026-08-04

[github-actions[bot]]

🤖 Copilot PR Prompt Pattern Analysis - 2026-08-04

Summary

Analysis Period: Last 30 days
Total PRs: 1000 | Merged: 751 (75.1%) | Closed: 240 (24.0%) | Open: 9 (0.9%)

Note: dataset appears capped at 1000 PRs (fetch limit), so figures represent the most recent 1000 copilot/* PRs rather than a strict 30-day window.

Prompt Categories and Success Rates

Category	Total	Merged	Success Rate
Bug Fix	466	337	72.3%
Feature Addition	247	189	76.5%
Refactoring	23	15	65.2%
Documentation	52	37	71.2%
Testing	68	63	92.6%
Other/Uncategorized	143	109	76.2%

65 PRs had placeholder/boilerplate bodies ("Thanks for asking me to work on this...") with no substantive prompt content captured yet; 1 PR had an empty body.

Prompt Analysis

✅ Successful Prompt Patterns

Common characteristics in merged PRs:

• Average prompt length: ~116 words (slightly longer than closed PRs)
• Top keywords: workflow, github, test, fixes, added, agent, changes, safe, path, error, output
• Testing-related prompts have by far the highest success rate (92.6%)

Example successful prompts:

1. PR Fix CLI/docs consistency: document --shellcheck and --version, normalize update flag help #50280: "The CLI consistency audit found gh aw compile --shellcheck and the global --version flag missing from the docs..." → Merged
2. PR Pin poutine Docker image to the digest-pinned constant #50279: "Sighthound flagged five exec.Command sites in pkg/cli as command injection. All five pass argument slices (no shell)..." → Merged
3. PR Remove agentfinder mentions from skills.md #50278: ".github/aw/skills.md referenced a "GitHub Agent Finder" discovery service at agentfinder.github.com, which does not exist..." → Merged

These share a pattern: a specific tool/audit finding is cited first (what broke, what flagged it), followed by a concrete fix description.

❌ Unsuccessful Prompt Patterns

Common characteristics in closed PRs:

• Average prompt length: ~109 words (similar to merged, so length alone isn't the differentiator)
• Top keywords: workflow, github, fixes, copilot, agent, model, path, engine, version, explicit
• More checklist-style prompts (- [x] Investigate... - [ ] ...) appear disproportionately among closed PRs

Example unsuccessful prompts:

1. PR Harden dev-hawk workflow_run path with explicit actor authorization gating #50295: "dev-hawk.lock.yml was flagged by Runner-Guard (RGS-004) because a workflow_run-triggered path could perform privileged actions..." → Closed
2. PR [WIP] Fix configuration of goose and disable MCP support #50281: "- [x] Investigate failing Smoke Goose workflow logs - [x] Identify Goose engine configuration..." → Closed
3. PR Align agent sandbox Go toolchain with go.mod by enabling auto toolchain resolution #50260: "The Go Logger Enhancement workflow is failing because the sandbox runs Go 1.25.12 while this repo now requires go >= 1.26.5..." → Closed

Key Insights

• Testing-focused prompts dominate: prompts framed around "add test"/"test coverage" merge at 92.6%, ~17-20 points above other categories — likely because scope is narrow and verification is objective (tests pass/fail).
• Prompt length is not the discriminator: merged (116 words) vs closed (109 words) are nearly identical, so success correlates more with specificity of root cause than raw verbosity.
• Refactoring is riskiest: lowest success rate (65.2%) and smallest sample (23), consistent with broader/ambiguous scope being harder for the agent to land cleanly.
• Environment/dependency-version prompts underperform: several closed PRs stem from cross-environment mismatches (e.g., sandbox Go version vs repo requirement) that the agent can't reliably resolve without human intervention.

Recommendations

1. DO: Frame prompts around a specific finding/tool flag with the exact file, symbol, or check that triggered it (pattern seen in top merged PRs).
2. DO: Prefer narrowly-scoped testing/coverage tasks when possible — they have the highest merge rate.
3. AVOID: Multi-step checklist prompts spanning investigation + fix + follow-up in one PR; these correlate with closures, likely due to scope creep or incomplete follow-through.
4. AVOID: Prompts describing environment/toolchain mismatches (e.g., version pinning issues) without clear actionable resolution steps — these are harder for the agent to resolve unilaterally.

Historical Trends

Date	PRs	Success Rate	Top Category (≥5 samples)
2026-08-04	1000	75.1%	test
2026-08-02	1000	75.7%	feature_addition
2026-08-01	1000	74.6%	testing
2026-07-31	1000	75.3%	testing
2026-07-30	1000	75.7%	testing
2026-07-29	1000	75.1%	testing
2026-07-28	1000	74.6%	feature

Trend: Success rate has held steady in a narrow 74.6–75.7% band over the last week, with testing-oriented prompts consistently ranking as the top or near-top category — reinforcing that this pattern is stable, not a one-off.

---

Generated by Copilot PR Prompt Analysis (Run: 30950241226)

Generated by 🔍 Copilot PR Prompt Pattern Analysis · auto · 26.6 AIC · ⌖ 8.22 AIC · ⊞ 10.9K · ◷

• expires on Aug 5, 2026, 1:05 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot PR Prompt Pattern Analysis.

A newer discussion is available at Discussion #50692.