[prompt-analysis] Copilot PR Prompt Analysis - 2026-08-13

[github-actions[bot]]

🤖 Copilot PR Prompt Pattern Analysis - 2026-08-13

Summary

Analysis Period: Last 30 days (2026-07-28 to 2026-08-13)
Total PRs: 1000 | Merged: 741 (74.1%) | Closed: 243 (24.3%) | Open: 16 (1.6%)

Prompt Categories and Success Rates

Category	Total	Merged	Success Rate
Feature Addition	334	264	79.0%
Bug Fix	281	200	71.2%
Testing	141	107	75.9%
Other	115	89	77.4%
Documentation	100	72	72.0%
Refactoring	13	9	69.2%

Categorized by keyword matching on the PR body (task prompt); "Other" covers prompts without a clear dominant action verb. Rates computed against completed PRs (merged + closed), excluding still-open PRs.

Prompt Analysis

✅ Successful Prompt Patterns

Common characteristics in merged PRs:

• Average prompt length: ~150 words
• Most common (non-generic) keywords: workflow, engine, test, actions, runs, agent, pkg
• Prompts frequently name concrete files, functions, or config keys (e.g., DefaultCopilotVersion, install-*, specific .md/.go paths)

Example successful prompts:

1. PR Route PR code-quality reviews through the Copilot gateway #52377 (75w): "The Pi-based PR Code Quality Reviewer fell back to its native Copilot provider, attempting the blocked api.individual.githubcopilot.com en..." → Merged
2. PR docs: note that non-default labels must exist before running the triage workflow #52376 (89w): "The AI issue triage guide lists needs-info and priority/p0–p2 in add-labels.allowed, but those are not GitHub default labels..." → Merged
3. PR Fix PR context renderer test assertion #52201 (53w): "The PR checkout integration test expected the obsolete shell-based prompt inclusion and failed after prompt files moved to the structured re..." → Merged

❌ Unsuccessful Prompt Patterns

Common characteristics in closed PRs:

• Average prompt length: ~171 words (excluding the security-batch outlier below), notably longer than merged prompts
• A distinct cluster of 48 automated security/CVE-remediation PRs (containing CVE, SecurityTracker, or REDACTED boilerplate) had only a 43.8% merge rate, far below the 76.9% rate for all other PRs — this batch skews the overall "closed" statistics
• Non-security closed prompts often describe compliance/lint-driven bulk changes across many files ("five scoped files", "OTel Level 1 compliance") or reactive CI-failure fixes with less specific remediation guidance

Example unsuccessful prompts:

1. PR Wire intent policy enforcement and OTel Level 1 contract coverage #52329 (184w): "Daily SPDD review found intent policies were compiled but advisory-only, and OTel Level 1 compliance still had unimplemented T-OT-008–011 co..." → Closed
2. PR Improve actionable CLI error messages #52175 (118w): "Updates low-compliance CLI validation errors in five scoped files to consistently state the invalid condition, expected input, and a concret..." → Closed
3. PR Raise error-message compliance in dispatch validation and CLI evaluation/logging paths #52173 (247w): "This PR addresses low compliance reported by lint-error-messages in five targeted files by normalizing validation/operator-facing errors t..." → Closed

Key Insights

• Security/CVE-batch prompts underperform sharply: the 48 automated CVE-remediation PRs merged at only 43.8%, vs 76.9% for every other PR in the dataset — nearly a 2x gap. These prompts tend to be templated advisory text rather than task-specific instructions.
• Feature-addition prompts have the highest success rate (79.0%) among the five main categories, followed closely by "Other" (77.4%) and Testing (75.9%). Refactoring prompts (69.2%) and Documentation (72.0%) trail slightly, though sample sizes for Refactoring (n=13) are small.
• Conciseness and specificity correlate with success: merged prompts average ~150 words vs ~171 for closed (non-security) prompts, and the most-cited merged examples name specific files, functions, or config keys rather than describing broad multi-file compliance sweeps.

Recommendations

Based on today's analysis:

1. DO: Reference specific files, functions, or config keys in the prompt (e.g., "the add-labels.allowed list in X.md") — this pattern appears repeatedly among merged PRs.
2. DO: Keep prompts concise and scoped to a single well-defined problem; the merged-PR average (~150 words) is shorter than the closed-PR average.
3. AVOID: Templated, boilerplate-heavy prompts (e.g., bulk CVE/security advisories) without task-specific remediation detail — this category shows the lowest success rate (43.8%) in the dataset.

Historical Trends

No prior historical data was found in cache memory — this is the first recorded analysis run. Today's results have been saved as the baseline for future day-over-day comparison.

Date	PRs	Success Rate	Top Category
2026-08-13	1000	74.1%	Feature Addition (79.0%)

Trend: Baseline established; revisit in future runs to compare category success rates and identify shifts (e.g., whether the security-batch merge rate improves).

---

Generated by Copilot PR Prompt Analysis (Run: 31743073820)

Generated by 🔍 Copilot PR Prompt Pattern Analysis · auto · 24.3 AIC · ⌖ 5.23 AIC · ⊞ 10.6K · ◷

• expires on Aug 14, 2026, 12:59 PM UTC-08:00