Skip to content

dotnet-test: trim skill descriptions to fit skill-menu under 15K - #816

Merged
Evangelink merged 2 commits into
dotnet:mainfrom
Evangelink:evangelink-fix-dotnet-test-skill-menu-size
Jun 25, 2026
Merged

dotnet-test: trim skill descriptions to fit skill-menu under 15K#816
Evangelink merged 2 commits into
dotnet:mainfrom
Evangelink:evangelink-fix-dotnet-test-skill-menu-size

Conversation

@Evangelink

Copy link
Copy Markdown
Member

Problem

The dotnet-test plugin failed to load with:

❌ [plugin:dotnet-test] Plugin 'dotnet-test' rendered skill-menu size is 15,428 characters — maximum is 15,000.

Fix

Lightly trimmed the longest skill descriptions to bring the rendered skill-menu back under the 15,000-character limit. The changes mostly collapse verbose per-framework parentheticals (e.g. .NET (MSTest/xUnit/NUnit/TUnit), Python (pytest/unittest)....NET, Python, ...) and drop a few redundant clauses. All routing keywords and DO NOT USE FOR redirects are preserved.

Touched 8 descriptions: assertion-quality, code-testing-agent, migrate-mstest-v3-to-v4, migrate-vstest-to-mtp, run-tests, test-anti-patterns, test-gap-analysis, test-tagging.

Aggregate name+description length now lands at ~14.5K, leaving ~500 characters of headroom.

The plugin's rendered skill-menu was 15,428 chars, over the 15,000 limit. Compressed the verbose per-framework parentheticals in the longest polyglot descriptions and dropped a few redundant clauses, preserving all routing keywords. Aggregate name+description length now ~14.5K.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings June 24, 2026 13:39

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR trims several dotnet-test skill frontmatter descriptions to reduce the rendered skill-menu size below the 15,000-character Copilot CLI limit, while aiming to keep skill routing behavior intact.

Changes:

  • Shortened frontmatter descriptions across 8 dotnet-test skills by collapsing per-framework parentheticals and removing some redundant detail.
  • Simplified “polyglot/framework support” listings to reduce aggregate menu size headroom.
  • Tweaked several skill descriptions to be more compact while keeping overall intent and “DO NOT USE” guidance.
Show a summary per file
File Description
plugins/dotnet-test/skills/test-tagging/SKILL.md Shortens description; drops some framework keywords from the frontmatter list
plugins/dotnet-test/skills/test-gap-analysis/SKILL.md Collapses the “Polyglot” framework list to a language-only list
plugins/dotnet-test/skills/test-anti-patterns/SKILL.md Condenses “Polyglot” framework list in the block-scalar description
plugins/dotnet-test/skills/run-tests/SKILL.md Removes some example parentheticals in the dotnet test description
plugins/dotnet-test/skills/migrate-vstest-to-mtp/SKILL.md Trims xUnit.net v3 filter flag names from the frontmatter description line
plugins/dotnet-test/skills/migrate-mstest-v3-to-v4/SKILL.md Removes a couple of MSTest v4-specific keywords from the frontmatter description
plugins/dotnet-test/skills/code-testing-agent/SKILL.md Reflows/condenses test generation description wording
plugins/dotnet-test/skills/assertion-quality/SKILL.md Collapses the “Polyglot” framework list to a language-only list

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 8/8 changed files
  • Comments generated: 3

Comment thread plugins/dotnet-test/skills/test-tagging/SKILL.md Outdated
Comment thread plugins/dotnet-test/skills/migrate-mstest-v3-to-v4/SKILL.md
Comment thread plugins/dotnet-test/skills/migrate-vstest-to-mtp/SKILL.md Outdated
@github-actions github-actions Bot added the waiting-on-author PR state label label Jun 24, 2026
@github-actions

Copy link
Copy Markdown
Contributor

👋 @Evangelink — this PR has 3 unresolved review thread(s),merge conflict. When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the no-stale label to silence further pings.)

@Evangelink

Copy link
Copy Markdown
Member Author

@copilot address review comments

…-test-skill-menu-size

# Conflicts:
#	plugins/dotnet-test-migration/skills/migrate-vstest-to-mtp/SKILL.md
#	plugins/dotnet-test/skills/assertion-quality/SKILL.md
#	plugins/dotnet-test/skills/code-testing-agent/SKILL.md
#	plugins/dotnet-test/skills/run-tests/SKILL.md
#	plugins/dotnet-test/skills/test-anti-patterns/SKILL.md
#	plugins/dotnet-test/skills/test-gap-analysis/SKILL.md
@Evangelink

Copy link
Copy Markdown
Member Author

/evaluate

@Evangelink
Evangelink enabled auto-merge (squash) June 25, 2026 07:54
@github-actions

Copy link
Copy Markdown
Contributor

Skill Validation Results

Skill Scenario Quality Skills Loaded Overfit Verdict
migrate-vstest-to-mtp Migrate MSTest project from VSTest to Microsoft.Testing.Platform 3.3/5 → 5.0/5 🟢 ✅ migrate-vstest-to-mtp; tools: skill, report_intent, view ✅ 0.12 [1]
migrate-vstest-to-mtp Migrate NUnit project from VSTest to Microsoft.Testing.Platform 1.0/5 → 2.7/5 🟢 ✅ migrate-vstest-to-mtp; tools: skill, bash, edit, create, glob / ✅ migrate-vstest-to-mtp; tools: skill, bash, edit, create ✅ 0.12
migrate-vstest-to-mtp Migrate xUnit.net v2 project from VSTest to Microsoft.Testing.Platform 1.0/5 → 1.3/5 🟢 ✅ migrate-vstest-to-mtp; tools: skill, read_bash, create, glob / ✅ migrate-vstest-to-mtp; tools: skill, create ✅ 0.12 [2]
migrate-vstest-to-mtp Update Azure DevOps pipeline from VSTest task to MTP 3.0/5 → 4.0/5 🟢 ✅ migrate-vstest-to-mtp; tools: skill / ✅ migrate-vstest-to-mtp; tools: skill, bash ✅ 0.12 [3]
migrate-vstest-to-mtp Migrate MSTest.Sdk project that explicitly uses VSTest 2.7/5 → 3.3/5 🟢 ✅ migrate-vstest-to-mtp; tools: skill / ✅ migrate-vstest-to-mtp; tools: skill, glob, create ✅ 0.12 [4]
migrate-vstest-to-mtp Translate dotnet test VSTest arguments to MTP equivalents 3.7/5 → 5.0/5 🟢 ✅ migrate-vstest-to-mtp; tools: skill, report_intent, view ✅ 0.12
migrate-vstest-to-mtp Handle exit code 8 when migrating from VSTest to MTP 2.7/5 → 5.0/5 🟢 ✅ migrate-vstest-to-mtp; tools: skill, report_intent, view, edit, bash / ✅ migrate-vstest-to-mtp; tools: report_intent, skill ✅ 0.12
migrate-vstest-to-mtp Configure dotnet test MTP mode on .NET 10 SDK 2.0/5 → 5.0/5 🟢 ✅ migrate-vstest-to-mtp; tools: skill ✅ 0.12
migrate-vstest-to-mtp Migrate xUnit.net VSTest filter syntax to MTP 1.7/5 → 4.0/5 🟢 ✅ migrate-vstest-to-mtp; tools: skill ✅ 0.12
migrate-vstest-to-mtp Set OutputType=Exe only for test projects in Directory.Build.props 2.0/5 → 5.0/5 🟢 ✅ migrate-vstest-to-mtp; tools: skill / ✅ migrate-vstest-to-mtp; tools: report_intent, skill ✅ 0.12
migrate-vstest-to-mtp Full VSTest to MTP migration plan for MSTest solution 3.0/5 → 5.0/5 🟢 ✅ migrate-vstest-to-mtp; tools: skill ✅ 0.12

[1] ⚠️ High run-to-run variance (CV=59%) — consider re-running with --runs 5
[2] ⚠️ High run-to-run variance (CV=335%) — consider re-running with --runs 5. (Isolated) Quality improved but weighted score is -5.4% due to: judgment, quality, tokens (64902 → 106321), tool calls (7 → 11), time (75.7s → 92.5s)
[3] ⚠️ High run-to-run variance (CV=145%) — consider re-running with --runs 5
[4] ⚠️ High run-to-run variance (CV=159%) — consider re-running with --runs 5

Model: claude-opus-4.6 | Judge: claude-opus-4.6

🔍 Full Results - additional metrics and failure investigation steps

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

@Evangelink
Evangelink merged commit cbb6121 into dotnet:main Jun 25, 2026
37 checks passed
@Evangelink
Evangelink deleted the evangelink-fix-dotnet-test-skill-menu-size branch June 25, 2026 08:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

waiting-on-author PR state label

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants