Enforce description size limits and add spec conformance checks by danmoseley · Pull Request #238 · dotnet/skills

danmoseley · 2026-03-05T21:42:22Z

Adds description size limit enforcement and agentskills.io spec conformance checks to SkillProfiler. (Supersedes #224, which has been closed.)

Description size limits (from #224, closes #222)

Per-skill limit (1024 chars):

SkillProfiler flags the error in Errors list
ValidateCommand blocks with FailureKind = "spec_conformance_failure" (survives --verdict-warn-only)

Per-plugin aggregate limit (15,000 chars):

ValidateCommand.CheckAggregateDescriptionLimits() groups discovered skills by plugin and checks aggregate size
Runs as part of the existing evaluation.yml pipeline — no separate workflow needed

Current state: all plugins pass (max skill: 769 chars, max plugin: 5,765 chars).

Spec conformance checks (new)

Blocking errors:

Name: max 64 chars, lowercase alphanumeric + hyphens only, no leading/trailing/consecutive hyphens, must match directory name
Body line count: max 500 lines
File reference depth: max 1 level deep from SKILL.md (no parent-directory traversals)

Warnings:

Compatibility: max 500 chars if provided

Other fixes

Fixed $RepoRoot in build.ps1 (was resolving to agentic-workflows/ instead of the repo root after the dotnet-msbuild subfolder reorg)
Replaced DescriptionTooLong bool with a general Errors list on SkillProfile, so all spec conformance violations flow through the same mechanism

Tests: 9 C# unit tests for description limits (per-skill + aggregate); 21 C# unit tests for spec conformance checks.

Copilot

Pull request overview

This PR adds agentskills.io specification conformance checks to the SkillProfiler in the skill validator, building on the description size limit enforcement from PR #224. It introduces name validation rules, compatibility field length checks, body line count warnings, and file reference depth warnings. A new Compatibility field is threaded through the model layer and discovery pipeline, and a CI workflow plus PowerShell validation script enforce description size limits centrally.

Changes:

Adds name, compatibility, body line count, and file reference depth validation to SkillProfiler.AnalyzeSkill(), with description_too_long as a blocking failure in the validate command (even in --verdict-warn-only mode).
Introduces eng/validate-descriptions.ps1 with CI workflow and PowerShell integration tests for per-skill (1024 chars) and per-plugin aggregate (15K chars) description size limits.
Adds Compatibility field to SkillInfo, RawFrontmatter, and SkillDiscovery, plus updates CODEOWNERS to remove nuget-trusted-publishing and @dotnet/appmodel from dotnet-aot-compat.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`eng/skill-validator/src/Services/SkillProfiler.cs`	Adds name validation, description length check, compatibility length check, body line count check, file reference depth check, and two new generated regexes
`eng/skill-validator/src/Models/Models.cs`	Adds optional `Compatibility` field to `SkillInfo` record
`eng/skill-validator/src/Services/EvalSchema.cs`	Adds `Compatibility` property to `RawFrontmatter` for YAML deserialization
`eng/skill-validator/src/Services/SkillDiscovery.cs`	Threads `Compatibility` from parsed frontmatter into `SkillInfo` construction
`eng/skill-validator/src/Commands/ValidateCommand.cs`	Adds `description_too_long` as a non-warnable blocking failure and early-return verdict
`eng/skill-validator/tests/SkillProfileTests.cs`	17 new unit tests for name, description, compatibility, body lines, and file reference depth validation
`eng/validate-descriptions.ps1`	New script for validating per-skill and per-plugin aggregate description sizes
`eng/tests/validate-descriptions.tests.ps1`	6 PowerShell integration tests for the description validation script
`.github/workflows/validate-descriptions.yml`	New CI workflow to run description validation on PRs/pushes affecting plugins
`agentic-workflows/dotnet-msbuild/build.ps1`	Delegates description size checks to the central validation script
`.github/CODEOWNERS`	Removes `nuget-trusted-publishing` entries, removes `@dotnet/appmodel` from `dotnet-aot-compat`

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Add infrastructure to reject skills with oversized descriptions, per issue dotnet#222: - SkillProfiler: check description length, flag DescriptionTooLong - ValidateCommand: block oversized descriptions (survives --verdict-warn-only) - New eng/validate-descriptions.ps1: validates per-skill and per-plugin aggregate limits - New CI workflow: runs validation on PRs touching plugins/ - build.ps1: adds size checks to local build validation - Unit tests for all new SkillProfiler behavior Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Address review feedback: the option description now lists description_too_long alongside execution errors and --require-evals as conditions that remain fatal even when --verdict-warn-only is enabled. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Validate name rules (max 64 chars, lowercase+hyphens, no edge/consecutive hyphens, must match directory), compatibility field max 500 chars, body max 500 lines, and file reference depth. Scan of existing skills found 2 violations: - dotnet-msbuild/msbuild-antipatterns: 582 lines (max 500) - dotnet-msbuild/directory-build-organization: cross-skill ref 3 levels deep Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Fix MakeSkill SkillMdPath to derive from resolved path (SkillProfileTests.cs) - Fix RepoRoot in build.ps1 to be actual repo root (was agentic-workflows/) - Fix body line count off-by-one: trim trailing newlines before splitting - Fix file ref depth: normalize paths, handle .. traversals, strip #anchor, compute depth from directory segments only - Add tests for parent traversal, anchor fragments, ./ normalization, and trailing newline body count Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

JanKrivanek

LGTM.
I'd prefer this to be part of regular validation run

ViktorHofer

Love it. As Jan already commented on, I would also remove implement this purely in the skill-validator and avoid the powershell scripts and the additional workflow.

…mance-validation

Move the per-plugin aggregate description limit (15K chars) from the standalone PowerShell script into ValidateCommand, where it runs after skill discovery. This eliminates the separate validate-descriptions.ps1, its CI workflow, its PowerShell tests, and the delegation from build.ps1. Also rename FailureKind from "description_too_long" to the generic "spec_conformance_failure" so future spec checks reuse the same kind. Deleted: - eng/validate-descriptions.ps1 - eng/tests/validate-descriptions.tests.ps1 - .github/workflows/validate-descriptions.yml Added: - ValidateCommand.CheckAggregateDescriptionLimits() - ValidateCommand.DerivePluginName() - 6 C# tests for aggregate limit + DerivePluginName Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Now that dotnet#247 has merged and fixed the two existing violations (msbuild-antipatterns 582 lines, directory-build-organization cross-ref), these spec conformance checks can be blocking. Also replaces the DescriptionTooLong bool with a general Errors list on SkillProfile, so all spec conformance violations (description length, body lines, file ref depth) flow through the same mechanism. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The lock file diffs were side effects of running build.ps1 to verify the validation changes — they picked up upstream content changes unrelated to this PR. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

danmoseley · 2026-03-06T20:11:56Z

OK @ViktorHofer should be ready for re-review.

danmoseley · 2026-03-06T21:01:31Z

When this is merged, I will open PR for a followup for agent/plugin spec validation holes.
https://github.com/dotnet/skills/compare/main...danmoseley:skills:feature/agent-plugin-validation?expand=1

JanKrivanek

LGTM

danmoseley requested review from JanKrivanek, ViktorHofer and timheuer as code owners March 5, 2026 21:42

Copilot AI review requested due to automatic review settings March 5, 2026 21:42

danmoseley requested a review from dbreshears as a code owner March 5, 2026 21:42

Copilot started reviewing on behalf of danmoseley March 5, 2026 21:42 View session

danmoseley mentioned this pull request Mar 5, 2026

Fix msbuild skill spec conformance violations #247

Merged

Copilot AI reviewed Mar 5, 2026

View reviewed changes

Comment thread eng/skill-validator/tests/SkillProfileTests.cs Outdated

Comment thread .github/CODEOWNERS Outdated

Comment thread agentic-workflows/dotnet-msbuild/build.ps1 Outdated

danmoseley force-pushed the feature/spec-conformance-validation branch from d959b32 to a4d6011 Compare March 5, 2026 21:58

danmoseley and others added 4 commits March 5, 2026 15:09

danmoseley force-pushed the feature/spec-conformance-validation branch from a4d6011 to a072630 Compare March 5, 2026 22:16

danmoseley mentioned this pull request Mar 5, 2026

Enforce description size limits (per-skill 1024, per-plugin 15K) #224

Closed

danmoseley changed the title ~~Add agentskills.io spec conformance checks to SkillProfiler~~ Enforce description size limits and add spec conformance checks Mar 5, 2026

danmoseley commented Mar 5, 2026

View reviewed changes

Comment thread eng/skill-validator/src/Commands/ValidateCommand.cs Outdated

This was referenced Mar 5, 2026

Consider aggregating more of these skills into agents #249

Open

improve validator to cover holes #235

Closed

JanKrivanek approved these changes Mar 6, 2026

View reviewed changes

Comment thread .github/workflows/validate-descriptions.yml Outdated

ViktorHofer reviewed Mar 6, 2026

View reviewed changes

Comment thread eng/skill-validator/src/Commands/ValidateCommand.cs Outdated

danmoseley and others added 2 commits March 6, 2026 12:56

Merge remote-tracking branch 'upstream/main' into feature/spec-confor…

a4d2d3f

…mance-validation

danmoseley requested a review from a team March 6, 2026 19:59

danmoseley and others added 2 commits March 6, 2026 13:05

Revert unrelated lock file changes

63c1e34

The lock file diffs were side effects of running build.ps1 to verify the validation changes — they picked up upstream content changes unrelated to this PR. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

JanKrivanek approved these changes Mar 7, 2026

View reviewed changes

danmoseley merged commit 698b335 into dotnet:main Mar 7, 2026
10 checks passed

danmoseley deleted the feature/spec-conformance-validation branch March 7, 2026 07:16

This was referenced Mar 7, 2026

Add agent and plugin spec conformance validation #277

Closed

Add agent and plugin spec conformance validation #279

Merged

Evangelink mentioned this pull request May 15, 2026

skill-validator: bump per-plugin aggregate description cap from 15K to 20K (and document its informal origin) #655

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enforce description size limits and add spec conformance checks#238

Enforce description size limits and add spec conformance checks#238
danmoseley merged 8 commits into
dotnet:mainfrom
danmoseley:feature/spec-conformance-validation

danmoseley commented Mar 5, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JanKrivanek left a comment

Uh oh!

Uh oh!

ViktorHofer left a comment

Uh oh!

Uh oh!

danmoseley commented Mar 6, 2026

Uh oh!

danmoseley commented Mar 6, 2026

Uh oh!

JanKrivanek left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

danmoseley commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description size limits (from #224, closes #222)

Spec conformance checks (new)

Other fixes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JanKrivanek left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ViktorHofer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

danmoseley commented Mar 6, 2026

Uh oh!

danmoseley commented Mar 6, 2026

Uh oh!

JanKrivanek left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

danmoseley commented Mar 5, 2026 •

edited

Loading