Skip to content

feat: add disabled_skills config and --no-skills CLI flag#216

Merged
spboyer merged 1 commit into
mainfrom
squad/126-control-skill-loading
Apr 21, 2026
Merged

feat: add disabled_skills config and --no-skills CLI flag#216
spboyer merged 1 commit into
mainfrom
squad/126-control-skill-loading

Conversation

@spboyer
Copy link
Copy Markdown
Member

@spboyer spboyer commented Apr 21, 2026

Summary

Implements #126 — allows users to control which skills are loaded during evals, including a "none" option to disable all skills.

Changes

New config field: disabled_skills

  • ["*"] disables all skill loading entirely
  • Specific skill names (e.g., ["noisy-skill"]) exclude individual skill directories
  • Works by matching against both full path and basename of skill_directories entries

New CLI flag: --no-skills

  • Equivalent to disabled_skills: ["*"] in YAML
  • Overrides any skill_directories configured in the eval spec

Implementation details:

  • Config.AllSkillsDisabled() — checks for wildcard disable or skill_directories: ["none"]
  • Config.FilteredSkillPaths() — returns skill paths with disabled entries removed
  • ExecutionRequest.NoSkills — signals CopilotEngine to skip getSkillDirs() and buildSkillSystemMessage()
  • validateRequiredSkills() — early returns when all skills disabled
  • Updated schema, CLI reference, eval-yaml guide, and schema reference docs

Testing

14 new test cases across 3 test functions:

  • TestConfig_AllSkillsDisabled (6 subtests)
  • TestConfig_FilteredSkillPaths (6 subtests)
  • TestBenchmarkSpec_DisabledSkillsDeserialization (2 subtests)

All existing tests pass. Docs site builds cleanly.

Usage

# Disable all skills
config:
  disabled_skills: ["*"]

# Disable specific skills
config:
  skill_directories: [./skills/a, ./skills/b, ./skills/c]
  disabled_skills: [b]  # only a and c are loaded
# CLI flag
waza run eval.yaml --no-skills

Working as Linus (Backend Developer)

Closes #126

@github-actions github-actions Bot enabled auto-merge (squash) April 21, 2026 21:51
Add support for controlling which skills are loaded during evals:

- New `disabled_skills` config field in eval YAML
  - Use `["*"]` to disable all skills
  - List specific skill names to exclude individual skills
- New `--no-skills` CLI flag to disable all skills from the command line
- `AllSkillsDisabled()` and `FilteredSkillPaths()` helper methods on Config
- Conditional skill loading in CopilotEngine (skips getSkillDirs/buildSkillSystemMessage)
- Early return in validateRequiredSkills when all skills disabled
- Updated eval.schema.json, CLI reference, schema reference, and eval-yaml guide
- 14 new test cases covering all disabled_skills scenarios

Closes #126

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@spboyer spboyer force-pushed the squad/126-control-skill-loading branch from b31ac9b to 9c023ac Compare April 21, 2026 21:54
@spboyer spboyer merged commit 9dda562 into main Apr 21, 2026
4 of 5 checks passed
@spboyer spboyer deleted the squad/126-control-skill-loading branch April 21, 2026 21:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow for control over which skills are loaded (ie, could be 'none') to do comparison grading

2 participants