Skip to content

feat: allow specifying skill_directories per task (#156)#215

Merged
spboyer merged 2 commits into
mainfrom
squad/156-per-task-skills
Apr 21, 2026
Merged

feat: allow specifying skill_directories per task (#156)#215
spboyer merged 2 commits into
mainfrom
squad/156-per-task-skills

Conversation

@spboyer
Copy link
Copy Markdown
Member

@spboyer spboyer commented Apr 21, 2026

Summary

Allow skill_directories to be specified per-task in eval YAML files, overriding the eval-level setting. This enables tasks that need different skills from the rest of the eval.

Changes

  • internal/models/testcase.go: Add SkillPaths []string field to TestCase struct (yaml:"skill_directories")
  • internal/orchestration/runner.go: Update buildExecutionRequest() to prefer task-level SkillPaths over eval-level, falling back when unset
  • internal/models/testcase_test.go: Add TestLoadTestCase_SkillDirectories — verifies YAML parsing of the new field
  • internal/orchestration/runner_test.go: Add TestBuildExecutionRequest_TaskLevelSkillPaths — verifies override and fallback logic
  • schemas/task.schema.json: Add skill_directories array property
  • site/src/content/docs/guides/eval-yaml.mdx: Add skill_directories to Task Fields table
  • site/src/content/docs/reference/schema.mdx: Add skill_directories section to Task File Format

Behavior

  • Task-level skill_directories replaces (not merges with) eval-level paths
  • If task has no skill_directories, falls back to eval-level
  • Paths resolved relative to the spec directory (same as eval-level)

Example

# eval.yaml
config:
  skill_directories:
    - ./default-skills

tasks:
  - file: special-task.yaml  # can override with its own skill_directories
# special-task.yaml
id: special-task
name: Special task with custom skills
skill_directories:
  - ./custom-skills
inputs:
  prompt: "Do something special"

Closes #156

Working as Linus (Backend Developer)

Add SkillPaths field to TestCase struct so tasks can override
eval-level skill_directories. Task-level paths replace (not merge
with) eval-level paths. If not specified, falls back to eval-level.

Changes:
- Add SkillPaths []string to TestCase (yaml: skill_directories)
- Update buildExecutionRequest() to prefer task-level skill paths
- Add model-level test (TestLoadTestCase_SkillDirectories)
- Add runner-level test (TestBuildExecutionRequest_TaskLevelSkillPaths)
- Update task.schema.json with skill_directories property
- Update eval-yaml guide and schema reference docs

Closes #156

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@spboyer spboyer added the squad:linus Assigned to Linus (Backend Developer) label Apr 21, 2026
@github-actions github-actions Bot enabled auto-merge (squash) April 21, 2026 21:43
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@spboyer spboyer merged commit fc1e1f8 into main Apr 21, 2026
6 checks passed
@spboyer spboyer deleted the squad/156-per-task-skills branch April 21, 2026 21:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

squad:linus Assigned to Linus (Backend Developer)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add the ability to specify skills on a per-task basis, instead of a per-evaluation basis

2 participants