Skip to content

Add skill body injection opt-out#292

Merged
github-actions[bot] merged 1 commit into
mainfrom
spboyer/issue-285-skill-injection-opt-out
May 23, 2026
Merged

Add skill body injection opt-out#292
github-actions[bot] merged 1 commit into
mainfrom
spboyer/issue-285-skill-injection-opt-out

Conversation

@spboyer

@spboyer spboyer commented May 23, 2026

Copy link
Copy Markdown
Member

Trigger-precision evals need to measure whether an agent invokes a skill, but the existing skill: path injected the full target skill body into the system prompt. This adds an explicit opt-out so evals can keep their skill association while requiring the agent to use the skill tool for the body.

Summary

  • Adds config.inject_skill_body, defaulting to true to preserve existing behavior.
  • Threads the setting through normal eval execution and waza trigger so false suppresses only the target <skill_context> block.
  • Keeps skill discovery, SDK skill directories, and compact <available_skills> summaries active unless skills are fully disabled.
  • Updates schema, README, and docs site guidance for trigger-precision evals.

Testing

  • go test ./...
  • cd site && npm ci --silent && npm run build

Notes

This task was flagged as "needs review" because it touches medium-complexity CLI/execution behavior. Please have a squad member review before merging.

Fixes: #285

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings May 23, 2026 19:40
@github-actions github-actions Bot enabled auto-merge (squash) May 23, 2026 19:41
@github-actions github-actions Bot merged commit 0ab10ee into main May 23, 2026
8 of 9 checks passed

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new evaluation config flag to opt out of injecting the full target skill/agent body into the system prompt, enabling “trigger-precision” evals to measure actual skill tool invocation while preserving skill association and discovery behavior.

Changes:

  • Introduces config.inject_skill_body (default true) and threads it through eval orchestration and waza trigger via ExecutionRequest.SuppressSkillBody.
  • Updates Copilot SDK execution prompt construction to conditionally omit the target <skill_context> while keeping <available_skills> summaries.
  • Updates JSON schema + docs, and adds/extends unit tests across schema validation, spec parsing, orchestration/trigger runners, and skill message building.
Show a summary per file
File Description
site/src/content/docs/reference/schema.mdx Documents the new inject_skill_body config field and its trigger-precision use case.
site/src/content/docs/guides/eval-yaml.mdx Adds inject_skill_body to config examples/table and documents trigger-precision guidance.
schemas/eval.schema.json Extends eval JSON schema to allow config.inject_skill_body with default true.
README.md Documents how to disable target skill body injection for trigger-precision evals.
internal/validation/schema_test.go Adds schema validation coverage for inject_skill_body.
internal/trigger/runner.go Propagates config into execution via SuppressSkillBody.
internal/trigger/runner_test.go Verifies SuppressSkillBody is set when opting out.
internal/orchestration/runner.go Propagates config into execution requests via SuppressSkillBody.
internal/orchestration/runner_test.go Adds request-building coverage for default vs suppressed injection.
internal/models/spec.go Adds InjectSkillBody *bool and ShouldInjectSkillBody() defaulting logic.
internal/models/spec_test.go Tests defaulting + YAML deserialization of inject_skill_body.
internal/execution/skill_injection_test.go Adds unit test ensuring target <skill_context> is omitted when suppressed.
internal/execution/engine.go Extends ExecutionRequest with SuppressSkillBody.
internal/execution/copilot.go Threads suppression into buildSkillSystemMessage(..., injectSkillBody) behavior.
internal/execution/copilot_test.go Updates tests to match the new buildSkillSystemMessage signature.

Copilot's findings

  • Files reviewed: 15/15 changed files
  • Comments generated: 0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Provide an officially supported way to run skill trigger evals that do NOT inject the skill into the prompt

3 participants