Skip to content

Fix custom-agent eval example schema#282

Merged
github-actions[bot] merged 2 commits into
mainfrom
spboyer/fix-example-eval-schema
May 23, 2026
Merged

Fix custom-agent eval example schema#282
github-actions[bot] merged 2 commits into
mainfrom
spboyer/fix-example-eval-schema

Conversation

@spboyer

@spboyer spboyer commented May 23, 2026

Copy link
Copy Markdown
Member

This updates the checked-in custom-agent example so it matches the current eval and task schemas and can be copy/paste run with the current CLI.

  • Adds the required eval top-level fields and switches the example to a supported executor
  • Updates the prompt grader to the current config shape
  • Rewrites the example task files to the current task schema and references the fixture files directly
  • Adds a regression test to keep the checked-in example schema-valid

Validation: go test ./...

Docs impact: none beyond the example and schema regression coverage.

Closes #274

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings May 23, 2026 00:43
@github-actions github-actions Bot enabled auto-merge (squash) May 23, 2026 00:43

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the checked-in examples/custom-agent eval suite to match the current eval/task JSON schemas, and adds a regression test to ensure the example remains schema-valid as the schemas evolve.

Changes:

  • Add a schema regression test that validates examples/custom-agent/eval.yaml and its referenced task files.
  • Update the custom-agent example eval.yaml to include required top-level fields (skill, version, metrics) and the current prompt-grader config shape.
  • Rewrite the example task YAMLs to the current task schema, including attaching fixture files via inputs.files.
Show a summary per file
File Description
internal/validation/schema_test.go Adds a regression test to validate the checked-in custom-agent example against the current schemas.
examples/custom-agent/eval.yaml Updates eval spec to current schema (required fields + grader config shape).
examples/custom-agent/tasks/review-clean-code.yaml Migrates task to current task schema and attaches fixture file input.
examples/custom-agent/tasks/find-xss.yaml Migrates task to current task schema and attaches fixture file input.
examples/custom-agent/tasks/find-sql-injection.yaml Migrates task to current task schema and attaches fixture file input.

Copilot's findings

  • Files reviewed: 5/5 changed files
  • Comments generated: 2

Comment thread examples/custom-agent/eval.yaml
Comment thread examples/custom-agent/eval.yaml
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions github-actions Bot merged commit 8994bc1 into main May 23, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

examples: custom-agent eval.yaml is invalid against current schema

3 participants