-
Notifications
You must be signed in to change notification settings - Fork 0
Open
Description
Context
The eval file validation in eval-validator.ts uses procedural if/else logic (~500+ lines). A parallel Zod schema (eval-file.schema.ts) was added in #385 for JSON Schema generation, creating two sources of truth.
Proposal
Refactor eval-validator.ts to use the Zod schema as the single source of truth for both:
- Runtime validation (Zod
.safeParse()) - JSON Schema generation (
zod-to-json-schema)
Benefits
- Single source of truth for eval file structure
- Better error messages from Zod
- Removes ~500 lines of manual validation code
- Type-safe parsing (no type casts)
Considerations
- The current procedural validator supports warnings (not just errors) — Zod only does pass/fail
- Custom evaluator types use
.passthrough()which needs careful handling - Backward-compatible aliases (eval_cases, script, expected_outcome) need Zod transforms
- Extensive test coverage exists in
eval-validator.test.ts— migration should preserve all test cases
Scope
packages/core/src/evaluation/validation/eval-validator.ts→ refactor to use Zodpackages/core/test/evaluation/validation/eval-validator.test.ts→ update test setup- Remove the separate
eval-file.schema.tsonce validator uses Zod natively
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels