Releases: cyanheads/evals-mcp-server
Releases · cyanheads/evals-mcp-server
Release list
v0.1.2: EVALS_DATA_DIR default, mcq_choice_mismatch fix
EVALS_DATA_DIR default, mcq_choice_mismatch fix
Changed:
- EVALS_DATA_DIR defaults to ./evals-data when unset or blank — bare npx/bunx starts with no env setup (#2)
- evals_get_record description trimmed to the action, dropping the verification-subagent meta-coaching (#3)
Fixed:
- evals_create_draft / evals_revise_draft report mcq_choice_mismatch (was task_type_constraint) when an mcq grader's correct is not in choices[] (#1)
151 tests pass; bun run devcheck clean.
v0.1.1: first release — eval-authoring loop with a grader gate
first release — eval-authoring loop with a grader gate
Turn an agent into an eval author: draft → review → revise → submit, with the server running each record's own grader and gating submission on a committability check. Records are plain JSON under EVALS_DATA_DIR; no external API.
Added:
- 9 tools, all
evals_-prefixed — describe_schema, create_draft, get_record, revise_draft, discard_draft, run_check, submit_draft, list_records, export_records eval://record/{id}resource — mirror of evals_get_record- Record schema — Zod discriminatedUnion on task_type: numeric, exact_answer, set_answer, mcq, regex_answer, json_answer, free_response
- Grader DSL — deterministic kinds (numeric via math.js, exact_match, set_match, regex, mcq, json_match) run server-side; llm_rubric routes to sampling
- Committability gate at submit — the gold passes its own grader, ≥1 declared negative is rejected, and a recorded independent verification agrees
- Export to JSONL, CSV, Inspect AI, and lm-evaluation-harness
- Deployment — STDIO + Streamable HTTP, an .mcpb bundle, a Dockerfile, and Codex / Claude Code plugin manifests
147 tests pass; bun run devcheck clean.