Skip to content

Releases: cyanheads/evals-mcp-server

v0.1.2: EVALS_DATA_DIR default, mcq_choice_mismatch fix

Choose a tag to compare

@cyanheads cyanheads released this 28 Jun 04:59
v0.1.2
b250ed9

EVALS_DATA_DIR default, mcq_choice_mismatch fix

Changed:

  • EVALS_DATA_DIR defaults to ./evals-data when unset or blank — bare npx/bunx starts with no env setup (#2)
  • evals_get_record description trimmed to the action, dropping the verification-subagent meta-coaching (#3)

Fixed:

  • evals_create_draft / evals_revise_draft report mcq_choice_mismatch (was task_type_constraint) when an mcq grader's correct is not in choices[] (#1)

151 tests pass; bun run devcheck clean.

v0.1.1: first release — eval-authoring loop with a grader gate

Choose a tag to compare

@cyanheads cyanheads released this 28 Jun 00:30
v0.1.1
c5402a2

first release — eval-authoring loop with a grader gate

Turn an agent into an eval author: draft → review → revise → submit, with the server running each record's own grader and gating submission on a committability check. Records are plain JSON under EVALS_DATA_DIR; no external API.

Added:

  • 9 tools, all evals_-prefixed — describe_schema, create_draft, get_record, revise_draft, discard_draft, run_check, submit_draft, list_records, export_records
  • eval://record/{id} resource — mirror of evals_get_record
  • Record schema — Zod discriminatedUnion on task_type: numeric, exact_answer, set_answer, mcq, regex_answer, json_answer, free_response
  • Grader DSL — deterministic kinds (numeric via math.js, exact_match, set_match, regex, mcq, json_match) run server-side; llm_rubric routes to sampling
  • Committability gate at submit — the gold passes its own grader, ≥1 declared negative is rejected, and a recorded independent verification agrees
  • Export to JSONL, CSV, Inspect AI, and lm-evaluation-harness
  • Deployment — STDIO + Streamable HTTP, an .mcpb bundle, a Dockerfile, and Codex / Claude Code plugin manifests

147 tests pass; bun run devcheck clean.