Skip to content

v0.1.1: first release — eval-authoring loop with a grader gate

Choose a tag to compare

View all tags
@cyanheads cyanheads released this 28 Jun 00:30
v0.1.1
This tag was signed with the committer’s verified signature.
cyanheads Casey Hand
SSH Key Fingerprint: lkZCAfVCK5qsrcvEYCs0BnVRDCX7pKsdCZkw5yYKMuw
Verified
Learn about vigilant mode.
c5402a2
This commit was signed with the committer’s verified signature.
cyanheads Casey Hand
SSH Key Fingerprint: lkZCAfVCK5qsrcvEYCs0BnVRDCX7pKsdCZkw5yYKMuw
Verified
Learn about vigilant mode.

first release — eval-authoring loop with a grader gate

Turn an agent into an eval author: draft → review → revise → submit, with the server running each record's own grader and gating submission on a committability check. Records are plain JSON under EVALS_DATA_DIR; no external API.

Added:

  • 9 tools, all evals_-prefixed — describe_schema, create_draft, get_record, revise_draft, discard_draft, run_check, submit_draft, list_records, export_records
  • eval://record/{id} resource — mirror of evals_get_record
  • Record schema — Zod discriminatedUnion on task_type: numeric, exact_answer, set_answer, mcq, regex_answer, json_answer, free_response
  • Grader DSL — deterministic kinds (numeric via math.js, exact_match, set_match, regex, mcq, json_match) run server-side; llm_rubric routes to sampling
  • Committability gate at submit — the gold passes its own grader, ≥1 declared negative is rejected, and a recorded independent verification agrees
  • Export to JSONL, CSV, Inspect AI, and lm-evaluation-harness
  • Deployment — STDIO + Streamable HTTP, an .mcpb bundle, a Dockerfile, and Codex / Claude Code plugin manifests

147 tests pass; bun run devcheck clean.

Assets 2
Loading