Skip to content

Releases: dtduc-git/jevassert

v0.2.0 — adapter backends

Choose a tag to compare

@dtduc-git dtduc-git released this 19 Sep 11:48

Record any backend, not just Jev.

  • record --backend openai|anthropic|bedrock — LLM baselines through the official system-one-adapter (pip install 'jevassert[adapter]'). Bedrock uses AnthropicBedrock with the standard AWS credential chain.
  • check --input-price/--output-price — cost covers input and output tokens; defaults keep Jev pricing unchanged.
  • Reports and --json show the models actually present in the recording, not the pack's pinned version.
  • GitHub Action gains optional input-price/output-price inputs.
  • 73 tests, ruff clean; existing Jev evidence is byte-identical.

Used to record the first Jev Bench matrix (Jev 1.13.0 vs Claude Sonnet 5 vs a local 7B).

v0.1.0

Choose a tag to compare

@dtduc-git dtduc-git released this 19 Sep 06:57

jevassert v0.1.0

Record/replay regression tests for Jev question packs — assert accuracy, calibration and cost in CI.

Commands

  • record — concurrent calls with retries, --resume, --rpm, --dry-run cost estimate, --shuffle-options, --repeat N (stability)
  • check — accuracy with bootstrap CIs, ECE/Brier, coverage + author thresholds, threshold suggestion, --failures, --partition dev|test, JUnit + markdown evidence
  • compare — paired accuracy deltas + exact McNemar

Pack spec v0 loader (jevassert.packs), implemented from the canonical jev-packs SPEC.md.

Reference evidence: jev-packs registry — 9 packs, 2,990 cases / 6,430 items, verified against jev-1.13.0.

63 tests, offline CI, Apache-2.0.