Skip to content

v0.2.0

Choose a tag to compare

@PaulScemama PaulScemama released this 14 Oct 20:20
· 18 commits to main since this release
7599d0d

What's Changed

Breaking changes:

  • settings.json must say unsupported, the outputs the system can't produce; grading refuses a folder without it. Fix an existing run by running opb benchmark again with the same --out (it calls no vendor for pages already predicted), by opb grade RUN_DIR --unsupported ..., or by adding "unsupported": [];
  • unsupported is set once per run, not read off each page: a prediction that leaves out an output its page asks for fails those tests;
  • grading reads only predictions/<page_id>.json; records/ adds only cost. A page with a record but no readable prediction is predicted again (opb reparse rebuilds predictions from records without a call);
  • scores.jsonl: the record stamp is renamed prediction. Old lines are graded again once, automatically, with no vendor calls and the same verdicts;
  • Python: benchmark.read_record is removed; use recorded(run, page_id) for the record and predicted(run, page_id) for the outputs; and
  • an OpenAI-compatible base_url other than OpenRouter takes its key from OPB_ENDPOINT_KEY (none if unset); OPENROUTER_API_KEY and OPENAI_API_KEY go only to OpenRouter.

New:

  • opb grade DIR [--unsupported ...] grades any folder of predictions, such as your own model's; and
  • 13 open-weight providers (chandra-ocr, deepseek-ocr, dots-mocr, glm-ocr, hunyuan-ocr, infinity-parser2, lightonocr, mineru, nanonets-ocr, olmocr, paddleocr-vl, surya-ocr and unlimited-ocr), each with a pinned server image in providers/oss/ (see docs/providers.md).

Full Changelog: v0.1.2...v0.2.0