Repository navigation
v0.2.0
What's Changed
- Grading reads predictions/ alone by @PaulScemama in #18
- Open-weight OCR models: an adapter and a server image for each by @PaulScemama in #17
- Version 0.2.0 by @PaulScemama in #19
Breaking changes:
- settings.json must say
unsupported, the outputs the system can't produce; grading refuses a folder without it. Fix an existing run by runningopb benchmarkagain with the same--out(it calls no vendor for pages already predicted), byopb grade RUN_DIR --unsupported ..., or by adding"unsupported": []; unsupportedis set once per run, not read off each page: a prediction that leaves out an output its page asks for fails those tests;- grading reads only
predictions/<page_id>.json;records/adds only cost. A page with a record but no readable prediction is predicted again (opb reparserebuilds predictions from records without a call); scores.jsonl: therecordstamp is renamedprediction. Old lines are graded again once, automatically, with no vendor calls and the same verdicts;- Python:
benchmark.read_recordis removed; userecorded(run, page_id)for the record andpredicted(run, page_id)for the outputs; and - an OpenAI-compatible
base_urlother than OpenRouter takes its key fromOPB_ENDPOINT_KEY(none if unset);OPENROUTER_API_KEYandOPENAI_API_KEYgo only to OpenRouter.
New:
opb grade DIR [--unsupported ...]grades any folder of predictions, such as your own model's; and- 13 open-weight providers (chandra-ocr, deepseek-ocr, dots-mocr, glm-ocr, hunyuan-ocr, infinity-parser2, lightonocr, mineru, nanonets-ocr, olmocr, paddleocr-vl, surya-ocr and unlimited-ocr), each with a pinned server image in
providers/oss/(see docs/providers.md).
Full Changelog: v0.1.2...v0.2.0