v1.0.0

What's Changed

• update readme by @PaulScemama in #20
• Indra OCR by @PaulScemama in #21
• Abstraction consistency by @PaulScemama in #23
• 1.0.0rc1 by @PaulScemama in #24
• Rename providers/oss to providers/serve; drop the unused "server" in some requests by @PaulScemama in #25
• Nemotron 3.5 Parse: an adapter and a server image; every system's scores on GitHub by @VikParuchuri in #26
• Datalab accurate 94.1 and balanced 92.6: a rerun of the production API by @VikParuchuri in #29
• Read table neighbours off the grid; no more IndexError on overlapping spans by @hunter-heidenreich in #27
• v1.0.0 by @PaulScemama in #30
• README: opb systems by @PaulScemama in #31

New Contributors

• @VikParuchuri made their first contribution in #26
• @hunter-heidenreich made their first contribution in #27

Full Changelog: v0.2.0...v1.0.0

v1.0.0rc1

What's Changed

• update readme by @PaulScemama in #20
• Indra OCR by @PaulScemama in #21
• Abstraction consistency by @PaulScemama in #23

Full Changelog: v0.2.0...v1.0.0rc1

v0.2.0

What's Changed

• Grading reads predictions/ alone by @PaulScemama in #18
• Open-weight OCR models: an adapter and a server image for each by @PaulScemama in #17
• Version 0.2.0 by @PaulScemama in #19

Breaking changes:

• settings.json must say unsupported, the outputs the system can't produce; grading refuses a folder without it. Fix an existing run by running opb benchmark again with the same --out (it calls no vendor for pages already predicted), by opb grade RUN_DIR --unsupported ..., or by adding "unsupported": [];
• unsupported is set once per run, not read off each page: a prediction that leaves out an output its page asks for fails those tests;
• grading reads only predictions/<page_id>.json; records/ adds only cost. A page with a record but no readable prediction is predicted again (opb reparse rebuilds predictions from records without a call);
• scores.jsonl: the record stamp is renamed prediction. Old lines are graded again once, automatically, with no vendor calls and the same verdicts;
• Python: benchmark.read_record is removed; use recorded(run, page_id) for the record and predicted(run, page_id) for the outputs; and
• an OpenAI-compatible base_url other than OpenRouter takes its key from OPB_ENDPOINT_KEY (none if unset); OPENROUTER_API_KEY and OPENAI_API_KEY go only to OpenRouter.

New:

• opb grade DIR [--unsupported ...] grades any folder of predictions, such as your own model's; and
• 13 open-weight providers (chandra-ocr, deepseek-ocr, dots-mocr, glm-ocr, hunyuan-ocr, infinity-parser2, lightonocr, mineru, nanonets-ocr, olmocr, paddleocr-vl, surya-ocr and unlimited-ocr), each with a pinned server image in providers/oss/ (see docs/providers.md).

Full Changelog: v0.1.2...v0.2.0

v0.1.2

What's Changed

• Tighten benchmark primitives so users can build their own loops by @PaulScemama in #7
• The manifest says what to predict instead of having to inspect gold file by @PaulScemama in #8
• Grading refuses instead of re-grading: a run graded with another gold… by @PaulScemama in #9
• Some small tidying up by @PaulScemama in #11
• --workers per adapter by @PaulScemama in #12
• cli raises errors instead of turning them into SystemExit; a manifest… by @PaulScemama in #13
• The manifest's outputs column is required by @PaulScemama in #14
• Scorer digest by @PaulScemama in #15
• Version 0.1.2 by @PaulScemama in #16

Breaking changes:

• The dataset needs a new outputs column in its manifest. The current dataset (revision b8e0a5f, HF main) has it. 0.1.2 refuses older revisions (1c35a52 and earlier, or copies downloaded before) with an error that says how to get the current one. 0.1.1 can't read the current dataset, so upgrade, or stay on 0.1.1 with --dataset hf://datasets/datalab-to/omni_parse_bench@1c35a52.
• A run graded with another gold or other scoring code is refused, instead of being graded again without notice. Pass --rescore to grade every page again; nothing is predicted again. A run graded with 0.1.1 is refused by opb benchmark until it is run once with --rescore; its scores don't change. Later releases that change no scoring file won't need this.
• CLI errors are raised as Python exceptions with their traceback, instead of a one-line message and exit code 1. The exit code is still non-zero. A command whose extra isn't installed now raises ImportError, and the message names the pip install to run.
• gold.Page has a required outputs field: the output types its tests read, as a sorted tuple. Code that builds Page objects itself must pass it.
• A run no longer notices a changed page file, because the page digest was removed. To ask a page again after replacing its file, delete its record. settings.json now records only what the run asks for.

Worth noting:

• --workers now also takes a JSON object of calls at once per adapter, for example {"datalab": 8, "llm": 4}. A model id as a key is refused, with a hint.

Full Changelog: v0.1.1...v0.1.2

v0.1.1

What's Changed

• opb names the extra a command needs when an install lacks it by @PaulScemama in #5

Full Changelog: v0.1.0...v0.1.1

v0.1.0

What's Changed

• Initial release.