Omni Parse Bench

A document-parsing benchmark with 16,288 tests on 2,937 pages. See Hugging Face.

* Used in creating and verifying at least some tests.

This repository contains a way to benchmark document-parsing systems using our unit-test scorer and dataset as well as your own. Concretely, this repository holds:

• the scorer, which works on its own: a page's output and its tests as input, a verdict per test as output;
• the benchmark: run over our dataset on Hugging Face, or your own; and
• the systems: different registered systems that you can easily evaluate, including open models with a folder to serve each on a GPU.

Contents:

1. Results: scores, costs and refusal counts;
2. How scoring works: tests, families, the headline and tags;
3. Install: the scorer alone, or with the adapters and the dataset;
4. Run the benchmark: one system on the dataset, from a config;
5. Grade: a run again, or predictions of your own;
6. Score on your own: the scorer on its own, in code or from the command line;
7. Predict one page: one page through one system;
8. Results details: run notes, latency and price charts;
9. License; and
10. Citation.

A test is a yes/no question about one page. Each test type belongs to one of three families:

A family's score is the share of its tests that pass, and the headline is the mean of the three family scores.

Each test is tagged by what it checks:

• structure: table, math, form and multi_column;
• rendering: handwriting, tiny, rotated and degraded;
• role: heading, caption, footnote, list, code, figure and header_footer;
• script: ISO 15924 codes (Latn, Hani, Deva, ...); and
• language: ISO 639-3 codes (eng, fra, tha, ...).

Tags never decide whether a test passes, or the headline. They are only for slicing the results, such as the score on handwriting alone. The full design is docs/design.md.

uv pip install omni-parse-bench                 # the scorer, and grading a dataset folder on disk
uv pip install 'omni-parse-bench[harness]'      # + the adapters, to get predictions
uv pip install 'omni-parse-bench[benchmark]'    # + the Hugging Face client, to download our dataset

A run is one system on one dataset. It works in three stages:

1. Plan: check the config, the system's keys and the dataset's files before any call;
2. Predict: send each page to the system and save each answer in the run's folder; and
3. Grade: score the saved answers. Grading calls no system, so it can be done again for free.

A run is one config: a YAML file with every key written out. opb init writes one for a registered system, every option at its default, and on our dataset:

opb init datalab > datalab.yaml

dataset: hf://datasets/datalab-to/omni_parse_bench   # a folder, or a Hugging Face dataset
manifest: manifest.parquet                           # which of its pages, with which files
scorers:
  unit_tests: {}
rows:
  sample: 0
  limit: 0
  only: null
timeout: 900.0           # seconds
workers: 0               # pages in flight at once; 0 for the system's default
system:
  datalab:
    mode: accurate
    ...

opb systems lists every system you can run; opb systems <name> shows its options and the key it needs.

--set path=value changes one value the file has, read as YAML, and refuses a path it hasn't. This runs Datalab on 5 pages:

export DATALAB_API_KEY=...
opb run --config datalab.yaml --set rows.sample=5 --out runs/

opb run shows the plan and asks before calling any system.

Each open model has a folder, omni_parse_bench/harness/providers/serve/<model>/, that serves it on any GPU host:

docker build -t opb-chandra-ocr omni_parse_bench/harness/providers/serve/chandra-ocr
docker run --gpus all -p 8001:8001 opb-chandra-ocr
opb init chandra-ocr > chandra.yaml
opb run --config chandra.yaml --set system.chandra-ocr.base_url=http://<host>:8001

The system's served option names the weights and their revision. Before any call, opb asks the server what it serves, and refuses a run whose server serves something else. docs/providers.md has more information.

runs/datalab-20261026-185915-9c9e/
  config.yaml                         # resolved config; point to this to rerun
  settings.json                       # the question it answers and the launch; resume checks against it
  server.json                         # for a model you serve: what its server said it serves
  records/<id>.json                   # response from the system in full (cost, raw response)
  predictions/<id>.json               # prediction portion of the response
  unit_tests/summary.json             # summary of scores
  unit_tests/scores.jsonl             # scores (line-per-page)
  unit_tests/details/<id>.jsonl       # detailed info (verdict per test) per page

docs/API.md has more information, including how to run on your own dataset.

You can resume a run by giving a path to its folder.

opb resume runs/<run>                                         # continue where it stopped
opb resume runs/<run> --retry-failed                          # also ask again for timeouts, etc.
opb resume runs/<run> --retry-failed --set timeout=1800       # increase timeout

opb run grades its run when it finishes. To grade a run again, or a folder of your own predictions:

opb grade runs/datalab-20261021-160012-3f9a        # a run knows its own dataset
opb grade mine/ --dataset my-dataset/              # your predictions on your own dataset

A summary looks like this:

{
  "scorer": "unit_tests",
  "run": "datalab-20261021-160012-3f9a",
  "rows": 5,
  "failed": {"system": 1, "unusable": 0, "no_answer": 0, "not_scored": 0},
  "coverage": 0.8,
  "headline": 0.657,
  "headline_is": "score",
  "score": 0.657,
  "families": {
    "text": {"score": 0.72, "tests": 120, "documents": 5},
    "tables": {"score": 0.59, "tests": 40, "documents": 3},
    "layout": {"score": 0.66, "tests": 30, "documents": 5}
  }
}

A failed page counts with all its tests failed (UnitTests.score_failure), in the headline and in each family. failed says why pages failed, and coverage is the share of pages scored.

The scorer works on its own, with no dataset and no network. Here is one page with three tests:

from omni_parse_bench import data
from omni_parse_bench.scorers import unit_tests

tests = unit_tests.tests.parse("example-p1", {"tests": [
    {
        "id": "t1",
        "test_type": "present",
        "output_type": "html",
        "derivation": "text_layer",
        "tags": ["Latn", "eng"],
        "args": {"content": {"text": "Quarterly revenue rose 12%", "max_diffs": 1}},
        "settings": {},
    },
    {
        "id": "t2",
        "test_type": "order",
        "output_type": "html",
        "derivation": "text_layer",
        "tags": ["Latn", "eng", "heading"],
        "args": {
            "before": {"text": "Summary", "max_diffs": 0},
            "after": {"text": "Outlook", "max_diffs": 0},
        },
        "settings": {},
    },
    {
        "id": "t3",
        "test_type": "table_cell",
        "output_type": "html",
        "derivation": "publisher_markup",
        "tags": ["table"],
        "args": {
            "cell": {"text": "4.2", "max_diffs": 0},
            "top_heading": {"text": "Q3", "max_diffs": 0},
        },
        "settings": {},
    },
]})

pred = {
    "html": (
        "<h2>Summary</h2>"
        "<p>Quarterly revenue rose 12% on the year.</p>"
        "<table>"
        "<tr><th>Q2</th><th>Q3</th></tr>"
        "<tr><td>3.9</td><td>4.1</td></tr>"
        "</table>"
        "<h2>Outlook</h2>"
    ),
}
verdicts = unit_tests.score(pred, tests)

>>> verdicts
[Verdict(id='t1', result='pass', why=''),
 Verdict(id='t2', result='pass', why=''),
 Verdict(id='t3', result='fail', why="no cell matching '4.2' in any table")]

>>> page = data.Page("example-p1", "example", "example", "pages/example-p1.pdf", "gold/example-p1.json", ("html",))
>>> lines = unit_tests.lines(verdicts, tests, page)
>>> unit_tests.summarize(lines)
{'score': 0.5, 'families': {'text': {'score': 1.0, 'tests': 2, 'documents': 1},
                               'tables': {'score': 0.0, 'tests': 1, 'documents': 1}}}
>>> unit_tests.summarize([x for x in lines if "table" in x["tags"]])   # one tag: the same summary on a slice
{'score': 0.0, 'families': {'tables': {'score': 0.0, 'tests': 1, 'documents': 1}}}

A verdict is pass or fail, with why saying what went wrong.

The same from the command line, which prints the verdicts:

opb score --pred outputs.json --gold gold/<page_id>.json

With [harness] installed, each system is an adapter class. harness.system makes one from a system name and its options and harness.predict sends one page under the same rules a run uses (one time budget, retries, failures recorded) and returns its record:

from pathlib import Path

from omni_parse_bench import harness
from omni_parse_bench.scorers import unit_tests

datalab = harness.system("datalab", {"mode": "accurate"})
record = harness.predict(datalab, harness.Input(Path("pages/<page_id>.pdf"), frozenset({"html", "blocks"})))
record["output"]    # {"pages": [{"html": "...", "blocks": [...]}]}
record["raw"]       # the answer as the system gave it
record["cost"]      # what it cost, its wall time and attempts

unit_tests.score(record["output"]["pages"][0], unit_tests.tests.read("gold/<page_id>.json"))   # one page's outputs

opb predict --system datalab --page pages/<page_id>.pdf --options '{"mode": "accurate"}'

Note: Opus 5.5 and GPT-6.1 Sol use the full-page recorded wall time: HTML plus layout when requested, including failed pages and retries, with 32 workers per model, while the other points retain the earlier median of successful, unretried HTML calls only. These timing definitions differ and are not directly comparable.

Tests on a page the system refused or failed to answer count as failed. Not English is a test with a language tag other than English.

The code is Apache 2.0: see LICENSE, and NOTICE for the third-party code it includes (KaTeX, MIT; a rewrite of olmOCR-bench's table parser, Apache 2.0).

The data's licenses are per source. Each source of the pages, its license and the credit it requires are in licenses/, one file per source. The tests are CC BY 4.0, except those converted from olmOCR-bench (ODC-BY 1.0) and fr-bench-pdf2md (MIT), and those quoting Wikipedia on three handwriting pages (CC BY-SA 4.0).

@misc{omniparsebench2026,
  title        = {Omni Parse Bench: A Document-Parsing Benchmark},
  author       = {{Datalab}},
  year         = {2026},
  howpublished = {\url{https://github.com/datalab-to/omni_parse_bench}},
}

Please also cite the upstream datasets whose pages or tests this benchmark uses; they are listed in licenses/.