A document-parsing benchmark for the documents business actually runs on.
BizDocBench measures how well document parsers turn real corporate files — PDFs, PowerPoint decks, Word documents and web pages — into the structured text that RAG pipelines and agents consume. It scores six axes (coverage, speed, hierarchy, faithfulness, tables, retrieval) over 77 hand-picked business documents with human-authored ground truth.
| # | Tool | Version | Score | Coverage | Speed | Hierarchy | Faithfulness | Tables | Retrieval |
|---|---|---|---|---|---|---|---|---|---|
| 1 | DocSlicer | 0.2.0 | 0.880 | 1.000 | 0.884 | 0.847 | 0.982 | 0.805 | 0.760 |
| 2 | Docling | 2.115.0 | 0.704 | 1.000 | 0.381 | 0.491 | 0.893 | 0.747 | 0.711 |
| 3 | MarkItDown | 0.1.6 | 0.584 | 1.000 | 0.851 | 0.060 | 0.797 | 0.258 | 0.536 |
| 4 | Unstructured | 0.24.1 | 0.580 | 0.909 | 0.107 | 0.433 | 0.906 | 0.481 | 0.643 |
| 5 | OpenDataLoader-PDF | 2.5.0 | 0.536 | 0.584 | 1.000 | 0.385 | 0.648 | 0.266 | 0.332 |
| 6 | PyMuPDF4LLM | 1.28.0 | 0.452 | 0.597 | 0.649 | 0.109 | 0.646 | 0.355 | 0.355 |
| 7 | MinerU | 3.4.4 | 0.411 | 0.597 | 0.135 | 0.422 | 0.618 | 0.301 | 0.391 |
| 8 | Marker | 2.0.0 | 0.374 | 0.597 | 0.160 | 0.193 | 0.612 | 0.301 | 0.378 |
Full details of can be found in results/run_20260726_201631.json, with per-document numbers in the per-axis CSVs and a per-format breakdown in score_by_format_20260726_201631.csv.
Existing document-parsing benchmarks focus heavily on scanned images. However, modern knowledge workers primarily deal with native digital formats—PDFs, .docx contracts, .pptx decks, and HTML. Scanned or handwritten files are the exception, not the rule.
When a text layer is already present, OCR is irrelevant. The true bottlenecks are reading order, dense tables, slide geometry, heading hierarchy, and speed.
Speed is especially overlooked. While latency is tolerable for overnight batch jobs, nobody waits ten minutes for a 100-page credit agreement to parse inside a real-time agent loop. Heavy VLM parsers waste costly compute re-extracting text native files already contain, a poor trade-off for standard digital documents. Ultimately, a parser can top a scanned-page leaderboard while still scrambling a native financial report into unordered text.
| Benchmark | Corpus | Unit | Primary focus | Formats | Speed scored |
|---|---|---|---|---|---|
| OmniDocBench | 1,651 pages, 9 sources incl. handwriting, newspapers | Page image | Text, formula, table and reading-order extraction via edit distance | No | |
| olmOCR-Bench | 1,403 PDFs, 8 splits incl. degraded scans, tiny text | Page | Machine-checkable assertions on OCR output | No | |
| ParseBench | ~2,078 pages / 1,211 enterprise docs (insurance, finance, government) | Page | Tables, charts, content faithfulness, semantic formatting, visual grounding | No — cost ¢/page instead | |
| DocVQA | Document images | Page image | Visual question answering | Images | No |
| PubLayNet | 360K pages, PubMed Central | Page image | Layout element detection, 5 classes | PDF→image | No |
| DocBank | 500K pages, arXiv LaTeX | Page image | Token-level layout annotation, 13 classes | PDF→image | No |
| DocLayNet | 80,863 human-annotated pages | Page image | Layout segmentation, 11 classes | PDF→image | No |
| Office Comprehension Bench | 368 files, 12 industries | File | Whether a model comprehends an Office file | DOCX, XLSX, PPTX | No |
| BizDocBench | 77 curated business documents | Whole document | Parser output fidelity, retrieval survival, throughput | PDF, DOCX, PPTX, HTML | Yes |
BizDocBench scores what a parser emits — headings, tables, reading order, chunks — against human ground truth. It differs on four points:
- Native formats. PDF, DOCX, PPTX and HTML, in the proportions a knowledge worker encounters them.
- Whole documents, not pages. Agents reason over the whole file: they follow a heading hierarchy and retrieve chunks from anywhere in the document. Scoring at document level is what makes the hierarchy and retrieval axes possible.
- Retrieval as an outcome. One axis measures whether the evidence needed to answer a question survives parsing, chunking and a hard context budget.
- Speed as a scored axis, weighted equally with the four quality axes.
Scope: 77 documents, each quality axis covering 9–19 of them — smaller than the other benchmarks listed, so read the per-axis document counts alongside the scores. Scanned documents, handwriting and OCR are out of scope.
Important
BizDocBench is multi-format by design. 31 of the 77 documents are DOCX, PPTX or HTML, and roughly 40 % of the gold annotations sit on those files, so a PDF-only parser scores low here even if it is excellent at PDFs. That is the intended reading, not an artefact: a parser that cannot open a deck is not a candidate for a multi-format agentic workflow.
77 documents, published as a dataset on Hugging Face:
Every document is a real, publicly available artefact from a recognised institution. Selection prioritised three kinds of spread simultaneously — business topic, industry vertical, and file format — and within each vertical, only top-tier issuers.
| Domain | Docs | What's in it |
|---|---|---|
| Legal | 21 | Commercial agreements and licences (Adobe, NVIDIA, BMO, an executed merger agreement), statutes and regulations (EU AI Act, EU directives, US House bills), court rulings and consent orders, counsel letters and practice guides — from firms including Wachtell, Skadden and more |
| Finance | 19 | SEC filings (Apple and JP Morgan 10-Ks, EDGAR HTML), annual reports (HSBC, Aston Martin), earnings releases and slides (Alphabet, AstraZeneca, Pfizer), investor presentations (JP Morgan Investor Day, Morgan Stanley, Perella Weinberg), equity research (DBS), financial review decks |
| Technical | 12 | Specifications and prescribing information, SOPs and deviation/CAPA procedures, service and owner manuals (Range Rover workshop manual, 737 flight manual), regulatory guidance (EU GMP), API documentation |
| Academic | 10 | Peer-reviewed papers (BERT, PageRank, CRISPR, Black–Scholes), conference and research presentations (Eli Lilly, Novo Nordisk), a course handout (Cambridge University), and a 1,450-page physics textbook |
| Business | 8 | Day-to-day operations material: consulting decks (McKinsey, BCG), engagement and review reports (Lazard, KPMG), supply-chain updates (NHS), brand guidelines (P&G), a product catalog, a project status update, a press release |
| Policy | 7 | HR policy manuals, codes of conduct (Wells Fargo), information-security policy, responsible-AI standards (Microsoft), and public-sector policy (SFPD, FAA) |
Document length ranges from 3 to 1,952 pages, so parsers are exercised on short memos and on documents long enough to expose real throughput differences.
| Format | Docs |
|---|---|
| 46 | |
| PPTX | 12 |
| HTML | 10 |
| DOCX | 9 |
Documents are not redistributed in this repository. documents.json is the manifest (id, domain, format, page count, origin URL, annotation coverage); the files themselves come from the Hugging Face dataset.
Six axes. The four quality axes produce per-document numbers that are macro-averaged (every document counts equally) into a per-tool axis score in [0, 1]; Speed and Coverage are aggregates, described below.
Wall clock from the moment the parser is handed the document until it returns a result. Measured uncached on all 77 documents. Chunking is excluded — the timer covers parsing only.
The scored number is aggregate throughput: total pages ÷ total parse seconds, over the documents the tool actually parsed. This is the one axis that is not macro-averaged. Averaging per-document rates would weight a 3-page memo the same as a 1,952-page textbook, which understates the difference between tools that degrade on long documents — and long documents are where the wall time goes. On run 20260726_201631 the two aggregations differ by nearly 2×: docslicer is 6.8× docling by mean per-document rate but 12.7× by aggregate throughput, over 316 s vs 4,005 s across the same 46 PDFs.
Both are written to the CSV — pages_per_sec_mean and pages_per_sec_aggregate, each also broken out PDF-only — but only the aggregate feeds the composite.
Against hand-annotated heading trees. Headings are matched fuzzily (Levenshtein ≤ 2 after normalising case, whitespace and leading section numbering), so 2.1 Risk Factors matches Risk Factors. Normalisation strips decimal (1.2.3), Roman (IV.) and alphabetic (A.) numbering, and collapses internal whitespace so line-break hyphenation artefacts like PAGE-\nRANK survive.
Recall and precision are computed as two independent sweeps — each gold heading looks for its best match among the tool's headings, and each tool heading looks for its best match among gold — rather than as a one-to-one assignment. Where a document repeats a heading text, this is more forgiving than bipartite matching.
Parent-chain accuracy scores each gold parent→child edge: the tool must find the child, report a parent for it, and have that parent fuzzy-match the gold parent. A child the tool never found, or found with no parent, scores the edge zero. Root headings have no edge and are not counted.
Does the parser return the text in the order a human reads it? Hard layouts are where this breaks: a multi-column PDF, or a slide deck with textboxes arbitrarily arranged on the slides. The text can be fully present and still be useless if it arrives scrambled.
One number: normalised indel distance between the tool's full output and a hand-written gold rendering of the document in reading order — concretely, rapidfuzz.fuzz.ratio(gold, pred) / 100, an indel similarity over the whole span. Both sides are normalised identically (NFKC, HTML tags stripped, markdown heading/list markers and inline emphasis stripped, whitespace collapsed to single spaces); case is preserved, per NID convention. Because the gold is plain text, a tool's ## , - or ** must not register as a difference — hence the marker stripping.
Tables are included inline as plain cell text, and both sides must reduce to the same plain text: tool pipe tables are run through an HTML conversion first, so | a | b | collapses to a b exactly as the gold's <td>a</td><td>b</td> does. Charts and images carry no scorable text and are dropped from the gold — this benchmark does no OCR or chart reading.
A global similarity is deliberate. A per-token metric makes every stray token a categorical penalty, which forces an exclusion machine around furniture and line-merge noise; under fuzz.ratio a difference moves the score only by its character fraction of the document, which for a retained footer is negligible. That is why furniture (headers, footers, page labels) stays inline in the gold and is simply scored as content.
4. Tables — table_detection_recall, table_detection_precision, table_record_match, table_header_accuracy
Detection first, content second. A parser table matches a gold table when ≥ 50 % of the gold table's cell-value fingerprint appears in it. Content is then compared row-by-row, aligned by column index (gold headers are often placeholders like col_1, so name-based alignment is unreliable), with numeric normalisation for currency symbols, thousands separators, percentages, parenthesised negatives and k/M/B suffixes. Row order is not required to match: for each gold row, every row in the matched parser table is tried and the best-scoring one wins.
Three thresholds govern the axis:
| Constant | Value | What it gates |
|---|---|---|
DETECTION_OVERLAP_THRESHOLD |
0.50 | fraction of a gold table's cell fingerprint that must appear in a parser table for the two to be considered the same table |
RECORD_MATCH_THRESHOLD |
0.80 | cell agreement at which a gold row counts as matched — the axis's one genuinely tunable knob |
TEXT_SIMILARITY_THRESHOLD |
0.90 | string similarity for cell and header-column comparison, consistent with the faithfulness fuzzy threshold |
Header handling is scored as its own KPI rather than folded into content accuracy, and per header column rather than per header row: each gold header column's content must appear in a row the tool also flagged as a header. Classification is still required — a tool that emits header text as an ordinary data row, or that never marks headers at all (MinerU's HTML has no <th>), scores zero here even when its record match is fine. Two consequences: a tool that emits a two-row header as one row is not penalised for the shape, and a tool that misclassifies a header does not have that error silently multiplied across every data row. Gold tables with no header row contribute nothing to this KPI. Undetected tables count as missed rows and missed header columns — never as skipped.
Both gold and parser tables are a flat list of rows carrying a header flag, and content matching pools all of them. This diverges from ParseBench, which splits headers from records and scores only the records — under that split, classifying a gold data row as a header removes it from the comparison pool entirely, guaranteeing an unrecoverable miss, while the reverse error costs nothing at all.
The end-to-end question a RAG builder actually cares about: given a context budget, does the evidence survive the parse?
246 human-authored questions across 19 documents, each with verbatim evidence spans. Each tool's chunks are embedded with a frozen local embedder (nomic-embed-text), ranked by cosine similarity to the question, concatenated and hard-truncated at 2 000 / 8 000 / 32 000 characters. The score is the fraction of gold evidence characters present in the truncated context, averaged over questions and then over documents. Fully deterministic — no LLM judge, no answer-generation layer.
Three details that affect the number:
- Matching is per span, not per character. A span counts only if it appears whole in the context as a whitespace-normalised substring; it then contributes its full length. Spans are all-or-nothing, weighted by how long they are — half a span present scores zero.
- There is an answer fallback. If no span matches, the gold
answeris checked as a verbatim substring, and a hit scores the question 1.0. This exists for table-row evidence, where parsers emit wildly different pipe-vs-prose formats but the value itself survives. It is a deliberate leniency and the main reason a score can exceed what span matching alone would give. - A tool's heading is prepended to its chunk when the tool reports one separately (docling's
meta.headings, unstructured'sorig_elements). Tools that already carry the heading inside the chunk text reportNoneand get no duplication. Without this, evidence appearing only in a heading would be invisible to every tool that separates them.
The QA files also carry gold answers, and annotations/qa/HOWTO_qa_authoring.md describes answer_em / answer_f1. That answer layer is not implemented — retrieval is the only scored layer today, and the answers are used solely for the fallback above.
How many of the 77 documents the parser produced usable output for. Reported per format as well as in total, so a narrow-but-solid parser is distinguishable from a broad-but-fragile one.
A document counts only if the parser handled it itself, which rules out three cases:
- a hard error,
- a file whose format the tool does not support,
- an assisted parse — the harness records
fetch_assistedwhen a tool needed the fallback fetcher to retrieve an http(s) source, andparse_assistedwhen content had to be preprocessed before the tool would read it (SGML, for instance). A fallback did the reading, so it is not the tool's own coverage.
All three score zero on Coverage and on the four quality axes for that document, and are excluded from Speed entirely. The assisted case mostly affects HTML documents, and the per-tool assisted rate is reported alongside coverage.
| Axis | Annotated docs | Ground truth |
|---|---|---|
| Speed | 77 | none needed |
| Hierarchy | 9 | nested heading trees |
| Faithfulness | 9 | full reading-order markdown + HTML tables |
| Tables | 12 | cell-level table records with header flags |
| Retrieval | 19 | 246 Q&A pairs with evidence spans |
| Coverage | 77 | none needed |
Every annotated document is scored on its axis; no documents are excluded.
annotations/content/<id>.content.md has no JSON schema — it is the plain reading-order text described under Faithfulness above, consumed directly by eval/faithfulness.py with no compile step: furniture inline and untagged, tables and TOC entries as self-delimiting <table> HTML with col/rowspan preserved, charts and images omitted, no page markers.
Authoring workflow, seed tools and the rules for each axis: CONTRIBUTING.md, and annotations/qa/HOWTO_qa_authoring.md for QA specifically.
A single composite in [0, 1]. The six axes are weighted equally, and within each axis its KPIs are weighted equally.
BizDocBench Score = ( Coverage + Speed + Hierarchy + Faithfulness + Tables + Retrieval ) / 6
| Axis | Score | KPIs |
|---|---|---|
| Speed | clamp( log10(1 + parse_pages_per_sec) / log10(1 + 50), 0, 1 ) |
1 |
| Hierarchy | mean( heading_recall, heading_precision, parent_chain_accuracy ) |
3 |
| Faithfulness | nid |
1 |
| Tables | mean( det_recall, det_precision, record_match, header_accuracy ) |
4 |
| Retrieval | mean( recall@2K, recall@8K, recall@32K ) |
3 |
| Coverage | documents parsed successfully / 77 |
1 |
Equal weighting. In an agent workflow no axis matters more than another: a scrambled reading order, a missed table, a flattened hierarchy, evidence that never retrieves, a 10-minute parse and a file that will not open are all disqualifying. Weighting by axis rather than across all 13 KPIs keeps that balance — tables decompose into four numbers and faithfulness into one, which reflects metric design, not importance.
- Speed is scored against a fixed 50 pages/sec, which earns full marks — a 100-page document parsed in two seconds, fast enough that an agent workflow no longer waits on it. The reference is fixed rather than set by the fastest tool in the run, so a tool's score does not shift when other tools are added.
- Detection recall and precision are averaged, not combined as F1, which would re-weight the table axis. Both are reported individually.
- The quality axes are macro-averaged by document, so one 200-row table cannot dominate an axis. Speed is the exception — it is aggregate throughput, since a rate is not a per-document score (see the Speed axis above).
- A parse failure scores 0 on the four quality axes for that document, never a silent skip. This is why
eval/score.pyrebuilds the document set fromdocuments.jsonrather than trusting the CSVs: the quality axes only write a row when they had a cached parse to score, so a tool that crashed is simply absent from their CSV, and averaging the rows that are present would reward failing loudly. The denominator is every in-scope document annotated for that axis; a missing row counts as 0. - Speed covers only the documents a tool parsed. Failure is what Coverage measures; zero-filling Speed as well would charge a tool twice for the same failure and give Coverage a de facto double weight. Speed answers how fast on what it can handle, Coverage how much it can handle. A tool that parses nothing scores 0, having demonstrated no speed at all.
- Weights are configuration. Recompute with your own if your workflow values something else.
- Coverage is scored over the full 77. A PDF-only parser fails 31 documents and is marked down for it. Intended: a tool that cannot open a deck is not a candidate for a multi-format workflow.
- Only a full run yields the headline number. Scope is whatever the speed CSV covers, so under
--idsor--limitCoverage becomes a fraction of that scope rather than of 77. Partial runs are comparable within themselves, not against a published figure.
Eight adapters ship with the harness. Each is pinned in a lockfile so a run is reproducible.
| Tool | Version | Notes |
|---|---|---|
| Docling | 2.115.0 | native chunker |
| MinerU | 3.4.4 | isolated venv (transformers < 5) |
| Marker | 2.0.0 | isolated venv (transformers ≥ 5.12), needs a llama-server binary |
| Unstructured | 0.24.1 | no hierarchy concept |
| PyMuPDF4LLM | 1.28.0 | |
| MarkItDown | 0.1.6 | |
| OpenDataLoader-PDF | 2.5.0 | bundled Java CLI, needs a JRE 11+ |
| DocSlicer | — | native chunker |
Marker and MinerU have mutually incompatible transformers pins, so each gets its own virtualenv and is driven over a pipe (eval/parsers/isolated.py).
Adding your own parser means writing one adapter in eval/parsers/ implementing the contract in eval/parsers/base.py — parse / chars / dump / load / chunks, plus an optional warmup — registering it in eval/registry.py, and adding an entry to each axis's extractor map. See CONTRIBUTING.md.
git clone https://github.com/DocSlicer/BizDocBench.git
cd bizdocbench
# Three venvs: main harness + one each for marker and mineru
./scripts/setup_envs.sh python3.11
# Fetch the corpus into documents/ (pinned to the v1.0 tag)
huggingface-cli download DocSlicer/BizDocBench --repo-type dataset --revision v1.0 --local-dir documentsExternal requirements: a Java 11+ runtime for OpenDataLoader, and a llama-server binary (brew install llama.cpp, or Ollama's bundled copy) for Marker's table fallback. Model weights download on each parser's first run. Nothing is sent to a hosted API.
./run_all.shSix stages, in this order:
- Parse — the only stage that parses. Per
(tool, document): warm up, time the parse, cache the artifact it just measured, chunk it. Speed KPIs come from this single pass, so the bytes each quality axis scores are the bytes that were timed. - Hierarchy · 3. Faithfulness · 4. Tables · 5. Retrieval — all read
cache/and cannot parse. - Score — reads only the CSVs the first five stages wrote and prints the composite leaderboard. No parsing, no cache access, so it is re-runnable at any time.
Useful flags:
./run_all.sh --ids finance_004 legal_001 # specific documents
./run_all.sh --limit 3 # first N documents
python3 -m eval.parse --skip-fresh # fill cache gaps without re-measuring
python3 -m eval.parse --tools docslicer # re-measure one toolIndividual axes:
.venv/bin/python -m eval.hierarchy --tool all
.venv/bin/python -m eval.faithfulness --tool all
.venv/bin/python -m eval.table --tool all --doc-id finance_002
.venv/bin/python -m eval.rag --tool allThe composite, over whichever CSVs already exist:
.venv/bin/python -m eval.score # newest run in results/
.venv/bin/python -m eval.score --run-id 20260726_195954Results land in results/ as one CSV per axis, <axis>_<run_id>.csv, plus score_<run_id>.csv with one row per tool — the composite and its six axis scores. Each row carries the tool_version that produced the output being scored and the run_id, so CSVs from several runs concatenate into one analyzable table.
Everything constant within a run — machine (CPU, cores, RAM), OS, Python, git commit and whether the tree was dirty, per-axis argv, the tool→version map, the RAG embedder — is recorded once in results/run_<run_id>.json, joined to any CSV on run_id. ./run_all.sh exports a single BIZDOCBENCH_RUN_ID so all five axes share one record; running an axis by hand mints its own. Speed numbers in particular are not comparable across machines, so read them against that file.
documents.json manifest: 77 docs — id, domain, format, pages, origin URL
annotations/
hierarchy/ <id>.headings.json nested gold heading trees
content/ <id>.content.md gold reading-order rendering
tables/ <id>.tables.json gold table records
qa/ <id>.qa.json questions + evidence spans
eval/
parse.py step 1: the only module that parses (speed KPIs + cache)
hierarchy.py faithfulness.py table.py rag.py
score.py step 6: the composite score, from the other steps' CSVs
cache.py addressing + staleness for cache/; the axes' only reader
registry.py per-tool facts: formats, versions, isolation, config
embed_cache.py frozen local embedder for the retrieval axis
parsers/ one adapter per tool + isolated-venv RPC
annotation_tools/ helpers used to author and validate the ground truth
requirements/ three lockfiles (main / marker / mineru)
run_all.sh end-to-end reproducible run
Metric definitions, matching rules, thresholds and annotation schemas are all in this file — see What is measured and The BizDocBench Score above. Each module in eval/ documents its own implementation in its docstring.
New parser adapters, additional annotated documents, and metric critiques are all welcome. Annotation is the bottleneck — a document with a hand-authored heading tree, table set or QA file is the single most useful contribution. See CONTRIBUTING.md.
Benchmark code: see LICENSE. Corpus documents remain under their original publishers' terms; the Hugging Face dataset card records provenance per document.