Call for parser submissions: DocFailBench v0.1 Combined Public RC

[Travor278]

DocFailBench v0.1 Combined Public RC is open for parser submissions.

DocFailBench is a failure-oriented benchmark for PDF-to-Markdown, OCR, and VLM document parsers. It does not only ask whether a page looks broadly similar; it checks small executable facts: table cells, formulas, reading order, captions, page furniture, and optional bbox grounding.

Release links:

• GitHub repo: https://github.com/Travor278/DocFailBench
• Release: https://github.com/Travor278/DocFailBench/releases/tag/v0.1-combined-public-rc
• Hugging Face dataset: https://huggingface.co/datasets/Travor278/DocFailBench
• Submission issue: Call for parser submissions: DocFailBench v0.1 Combined Public RC #1

Frozen target:

• Release name: DocFailBench-v0.1-combined-public-rc
• Cases: 116
• Assertions: 877
• Cached baselines: 7 parsers
• Case file: data/releases/docfailbench_v0_1_combined_public_rc_cases.json

Current baseline snapshot:

Parser	Passed	Failed	Score
Marker	621	256	0.7081
PyMuPDF bbox	612	265	0.6978
Docling	599	278	0.6830
PyMuPDF plain	589	288	0.6716
Qwen-VL API	559	318	0.6374
MinerU	496	381	0.5656
PaddleOCR	334	543	0.3808

What we are looking for:

• new parser baselines,
• reproduction reports for existing parsers,
• failure cases where the assertions are too weak or too brittle,
• public PDF suggestions for future releases,
• adapters that make parser submissions easier to reproduce.

A useful parser submission should include parser name/version, exact run command, prediction JSON, evaluation JSON, hardware/runtime metadata, and model/API run date for hosted parsers. The full guide is here:

https://github.com/Travor278/DocFailBench/blob/main/docs/submitting-parser-results.md

Please post general feedback in this discussion. For concrete parser results, use the submission issue or open a PR so the result can be reviewed and added to the leaderboard.