docmutant: real-document evidence for parser bugs, without sharing the document

[wittjeff]

Hi all,

We have started a small open-source tool, docmutant (MIT), and we would like to say what we intend with it and invite you to use it and help shape it.

The problem. Many parser and renderer bugs show only on documents that cannot be shared: manuals, reports, customer files. docling-project/docling-parse#428 rightly asks for evidence from real documents under a license that allows redistribution. For some of our fixes we have the real document (a scanned manual page that crashes Blend2D, a report page that hangs it) but not the right to put it in the regression dataset.

What docmutant does. It changes a PDF at the level of its structure and checks every change with an oracle: a command that says whether the bug still shows. Today that is a crash or a hang; next is a render difference against pypdfium2 with the same metrics as the regression tests. The reduce mode keeps only what the bug needs: one page, the operators that trigger it, image placements with new samples, text swapped for codes of equal width, and no font programs. Each output file gets a manifest, <file>.docmutant.json (no clash with the dataset's .pdf.json), with the source hash, every mutation, the tool version and seed, the oracle results on the buggy and the fixed build, measurements of what remains from the source, the file's license, and, for sources that cannot be shared, a signed reviewer statement.

First run. A scanned manual page that crashes Blend2D at scale 3.0 (the docling-project/docling-parse#365 bug) reduces, in 13 oracle runs, to a 2 KB file: one sheared image placement with flat grey samples. It crashes docling-parse main and renders with the consolidated Blend2D patches; the manifest records both results, and that no words, images or font programs of the source remain.

What we intend.

• Use it for the evidence that Consolidating the open rendering, text and font PRs: scope and evidence requirements docling-parse#428 asks for, and propose a derived file only where no dataset document shows the problem.
• Add a fuzz mode and a metamorphic mode (changes that should not matter, such as equal-width text swaps or page rotation, checked against invariants). Both produce files with exact ground truth.
• Later, Docling-guided anonymization: Docling says what each region is, and docmutant replaces it in place, in every layer of the file, including the tags.

Invitation. It is early: a manifest, a validator and a first reduce mode. You are welcome to use it, and to tell us what a manifest would need before you would accept a derived file as evidence. Contributions are welcome too: oracles for your targets, mutators, or test documents. Nothing is needed from you otherwise, and our own PRs will follow the docling-project/docling-parse#428 rules.

Design notes: docs/DESIGN.md