Benchmarking document-conversion quality and fidelity #11759
StephanMeijer
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Does anyone know of tools, frameworks, or established methods for benchmarking document-conversion quality?
I’m particularly interested in evaluating how faithfully a conversion such as DOCX → Markdown preserves document structure, including headings and section hierarchy, tables, ordered and unordered lists, links, emphasis, inline formatting, textual content, and logical reading order, etc.
Ideally, I’m looking for a way to compare the converted output against a reference and produce a meaningful score or structured diff, rather than relying entirely on manual review.
Are there existing benchmarks, metrics, test suites, or recommended approaches for this?
All reactions