Proposal: structured JSON output for extraction results

[muhammadhreden]

Summary

Would you be open to an additive JSON serialization of the extraction result — pages, blocks, the per-page verdict — alongside the existing text/HTML output? Raising this here before writing code, per the guidelines.

Why

Training data for Arabic document models. Layout-aware VLMs and document transformers train on pairs of page image and structured target: text spans, geometry, reading order. Public corpora of that kind are almost entirely English. qalam already solves the hard part for Arabic — logical word order, base letters, intact ligatures — which is exactly what a corpus built on a generic extractor gets wrong. The missing piece is a machine-readable output, so the structure the layout pass already computed doesn't have to be re-parsed out of HTML.

The verdict is a filtering signal. Keep ok pages as supervision, route fail to OCR, use degraded pages with their reasons as a diagnostic set. In JSON that signal travels with the page instead of being recovered out of band.

Also useful outside training. RTL tables that Markdown mirrors with no way to fix, chunking for RAG, diffing extraction output across commits.

Proposed shape

{
  "source": "file.pdf",
  "page_count": 12,
  "pages": [{
    "number": 1,
    "width": 595.0,
    "height": 842.0,
    "verdict": "ok",
    "score": 1.0,
    "reasons": [],
    "blocks": [{
      "type": "paragraph",
      "text": "...",
      "bbox": [x0, y0, x1, y1],
      "font_size": 12.0,
      "direction": "rtl"
    }]
  }]
}

[khalil-Hennara]

it's good idea, and for the mentioned structure we've already have all these field so making this would be easy task, the code should follow to_html pattern, also should be mirror the changes to the Python API. So the changies should not affect the logic of extraction only the way of presenting the output

[muhammadhreden]

Thanks....that's exactly the constraint I was hoping for. Plan, then:

Mirror the to_html implementation as to_json on the Rust side, serializing the fields that already exist. No change to the extraction path, presentation layer only.
Expose it through the bindings so the Python API has the same method, matching how to_html is surfaced there.
Tests over a small fixture covering an ok page, a degraded one, and an RTL table, plus a README section.