v1.0.0 — a document forge

OCRSmith v1.0.0 — a document forge, not a line-image generator.

The v0 engine produced one image with one string. v1 produces whole pages — multi-column,
with titles, tables, figures, forms and running headers — degraded to look like something
a scanner or a phone actually produced, and emits the ground truth for every objective
those pages can supervise.

From one render, five kinds of label

Objective	What you get
Recognition	Line and word crops with logical-order text
Detection	Word, line and region boxes; polygons where the page is warped
Layout analysis	Typed regions (title, table, figure, key_value, …) in reading order
Document → markup	The page serialised back to Markdown and HTML
Table structure	Cell grid as HTML and OTSL

Getting Arabic right

Four failures that Latin-first generators make silently, and what v1 does instead:

• Logical vs visual order. The label is the logical string; the pixels are visual. The
  two are modelled separately and never conflated. Two interchangeable shaping backends
  (Raqm/HarfBuzz, or arabic-reshaper + python-bidi) produce identical labels, so a dataset
  does not depend on how Pillow was compiled.
• Missing glyphs. Font coverage is read from the cmap before a face is chosen, so a
  font that cannot draw a character is rejected instead of emitting tofu the label
  contradicts.
• Boxes that do not follow the pixels. Every geometric degradation maps the whole
  annotation through the same transform.
• Silent truncation. Wrapping is lossless; text that does not fit is reported, not
  discarded.

Scale and reproducibility

• Generators end to end — a ten-million-page run holds one page in memory.
• Sharded work, workers writing their own shards; completed shards are marked, so an
  interrupted job resumes instead of restarting.
• Every sample's seed is derived from (config.seed, index), so sample 8 412 can be
  regenerated on its own months later.

Output formats

jsonl · parquet · webdataset · coco · paddleocr · chat (vision-language
instruction pairs).

Quality and evaluation

Six validators run as a gate inside the pipeline, catching the failures that do not raise
— blank pages, washed-out text, boxes off the canvas, overlapping blocks, illegibly small
lines. A shard rejecting too many samples aborts rather than shipping a dataset full of
holes. ocrsmith stats reports the distributions the corpus actually has; ocrsmith evaluate scores a model against it with CER, WER, edit similarity and table similarity.

Getting started

pip install -e ".[data,dev]"
ocrsmith doctor
ocrsmith preview --count 3 --boxes
ocrsmith generate -n 10000 --workers 8 --format webdataset -o data/train

Docs: architecture · configuration ·
formats · example corpora

Full detail in CHANGELOG.md.