v1.1.0 — closing the gaps the literature names

Six tracks of work, chosen from a survey of what the current OCR literature says synthetic data is missing — and from the limitations AtlasOCR reports about itself.

OCRSmith produced 86% of the training data behind AtlasOCR, the first open-source Moroccan Darija OCR model (~10.7M words), which beats comparable models on KITAB-Bench. v1.1 exists to close the gaps AtlasOCR names.

Fonts: 11 → 57 families, 101 → 381 faces

Font diversity is the dominant factor in the literature. ocrsmith fetch-fonts --subset arabic downloads open-licensed families on demand — only from permissively licensed directories of google/fonts, with each family's licence file beside its fonts and a manifest for reproducibility. Fonts are not committed; they are other people's work.

Variable fonts are expanded into their named instances. About half the Arabic families on Google Fonts are variable, and a variable font renders only its default instance — so light, regular and bold of such a family were all the same face.

Diacritics, without inventing them

Arabic OCR handles vocalisation badly, and it is the first limitation AtlasOCR names. The cause is distributional: real Arabic is partially diacritised and the proportion varies by genre.

DiacriticsPolicy samples per document across keep / strip / partial / mixed and records the kept fraction in provenance, so a diacritics ablation is possible later.

Marks are only ever removed, never invented. Diacritising bare text needs a diacritiser model and would make the label assert vowels nobody wrote — fabricated ground truth that looks entirely plausible.

Charts → JSON, formulas → LaTeX

Two tasks the benchmarks evaluate and no synthetic generator covered for Arabic. Both follow one principle: the data is the input and the drawing is derived from it, so the picture and its ground truth cannot drift.

• Bar, horizontal-bar, line and pie charts, emitting the series values a chart-to-JSON model predicts. Labels are real annotated text, so a chart also supervises recognition.
• A baseline-relative typesetter for fractions, powers, indices, radicals and sums/integrals with limits, emitting LaTeX from the same tree. No LaTeX toolchain.

Paper that is not flat, light that is not even

Wrinkles, PageCurl and IlluminationField. These displace pixels, so they carry the annotation through the same field — verified by measuring where the ink actually landed and requiring the boxes to be there (IoU > 0.85).

Three more genres

contents (dot leaders — runs of identical glyphs models miscount), slide (very large type, very little of it), and notes (handwriting-style setting for the handwriting-heavy Arabic benchmarks). Ten genres in total.

Handwriting here is a font plus baseline and word-gap jitter. It covers the layout and visual regime; it is not a substitute for real handwriting data, and the code says so.

The non-prose content real documents contain

Dates in three formats, amounts with currencies, reference codes, phone numbers, percentages — in whichever numeral system the config asks for, since Arabic documents mix Western and Arabic-Indic digits freely. Plus partial words and contextless runs: the cases where a recogniser has no language model to lean on.

---

589 tests. Lint and format clean on Linux and Windows, Python 3.10 and 3.12.

One practical note

Installing Pillow with Raqm raises the usable font pool from 85 to 203 faces, because most modern Arabic families join via GSUB and carry no presentation-form glyphs. ocrsmith doctor reports which backend you have.

conda install -c conda-forge pillow

v1.0.2 — coverage checked against what is drawn

Fixes a silent-corruption bug found by rendering samples and looking at them.

Font coverage was probed on the logical string while the renderer draws presentation forms on non-Raqm builds. Fonts like Fustat and Mada cover base Arabic via GSUB and carry no presentation-form glyphs, so the probe reported 100% coverage and every glyph rendered as an empty box — with the label still claiming the text.

Also fixed: the coverage probe skipped table cells and list items entirely, and the fallback path handed documents a font that could not draw their script at all.

466 tests. Non-Raqm builds only; Raqm builds were never affected.

v1.0.1 — shaping performance

Patch release from profiling the v0-vs-v1 benchmark.

Fixed

68% of generation time was a bug in arabic-reshaper 3.0.0. Its ligature-regex cache is guarded by hasattr(self, '__ligatures_re') but written to self.__ligatures_re — which Python mangles to _ArabicReshaper__ligatures_re inside the class body, while the string literal passed to hasattr is not mangled. The guard checks a name that is never set, so the regex was rebuilt on every call, re-reading ~290 configparser entries. Laying out one page called it ~1,800 times.

Mitigated inside ocrsmith.text.shaping: results are cached (shaping is a pure function of the string), and the reshaper's cache is warmed once so the library's own guard fires from the second call.

2,000 distinct strings: 26s → 0.36s.

Changed

Wrapping now measures a line by summing word advances plus space advances, which is how the renderer actually draws it. This is a correctness fix as well as a speed one — the two could previously disagree about where a line ends.

Numbers

• Test suite: 95s → 20s
• End-to-end generation: ~1.6× faster
• 460 tests pass; ligature formation and the additive measure are now pinned by tests

Full detail in CHANGELOG.md.

v1.0.0 — a document forge

OCRSmith v1.0.0 — a document forge, not a line-image generator.

The v0 engine produced one image with one string. v1 produces whole pages — multi-column,
with titles, tables, figures, forms and running headers — degraded to look like something
a scanner or a phone actually produced, and emits the ground truth for every objective
those pages can supervise.

From one render, five kinds of label

Objective	What you get
Recognition	Line and word crops with logical-order text
Detection	Word, line and region boxes; polygons where the page is warped
Layout analysis	Typed regions (title, table, figure, key_value, …) in reading order
Document → markup	The page serialised back to Markdown and HTML
Table structure	Cell grid as HTML and OTSL

Getting Arabic right

Four failures that Latin-first generators make silently, and what v1 does instead:

• Logical vs visual order. The label is the logical string; the pixels are visual. The
  two are modelled separately and never conflated. Two interchangeable shaping backends
  (Raqm/HarfBuzz, or arabic-reshaper + python-bidi) produce identical labels, so a dataset
  does not depend on how Pillow was compiled.
• Missing glyphs. Font coverage is read from the cmap before a face is chosen, so a
  font that cannot draw a character is rejected instead of emitting tofu the label
  contradicts.
• Boxes that do not follow the pixels. Every geometric degradation maps the whole
  annotation through the same transform.
• Silent truncation. Wrapping is lossless; text that does not fit is reported, not
  discarded.

Scale and reproducibility

• Generators end to end — a ten-million-page run holds one page in memory.
• Sharded work, workers writing their own shards; completed shards are marked, so an
  interrupted job resumes instead of restarting.
• Every sample's seed is derived from (config.seed, index), so sample 8 412 can be
  regenerated on its own months later.

Output formats

jsonl · parquet · webdataset · coco · paddleocr · chat (vision-language
instruction pairs).

Quality and evaluation

Six validators run as a gate inside the pipeline, catching the failures that do not raise
— blank pages, washed-out text, boxes off the canvas, overlapping blocks, illegibly small
lines. A shard rejecting too many samples aborts rather than shipping a dataset full of
holes. ocrsmith stats reports the distributions the corpus actually has; ocrsmith evaluate scores a model against it with CER, WER, edit similarity and table similarity.

Getting started

pip install -e ".[data,dev]"
ocrsmith doctor
ocrsmith preview --count 3 --boxes
ocrsmith generate -n 10000 --workers 8 --format webdataset -o data/train

Docs: architecture · configuration ·
formats · example corpora

Full detail in CHANGELOG.md.