Page Layout

khocr-gen layout: page, word and table annotations

Composes corpus lines into document-disjoint pages. Logical labels reflect the configured normalization and tab-to-space cleanup. Word and vertical modes use one RAQM paragraph and rasterize its positioned FreeType glyphs.

Command

khocr-gen layout --corpus FILE [--fonts DIR] [--output DIR]
    [--page-width PX] [--page-height PX] [--margin PX]
    [--line-spacing PX] [--max-pages-per-doc N] [--seed N]
    [--width PX] [--line-height PX] [--color-mode {1,3}]
    [--columns N] [--column-gap PX] [--page-direction {ltr,rtl}]
    [--include-baseline] [--include-polygon] [--hierarchy]
    [--include-words] [--writing-mode {horizontal-tb,vertical-rl,vertical-lr}]
    [--enable-tables] [--table-rows N] [--table-columns N]
    [--table-border-width PX] [--table-padding PX] [--config FILE]
    [--page-scale FACTOR] [--page-rotation DEGREES] [--page-crop X Y W H]
    [--page-flip {none,horizontal,vertical,both}]
    [--page-scale-range MIN MAX] [--page-rotation-range MIN MAX]
    [--page-perspective TLX TLY TRX TRY BRX BRY BLX BLY]
    [--page-elastic-amplitude PX] [--page-contrast FACTOR] [--page-noise N]
    [--min-length N] [--max-length N] [--lines N]
    [--multilingual] [--language TAG] [--font-pool DIR]
    [--direction {auto,ltr,rtl}] [--shaping-language TAG]
    [normalizer flags...]

--corpus auto-detects .jsonl manifests versus plain .txt by extension. --output must be absent or an empty directory. Files, nonempty directories and symlinks (including dangling symlinks) are refused. Exit codes: 0 published export, 1 occupied/unsupported destination, 2 invalid configuration, missing inputs, export error or zero pages.

Word, vertical and table contract (v5)

--include-words adds logical whitespace tokens with stable IDs, line links, reading order, polygons, boxes and start/end offsets. Offsets count Unicode code points, zero-based and half-open, into the exported line text, after any normalization: line.text[start:end] == word.text. English Hello world gives two tokens; unspaced Khmer gives one. U+200B is not Unicode whitespace and stays inside a token. This does not perform linguistic word segmentation. Word bounds come from the same shaped glyphs and resize/pad placement as the image.

--writing-mode vertical-rl and vertical-lr use true RAQM ttb shaping. Complete top-to-bottom lines advance through right-to-left or left-to-right columns and paginate without clipping. line_height controls vertical column thickness; width sets a horizontal-mode fixed canvas width. The vertical baseline is named vertical_canvas_centerline, a heuristic rather than a glyph baseline. Word/vertical modes require shared libraqm with its linked FreeType API and RAQM-enabled Pillow; the Rust extension is optional. Install your OS libraqm package when the startup check reports a missing shared library. Covering fonts are required.

--enable-tables fills cells row-major following --page-direction. Defaults: --table-rows 3, --table-columns 3 (--table-cols alias), --table-border-width 1, --table-padding 6. Rows/columns must be positive; border/padding must be nonnegative and leave content space. Rows expand to fit; oversized content scales uniformly. Overflow starts another page; partial final rows retain empty cells. Tables and cells export stable IDs, polygons, boxes, row/column indices, spans of 1 and links to their lines. --hierarchy exports table blocks. Vertical text works in cells. Table mode supplies the grid rather than ordinary multi-column flow; source document groups remain separate.

These features select schema v5. Word, table and cell polygons share all page transforms and conservative crop validation. Mesh intersections add polygon vertices before outward AABB calculation. Failures publish no partial export. Defaults preserve earlier output. All layout options accept an explicit YAML --config FILE with CLI precedence; no recipe auto-loads. See configs/layout.yml and the complete coordinate contract.

Geometry flags

• --columns N (default 1): split content width into N columns.
• --column-gap PX (default 20): gap between columns (never outer margins).
• --page-direction {ltr,rtl} (default ltr): physical column emission order; reading order always follows logical columns.
• --include-baseline: add heuristic ink-mask baseline row to each line.
• --include-polygon: add heuristic ink-mask bounding-box polygon to each line.
• --hierarchy: add page-level blocks (grouped by column) and spans.

Fixed page transforms

• --page-scale FACTOR (default 1): positive finite scale. Rounded output dimensions determine actual x/y scale; dimensions below one pixel fail.
• --page-rotation DEGREES (default 0): finite signed rotation, visually counterclockwise when positive; expand to contain the rotated page.
• --page-crop X Y W H: optional integer crop within the expanded page.
• --page-flip {none,horizontal,vertical,both} (default none): flip raster and coordinates horizontally, vertically, or both after rotation and crop.
• --page-scale-range MIN MAX: positive finite range (min, max); samples scale uniformly per page using deterministic page-seeded pseudorandomness.
• --page-rotation-range MIN MAX: finite range (min, max); samples rotation uniformly per page using deterministic page-seeded pseudorandomness.

Order: resize → expanded rotation → crop → flip. Scale and rotation use one linear OpenCV resampling with white borders; crop is an exact slice; flip reflects raster and all polygon/baseline coordinates. Parameter sampling consumes deterministic SHAKE-256 streams keyed by page ID. Limits: 32,766 pixels per dimension and 64 million pixels for transformed canvases.

Perspective, elastic mesh and photometric degradation (v4)

• --page-perspective (eight floats: TL, TR, BR, BL x y offsets): each at most 10% of the smaller expanded page dimension; absent or all zeros is a no-op.
• --page-elastic-amplitude PX (default 0): seeded 5x5 piecewise-affine mesh, at most 10% of the smaller mesh cell; folds are rejected, never clamped.
• --page-contrast FACTOR (default 1, within 0.75 to 1) and --page-noise N (default 0, integer 0 to 8): pointwise effects applied after the crop. They never move pixels, so annotations are identical with or without them.

Order: resize, expanded rotation, perspective, support expansion, elastic mesh, integer crop, photometric degradation. Pixels use inverse mapping with linear interpolation and a white border. Annotation edges are split at every mesh edge and mapped exactly, so canvas polygons and baselines can gain vertices and box is the AABB of all of them. Spans copy the full line geometry; labels, IDs, blocks and reading order are unchanged. Any effective option selects schema v4 and transform.contract_version: 2; model is homography (full matrix) or piecewise_affine_mesh (pre_mesh_matrix plus mesh vertices and triangles, with no global matrix). interpolation is "linear" (with border) when resampled and "none" (no border) for photometric-only or crop-plus-photometric pages; support appears on perspective/mesh warps and raster.method names the exact execution path (inverse_map_remap, warp_affine, crop_slice or identity). Enabled contrast or noise add a degradation object. Separate seeded SHAKE-256 streams are keyed by seed, page id, operation and version, so noise cannot change the mesh. The crop must keep all line canvas geometry and every nonwhite warped pixel in every channel. A v4 failure (unsafe perspective, folded mesh, oversize page or crop) exits 2; on any page, including a later one, it rolls back the whole transactional export and leaves no partial output. No claim is made about recognizer accuracy or readability; see docs/PAGE_LAYOUT.md.

Nondefault scale, effective rotation or explicit crop selects schema v3. With transforms off, historical v1/v2 bytes stay unchanged. v3 coordinates use pixel edges [0,W] × [0,H], with centers (i+0.5,j+0.5). Lines gain float canvas_polygon corners and outward-rounded integer box bounds. Optional ink polygon uses pixel cell edges. Optional baseline becomes an object with kind: "ink_projection_peak" and two transformed source pixel-center endpoints; blank lines omit both heuristics. This is not a typographic baseline. Each hierarchy span copies its line's transformed box and canvas polygon; spans are one per line, not word geometry. Text, reading order, IDs, document grouping and source font metadata stay unchanged. Never sort by rotated position.

The page-level transform object records contract version, coordinate convention, the full source-to-output 3×3 matrix, source/output [W,H], ordered operations with requested parameters and actual dimensions, interpolation and border policy. Float precision is retained; only errors within 1e-9 pixels of integers snap. See the full reference for the OpenCV half-pixel conversion and field names.

A crop must preserve full line canvases (even blank padding), optional geometry, and every nonwhite pixel in every channel, including interpolation fringes. Unsafe crops exit 2. The complete export is staged in a hidden sibling directory and published with a single directory rename after all page and annotation writes close. Normal exceptions remove staging, including failures on later pages. An absent output stays absent; a preexisting empty output directory stays empty and retains its identity on failure. Content created by another process during export is preserved and prevents publication. A successful export may replace an empty destination directory. The Python API enforces the same destination rules. Operational paths are normalized without following destination symlinks, so . and x/.. work. The API result keeps the requested path spelling; after replacing its own cwd, an API caller must change to the absolute output path before using relative paths.

Fresh outputs follow normal mkdir permissions, umask, inherited group and default ACLs. Existing empty outputs retain owner/group, mode (including setgid), and readable extended attributes including POSIX ACLs on Linux; metadata is copied and verified before creating children. Metadata failures abort export. Replacement of existing directories on other platforms is rejected. Success replaces directory identity and timestamps; filesystem flags and access rules outside Linux extended attributes are not copied. Detected concurrent identity/access changes abort export; callers must serialize metadata changes through publication.

The output parent must be writable/searchable even if the empty destination itself is writable. Mountpoint/cross-device rename errors abort export without a non-atomic fallback.

Zero-page runs publish nothing and leave the destination untouched. The API returns rejection statistics; the CLI exits 2 without listing output artifacts. Created parent directories may remain after failure or zero pages. Forced termination or a system crash can leave hidden staging directories; crash recovery and power-loss durability are outside this guarantee.

Output

<output>/
  pages/page_000000.png ...
  annotations.jsonl

Annotation record (schema_version=1 at defaults, 2 for existing geometry flags, 3 for page transforms):

{
  "schema_version": 2,
  "page_id": "page_000000",
  "image": "pages/page_000000.png",
  "width": 1000,
  "height": 1400,
  "document_id": "__unlabeled__",
  "lines": [
    {
      "reading_order": 0,
      "text": "Hello World",
      "box": {"x": 40, "y": 40, "w": 215, "h": 48},
      "font": "fonts/english/KhmerOS_muol.ttf",
      "font_size": 28,
      "source_id": "...",
      "source_group": "...",
      "language": "en"
    }
  ]
}

source_id/source_group/language are omitted when absent. Boxes are axis-aligned rectangles around the clean-rendered line canvas, in pixels. baseline and polygon are present only when their flags are set. blocks and spans are present only when --hierarchy is set.

Invariants (tested)

• Boxes and geometry points lie fully inside their page and nothing is clipped; oversized lines are uniformly scaled down to column width. Lines too tall for a column are rejected, never truncated.
• Annotation text equals the rendered line text exactly.
• reading_order is contiguous and monotonic within each column, then across columns.
• Every input line is placed exactly once; nothing is dropped or duplicated.
• Identical seed + config reproduces identical annotations and page image bytes.
• Clean-only rendering; augmentation stays disabled.

Document grouping

Plain .txt lines form one synthetic document (__unlabeled__). JSONL lines group by source_group; lines without one join __unlabeled__, kept separate from named documents. Every page belongs to exactly one document; --max-pages-per-doc skips whole oversized documents and reports them.

See Page Layout for full reference.