Corpus and Normalization

Corpus and Normalization

The corpus is the source of every label in your dataset. It is a UTF-8 file with one string per line, either a plain .txt file or a JSONL .jsonl manifest.

JSONL corpus manifests

A .jsonl corpus is auto-detected by its file extension. Each line is a JSON object with a required "text" key and optional provenance attributes:

Key	Meaning
"text"	The line of text to render (required, string)
"source_id"	Identifier for the source line/document
"source_group"	Document/source group (alias "doc_id"); groups lines into the same split
"language"	Language or BCP-47 language tag
"script"	ISO 15924 script code (e.g. Khmr, Latn, Arab)
"domain"	Content domain (e.g. news, letters, forms, books)
"source_uri"	Source origin URL or repository identifier
"license"	License or rights statement
"retrieved_at"	Retrieval timestamp or date (YYYY-MM-DD)
"source_sha256"	SHA-256 digest of the source document
"normalization_policy"	Declared normalization policy (e.g. nfc, legacy-khmer)
"consent_or_restrictions"	Usage consent or restriction terms
"raw_text"	Original unnormalized text before any pipeline normalization

{"text": "សួស្តី អគ្គនានុវត្ត", "source_id": "col-a/doc-17/line-3", "source_group": "col-a/doc-17", "language": "km", "script": "Khmr", "domain": "government", "source_uri": "https://example.org/doc-17", "license": "CC-BY-4.0", "retrieved_at": "2026-01-15", "source_sha256": "e3b0c442...", "normalization_policy": "nfc", "consent_or_restrictions": "open"}
{"text": "Hello World"}

The same normalization, length filters, and U+2028/U+2029 line-separator rejection apply to the "text" field of each record.

Plain .txt files continue to work exactly as before and yield plain strings.

Provenance sidecars and additive metadata

When --record-metadata is enabled with a JSONL corpus:

• provenance.jsonl is written per split directory (train, val, test). Each entry maps the generated image filename to all manifest attributes: source_id, source_group, language, script, domain, source_uri, license, retrieved_at, source_sha256, normalization_policy, consent_or_restrictions, and original_text. Plain .txt corpora and image-dir mode emit no provenance.jsonl.
• metadata.jsonl includes additive fields: font_sha256 (SHA-256 hash of the selected font file), used_fonts (hashes and embedded license claims for every face used in the line), runtime_version (package version of the generator), and original_text (populated whenever NFC normalization produces a different string than raw input).
• run_manifest.jsonl appends at the dataset root and each split directory, capturing canonical recipe SHA-256, runtime dependency versions, available/used font inventories, observed ingestion counts and generation outcomes.

Skip tracking

Corpus ingestion tracks skipped lines across categories in load_corpus_with_stats and count_corpus:

• too_short: lines falling below --min-length.
• too_long: lines exceeding --max-length.
• blank / empty: lines consisting solely of whitespace or empty string values.
• separator: lines containing Unicode line or paragraph separators (U+2028 / U+2029).
• passing: lines accepted for dataset generation.

Corpus and split grouping

Splitting uses --seed (default 42). JSONL lines with source_group are grouped so a declared document/source stays in one split. Without it, identical normalized text is grouped across splits; this does not rule out related documents or near-duplicates. Use heldout real documents for OCR evaluation instead of treating generated test images as a real-image benchmark.

សួស្តី
Hello World
ស្វាគមន៍ Welcome

Mixed Khmer/English lines are the point of the tool. Include them liberally.

khocr-gen generate --corpus corpus/corpus.txt --fonts fonts --output data

--corpus defaults to corpus/corpus.txt.

Filtering

Flag	Default	Meaning
--min-length N	1	Drop lines shorter than N characters
--max-length N	260	Drop lines longer than N characters
--lines N	0	Use at most N lines (0 = all)

Preview before you commit

--count-only prints filter statistics and the estimated image count, then exits without generating anything:

khocr-gen generate --corpus corpus/corpus.txt --count-only

This is the cheapest way to sanity-check length filters and --copies before a long run.

Train / val / test split ratios

Situation	Result
Neither --val-percent nor --test-percent given	80% train / 10% val / 10% test
Only --val-percent 15	85% train / 15% val / 0% test
Only --test-percent 15	85% train / 0% val / 15% test
--split-ratios 70 15 15	explicit ratios, normalised to sum to 1.0
--split-ratios 100 0 0 or both percents 0	no val/test split
--test-file FILE	FILE is the sole source of the test split

--split-ratios overrides --val-percent/--test-percent when both are supplied.

# 80/10/10 (default)
khocr-gen generate --corpus corpus/corpus.txt --fonts fonts

# 85/15/0 — no test split
khocr-gen generate --corpus corpus/corpus.txt --fonts fonts --val-percent 15

# Explicit ratios
khocr-gen generate --corpus corpus/corpus.txt --fonts fonts --split-ratios 70 15 15

# Fixed held-out test corpus, carved from a file rather than by ratio
khocr-gen generate --corpus corpus/train.txt --test-file corpus/test.txt --fonts fonts

With fixed input bytes, fonts, configuration and software/runtime, --seed 42 controls both splitting and per-sample rendering, decorations, height sampling and augmentation. Each sample gets its own seed, so images, labels and metadata remain byte-identical when worker count or chunk scheduling changes. Seeded augmentation uses Python methods instead of native Rust; --seed none disables per-sample seeding. Different dependency versions or font files may still change output. See Variable Line Height and Evaluation for the separate offline OCR benchmark.

Rare-character oversampling

Khmer (and mixed) corpora often have a long tail: a handful of characters appear a few times while others dominate. Undersampling the tail hurts recognition of exactly the characters that are hardest to learn.

--oversample-rare-chars renders extra copies of training lines that contain rare characters:

Flag	Default	Meaning
--oversample-rare-chars	off	Enable tail oversampling
--rare-char-percentile PCT	5.0	Least-frequent PCT% of distinct characters count as "rare"
--rare-char-multiplier F	3.0	Copy multiplier applied to training lines containing a rare character

khocr-gen generate --corpus corpus/corpus.txt --fonts fonts \
  --oversample-rare-chars --rare-char-percentile 20 --rare-char-multiplier 3.0

val and test are never oversampled: they stay at one copy per line, so your evaluation split continues to reflect the natural corpus distribution. Oversampling affects train only.

Explicit multilingual policies and references

--norm-policy legacy-khmer retains the existing cleanup below. The new --norm-policy nfc applies only Unicode NFC; none applies no normalization. CLI --multilingual defaults to NFC, while legacy mode keeps legacy-khmer. NFC/none preserve accents, width, URLs, joiners and bidi controls; legacy cleanup knobs apply only to legacy-khmer. --norm-passthrough still overrides all policies. Count-only and parallel counts use the chosen policy, and normalizer options survive worker serialization.

External validation/test files deliberately keep the legacy pre-normalized reference contract: strip outer whitespace and sanitize tabs, without applying the corpus normalizer or length filter. Prepare/version them explicitly. The ordinary corpus reader filters complete lines and never truncates them; text splitting alone does not establish source/document-disjoint evaluation or check overlap with separately supplied files. See the multilingual guide for the allowed control policy and staged capabilities, and the provenance roadmap for source IDs, rights and real-document evaluation.

Text normalization

Khmer text needs canonical character ordering to render and label consistently, and real-world corpora carry encoding detritus (curly quotes, stray emoji, URLs, zero-width spaces). Normalization runs on the corpus before rendering.

Flag	Default	Effect
--norm-unicode-norm	NFKC	Unicode normalization form
--norm-emoji-replacement STR	""	Replacement string for emoji
--norm-url-replacement STR	""	Replacement string for URLs
--norm-no-remove-zwsp	off	Keep zero-width spaces
--norm-no-fix-encoding	off	Disable ftfy encoding fixes
--norm-no-uncurl-quotes	off	Disable quote uncurling
--norm-no-fix-line-breaks	off	Disable line-break normalization
--norm-passthrough	off	Skip all normalization

Notes:

• Normalization is provided by khmernormalizer (MIT), which normalizes Khmer internally.
• --norm-passthrough is the nuclear option: nothing is normalized. Use it only when your corpus is already canonical and you want byte-exact labels.
• The norm_no_* flags are negative switches: they disable an individual fix. Spelling matters.

Using pre-existing images (--image-dir)

--image-dir DIR bypasses text rendering entirely and uses images you already have. In this mode the "corpus" must be a labels.txt file (the same path<TAB>label format the generator writes), since labels are read from it rather than rendered.

This is how you fold an existing dataset into the same pipeline, LMDB packing and combine flow. See Output and Storage for the label format.

Practical advice

• Keep lines realistic. Line lengths in the corpus should match what your OCR model will see at inference; the default --max-length 260 is a guard rail, not a target.
• Check the tail. Run --count-only and look at the statistics before a long run.
• Use --test-file for benchmarks. A frozen, hand-curated test corpus makes model comparisons meaningful across runs, unlike a ratio-split that moves as the corpus grows.
• Split before you augment. Splits are disjoint at the text level, so an augmented variant of a training line can never leak into val/test.

See also

• CLI Reference: the Corpus flag tables
• Output and Storage: labels.txt, vocab.json, metadata.jsonl
• Configuration: the same settings as YAML keys