Repository navigation
PageLedger 0.1.7 — Unicode-safe quality evidence for multilingual archives
Multilingual quality evidence is now Unicode-script safe.
Fixed
- Quality metrics tokenize on Unicode letters together with their combining
marks, eliminating false fragmented-text and symbol-noise warnings on clean
Devanagari, Bengali, Gujarati, Gurmukhi, Tamil, Telugu, Kannada, and
Malayalam prose. - Regression coverage now spans those Indic scripts plus Arabic, Kazakh
Cyrillic, Latin, and Russian — while the existing OCR-fragment fixtures
still warn as they should. - Per-page
word_countnow uses the same Unicode tokenizer as the lexical
quality metrics instead of disagreeing with them on combining-mark scripts.
Packaging
- All seven JSON Schema contracts ship inside built wheels under
share/pageledger/schemas; CI and the publish workflow verify wheel
contents.
Compatibility
- Artifact schema version remains
0.1; no field was added, removed,
renamed, or retyped. Correctedword_count,suspicious_symbol_count,
and lexical metrics can differ from 0.1.6 for scripts that use combining
marks or non-ASCII punctuation.
Install
pip install -U pageledger
Testing a collection we haven't seen? Open a corpus report.