Skip to content

PageLedger 0.1.7 — Unicode-safe quality evidence for multilingual archives

Choose a tag to compare

@peterbussch peterbussch released this 13 Jul 14:48
· 398 commits to main since this release

Multilingual quality evidence is now Unicode-script safe.

Fixed

  • Quality metrics tokenize on Unicode letters together with their combining
    marks, eliminating false fragmented-text and symbol-noise warnings on clean
    Devanagari, Bengali, Gujarati, Gurmukhi, Tamil, Telugu, Kannada, and
    Malayalam prose.
  • Regression coverage now spans those Indic scripts plus Arabic, Kazakh
    Cyrillic, Latin, and Russian — while the existing OCR-fragment fixtures
    still warn as they should.
  • Per-page word_count now uses the same Unicode tokenizer as the lexical
    quality metrics instead of disagreeing with them on combining-mark scripts.

Packaging

  • All seven JSON Schema contracts ship inside built wheels under
    share/pageledger/schemas; CI and the publish workflow verify wheel
    contents.

Compatibility

  • Artifact schema version remains 0.1; no field was added, removed,
    renamed, or retyped. Corrected word_count, suspicious_symbol_count,
    and lexical metrics can differ from 0.1.6 for scripts that use combining
    marks or non-ASCII punctuation.

Install

pip install -U pageledger

Testing a collection we haven't seen? Open a corpus report.