saphes 0.1.0

First public release.

Readability and lexical diversity — two metrics, done carefully, with the parameters other implementations hardcode.

Why it exists

The LIX long-word threshold is hardcoded at 6 everywhere. That 6 is Björnsson's Swedish original. Measured over the full Hungarian Webcorpus, 44.5% of running tokens are longer than six letters, against 25.7% in Swedish — so the index saturates and stops discriminating. On real Hungarian prose that reads as LIX 60.4, "very difficult"; at the calibrated threshold it reads 43.4. Parameterising the threshold is the whole point.

Implementations disagree, because they count words, sentences and long words differently. So every function returns A, B, C (or types and tokens) alongside the score, plus every parameter used.

Highlights

• lix() with long_word_threshold as a first-class parameter, and three ways to supply the sentence count B — segmented, pre-split, or explicit. Real pipelines often cannot supply sentences at all.
• lexical_diversity() with a required unit, plus mattr() for length-robust comparison.
• The two metrics require opposite token streams — diversity wants lemmas, readability wants surface forms. Feed one stream to both and exactly one is silently wrong. Four guards stand against that, and tests/test_contracts.py is the regression guard.
• recommended_threshold("hu") → 8, calibrated by equipercentile matching against Swedish. Six independently computed curves agree: two Hungarian corpora nineteen years apart, three sampling strata, and a digraph-aware variant. Validated end to end on 10.3M words and 709k real sentences.
• Explicit length_policy, NFC by default — decomposed Unicode would otherwise inflate every word length, hitting polytonic Greek and accented Hungarian hardest.
• The core has no runtime dependencies.

Verified against textstat 0.7.13: given the same sentence count, LIX scores match exactly. All free-running divergence is sentence segmentation.

Links

• Documentation: https://saphes.readthedocs.io
• The calibration study: experiments/lix_calibration/
• Full changelog: CHANGELOG.md

Install

uv add saphes