Skip to content

0.1.0-dev.2

@dedickinson dedickinson tagged this 06 Jul 21:27
Stemmer previously wired up only English, throwing ArgumentError for
every other locale, even though the underlying snowball_stemmer
dependency already implements 28 languages. Extend the factory to map
all of them (excluding the generic `porter` variant, not a distinct
language). Each new mapping was verified against the actual stemmer
output for a representative inflected word per language, not just
type-checked.

Also bump the betto_icu dependency to 0.1.0-dev.2 and re-export its
new OffsetTokenizer/TokenSpan types, needed by kmdb's vault chunker
for position-aware tokenisation.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Assets 2
Loading