Repository navigation
1.3.0
- Simplified the pre-tokenizer's BMP combining-mark rule. ctok now uses Unicode
\p{Alphabetic}instead of a hand-maintained table of accents, viramas, tones, and annotations. - Versioning changed after v4.8 was found to use the same frame as Sonnet 5 and Fable 5. Thanks to @wdhwg001 for identifying this in #29
- Opus 5 is slightly different at the right edge: it effectively performs an
rstrip(). This is out of scope for now.
Contributors
wdhwg001