v1.3

1.3.0

• Simplified the pre-tokenizer's BMP combining-mark rule. ctok now uses Unicode \p{Alphabetic} instead of a hand-maintained table of accents, viramas, tones, and annotations.
• Versioning changed after v4.8 was found to use the same frame as Sonnet 5 and Fable 5. Thanks to @wdhwg001 for identifying this in #29
• Opus 5 is slightly different at the right edge: it effectively performs an rstrip(). This is out of scope for now.

v1.2

Vocabulary: 48,792 v3 pieces, 15,283 v4.7 pieces.

v1.1

Vocabulary: 48,787 v3 pieces, 15,280 v4.7 pieces.

Since v1.0: +142 v3 pieces; +41 v4.7 pieces and one dead marker dropped, for +40 net.

ctok v1.0

Vocabulary: 48,645 v3 pieces, 15,240 v4.7 pieces.