v2.2.0

   🚀 Features

• Split CJK text at ideographic and fullwidth punctuation  -  by @johannschopplich (a3c45)
• Attach CJK punctuation to the clause before it  -  by @johannschopplich (e11a8)
• cli: Accept decimal --chars-per-token values  -  by @johannschopplich (963d7)

   🐞 Bug Fixes

• Check custom language configs before the built-in ones instead of replacing them  -  by @johannschopplich (bad8b)
• Keep sliceByTokens boundaries consistent and surrogate-safe  -  by @johannschopplich (26ae5)
• Merge a line break into preceding CJK punctuation like into ASCII punctuation  -  by @johannschopplich (973df)
• Keep splitByTokens overlap from repeating a segment that fills a chunk  -  by @johannschopplich (09a7f)
• cli: Reject numeric options too large to represent  -  by @johannschopplich (38e40)

   🏎 Performance

• Stop isWithinTokenLimit once the limit is exceeded  -  by @johannschopplich (73253)
• Keep the ASCII shortcut for built-in language configs when custom ones are set  -  by @johannschopplich (df18e)
• Price CJK segments by script rates without trying the built-in language configs  -  by @johannschopplich (44ed6)

    View changes on GitHub