You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
Words are handled as characters instead of bytes in the parser, so multibyte text is no longer split in the middle of a character. This fixes keywords containing a stray question mark, such as «москва? (#21, #7)
The minimum word length is counted in characters instead of bytes, so short words are now filtered the same way in every language
Added tests for keyword encoding, the character based minimum length, and sentence punctuation
Latin text is unaffected, the output is unchanged for English.