New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tokenize correctly in languages that don't demarcate with whitespace #35

Open

stenskjaer opened this issue Jul 2, 2018 · 0 comments

Owner

stenskjaer commented Jul 2, 2018

Examples of such languages are Sanskrit and Arabic.

There an app note can contain only a fragment of a word...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment