Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Revise text tokenization to better support CJK languages #30

Closed
titusz opened this issue Jan 27, 2019 · 0 comments
Closed

Revise text tokenization to better support CJK languages #30

titusz opened this issue Jan 27, 2019 · 0 comments
Labels
Affects: Code Affects: Spec Priority: High Scope: Medium Type: Feature
Milestone
Version 1.1

Comments

@titusz
Copy link
Member

titusz commented Jan 27, 2019

Text tokenization should be designed to be simple and generic while also supporting CJK languages.
It must yield appropriate results with similarity encoding independent of language and character set. Tokenization should not assume that text can be extracted without word boundary and separation issues.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Affects: Code Affects: Spec Priority: High Scope: Medium Type: Feature
Projects
None yet
Milestone
Version 1.1
Development

No branches or pull requests
1 participant
@titusz