Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Japanese tokenizer #400

Closed
eklem opened this issue Jul 26, 2017 · 1 comment
Closed

Japanese tokenizer #400

eklem opened this issue Jul 26, 2017 · 1 comment
Assignees
@eklem
Labels
FEATURE This functionality would be cool to have

Comments

@eklem
Copy link
Collaborator

eklem commented Jul 26, 2017

Create a Japanese word tokenizer so that stopword removal works.
Part of custom docproc.

Base it on TinySegmenter
@eklem eklem added the FEATURE This functionality would be cool to have label Jul 26, 2017
@eklem eklem self-assigned this Jul 26, 2017
@fergiemcdowall
Copy link
Owner

All of the tokenisation functionality now lives outside of the core library. Would be nice to have though!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@eklem eklem

Labels
FEATURE This functionality would be cool to have
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@eklem @fergiemcdowall