Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove SpaCy/NLTK as an optional dependency by creating our own tokenizer for a number of languages #2245

Open
NoahSchiro opened this issue Mar 24, 2024 · 0 comments

Comments

@NoahSchiro
Copy link

🚀 Feature

Currently, when we call "get_tokenizer()" for any language other than English, we must rely on third party libraries such as SpaCy or NLTK to get these tokenizers. The devs current stance on this is that these are "optional dependencies." They are not officially dependencies of torchtext but are if users want to use this feature.

Motivation

Having these libraries optionally seems like a half-way solution to the problem. Especially for something as important as tokenization! Seeing as how we don't need the whole libraries, and just the tokenization bits, it makes sense that we just implement these ourselves.

Pitch

I propose one of two solutions; either we include these libraries as dependencies, which I believe unnecessarily bloats this tool and seems to not be the wish of the developers, or we implement our own tokenizers for a set of supported languages.

I would be happy to work on this.

Additional context

There has been previous discussion on this in issue #178

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant