Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Simplify tokenizer modules #143

Closed
jonatanklosko opened this issue Dec 30, 2022 · 0 comments · Fixed by #310
Closed

Simplify tokenizer modules #143

jonatanklosko opened this issue Dec 30, 2022 · 0 comments · Fixed by #310
Labels
kind:chore Internal improvements

Comments

@jonatanklosko
Copy link
Member

Currently each tokenizer module is the same, other than the default special tokens (see #141). Maybe we should kill the behaviour altogether, but we still need a place for the default special tokens.

This picture may change if we have a tokenizer that doesn't follow the current scheme exactly, such as whisper (#107).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind:chore Internal improvements
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Group all tokenizers undera single module and configure upfront elixir-nx/bumblebee
1 participant
@jonatanklosko