Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add default model_max_length for HF Tokenizer #2501

Closed
whoisjones opened this issue Nov 3, 2021 · 0 comments
Closed

add default model_max_length for HF Tokenizer #2501

whoisjones opened this issue Nov 3, 2021 · 0 comments
Assignees
@whoisjones
Labels
enhancement Improving of an existing feature

Comments

@whoisjones
Copy link
Member

Due to the increasing amount of issue for HF Tokenizer where model_max_length is not set, add a default processing for it.

Related issues: gh-2210, gh-2476, gh-2305, gh-2500
@whoisjones whoisjones added the enhancement Improving of an existing feature label Nov 3, 2021
@whoisjones whoisjones self-assigned this Nov 3, 2021
@whoisjones whoisjones mentioned this issue Nov 3, 2021
Merged
alanakbik added a commit that referenced this issue Nov 3, 2021
@alanakbik
Merge pull request #2502 from flairNLP/GH-2501
a5805f4 
default handling when Tokenizer's model_max_length is not set
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@whoisjones whoisjones

Labels
enhancement Improving of an existing feature
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@whoisjones