fast tok update #13036
fast tok update #13036
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
@sayakpaul here is the fix around the |
| tokenizer._update_trie() | ||
| # set correct total vocab size after removing tokens | ||
| tokenizer._update_total_vocab_size() | ||
| # Fast tokenizers: serialize, filter tokens, reload |
There was a problem hiding this comment.
Would it work with transformers (< v5) as well?
If not, maybe we could keep maintaining two code paths? One for v5 and another one for < v5? This way, in the next release cycle, we can pin transformers ver to >=5.0.0.
WDYT?
There was a problem hiding this comment.
sounds good! I added it back
sayakpaul
left a comment
sayakpaul
left a comment
There was a problem hiding this comment.
Thanks for this work! I left one comment regarding versioning. LMK what you think.
|
@bot /style |
|
Style bot fixed some files and pushed the changes. |
4 participants
Following transformers v5, we no longer have "slow" tokenizers that use a Trie - by default we use fast tokenizers. This script assumes always slow, so it is updated to work with fast!