Alternative to globals() #8667
Alternative to globals() #8667
Conversation
| (RobertaConfig, (BertweetTokenizer, None)), | ||
| (RobertaConfig, (PhobertTokenizer, None)), |
There was a problem hiding this comment.
Those were not in the values() of this dict since they are replaced in the next line.
There was a problem hiding this comment.
Hmm good point.
I feel like in the end this issues comes from the fact that we should have a tokenizer config instead of piggy-backing on the model config. What. do you think @LysandreJik @julien-c ?
There was a problem hiding this comment.
Yes, we should. This is an issue I'll try to solve next week.
| @@ -195,7 +193,6 @@ | |||
| (LayoutLMConfig, (LayoutLMTokenizer, LayoutLMTokenizerFast)), | |||
| (DPRConfig, (DPRQuestionEncoderTokenizer, DPRQuestionEncoderTokenizerFast)), | |||
| (SqueezeBertConfig, (SqueezeBertTokenizer, SqueezeBertTokenizerFast)), | |||
| (BertConfig, (HerbertTokenizer, HerbertTokenizerFast)), | |||
There was a problem hiding this comment.
Those were not in the values() of this dict since they are replaced in the next line.
| SLOW_TOKENIZER_MAPPING = { | ||
| k: (v[0] if v[0] is not None else v[1]) | ||
| for k, v in TOKENIZER_MAPPING.items() | ||
| if (v[0] is not None or v[1] is not None) | ||
| } | ||
| SLOW_TOKENIZER_MAPPING[HerbertTokenizer] = HerbertTokenizerFast |
There was a problem hiding this comment.
This was not in TOKENIZER_MAPPING.items() so not set.
There was a problem hiding this comment.
As seen offline, probably unnecessary.
| SLOW_TOKENIZER_MAPPING[HerbertTokenizer] = HerbertTokenizerFast | ||
|
|
||
|
|
||
| def tokenizer_class_from_name(class_name: str): |
There was a problem hiding this comment.
This function can also be transformed to a dict if we prefer.
| SLOW_TOKENIZER_MAPPING = { | ||
| k: (v[0] if v[0] is not None else v[1]) | ||
| for k, v in TOKENIZER_MAPPING.items() | ||
| if (v[0] is not None or v[1] is not None) | ||
| } | ||
| SLOW_TOKENIZER_MAPPING[HerbertTokenizer] = HerbertTokenizerFast |
There was a problem hiding this comment.
As seen offline, probably unnecessary.
* Make ci fail * Try to make tests actually run? * CI finally failing? * Fix CI * Revert "Fix CI" This reverts commit ca7923b. * Ooops wrong one * one more try * Ok ok let's move this elsewhere * Alternative to globals() (#8667) * Alternative to globals() * Error is raised later so return None * Sentencepiece not installed make some tokenizers None * Apply Lysandre wisdom * Slightly clearer comment? cc @sgugger Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
What does this PR do?
This PR does two things:
globals