Fix AddedToken duplicate 'special' kwarg for extra_special_tokens#44313
Closed
jashshah999 wants to merge 1 commit into
Closed
Fix AddedToken duplicate 'special' kwarg for extra_special_tokens#44313jashshah999 wants to merge 1 commit into
jashshah999 wants to merge 1 commit into
Conversation
Commit 8e663c7 (huggingface#44281) fixed the duplicate keyword argument error for regular special tokens by popping the 'special' key before passing it to AddedToken(..., special=True). However, the same pattern in the extra_special_tokens loop was not fixed, meaning tokenizers with extra_special_tokens dicts containing a 'special' key would still crash with: TypeError: AddedToken() got multiple values for keyword argument 'special'
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the same
TypeError: AddedToken() got multiple values for keyword argument 'special'that #44281 addressed, but for theextra_special_tokenscode path which was missed.#44281 (commit 8e663c7) correctly added
value.pop("special", None)beforeAddedToken(**value, special=True)for regular special tokens loaded fromspecial_tokens_map.json. However, the same pattern in theextra_special_tokensloop on the next line was not updated, so tokenizers whoseextra_special_tokensdicts contain a"special"key still crash.Related: #44062