fix: handle dict vocab in CamembertTokenizer for tokenizer.json (#44488)

[aayushbaluni]

Summary

Fixes #44488

CamembertTokenizer raised ValueError: too many values to unpack (expected 2) when loading models like cjvt/sleng-bert that provide vocab as a dict {token: id} from tokenizer.json (BPE format). The tokenizer expected a list of (token, score) tuples for Unigram.

Root cause

When AutoTokenizer.from_pretrained loads a model with tokenizer.json, convert_to_native_format passes vocab as a dict. CamembertTokenizer assumed list format and unpacked (tok, _) = token_string, causing the error.

Fix

Handle dict vocab by converting to list of (token, 0.0) tuples in id order before passing to Unigram.

Testing

• Added test_camembert_tokenizer_with_dict_vocab in test_tokenization_camembert.py
• Manually verified CamembertTokenizer(vocab=dict_from_sleng_bert) loads successfully

Made with Cursor

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: camembert

[Rocketknight1]

No drive-by code agent PRs!