sentencepiece\sentencepiece\src\sentencepiece_processor.cc(1102) [model_proto->ParseFromArray(serialized.data(), serialized.size())]

[showpiecep]

System Info

• transformers version: 4.24.0
• Platform: Windows-10-10.0.19041-SP0
• Python version: 3.9.2
• Huggingface_hub version: 0.10.1
• PyTorch version (GPU?): 1.13.0+cpu (False)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

Who can help?

@patrickvonplaten
@sgugger

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from transformers import T5Tokenizer
tokenizer = T5Tokenizer(vocab_file='vocab_ruturk.spm')

Traceback (most recent call last):
File "app.py", line 3, in
tokenizer = T5Tokenizer(vocab_file='vocab.ruturk.spm')
File "env\lib\site-packages\transformers\models\t5\tokenization_t5.py", line 157, in init
self.sp_model.Load(vocab_file)
File "env\lib\site-packages\sentencepiece_init_.py", line 910, in Load
return self.LoadFromFile(model_file)
File "env\lib\site-packages\sentencepiece_init_.py", line 311, in LoadFromFile
return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
RuntimeError: Internal: a\sentencepiece\sentencepiece\src\sentencepiece_processor.cc(1102) [model_proto->ParseFromArray(serialized.data(), serialized.size())]

Expected behavior

No errors

[sgugger]

It looks like you are using the tokenizer with a broken sentencepiece vocab. In any case, we would need a reproducer with a file we have access to to be able to investigate.

[cliangyu]

Ran into the same issue. How did you solve it?

[showpiecep]

Ran into the same issue. How did you solve it?

The whole problem was the vocab. I just took a different one.

[AlexWortega]

Whats wrong with vocab? how to change it correct?

[jiangzhuolin]

Whats wrong with vocab? how to change it correct?

make sure your vocab files(*.bin files) have been downloaded fully.in my case, I didn't install git-lfs. git clone repo from huggingface is failed for these files. download these files or use git-lfs.