Exception: stream did not contain valid UTF-8

[phamdinhkhanh]

I get bug when tokenize ByteLevelBPETokenizer() for diacritic language in utf-16 such as 'Viet Nam' language. Bellow are my code initialize tokenizer.

%%time 
from pathlib import Path

from tokenizers import ByteLevelBPETokenizer

paths = ['file1.txt', 'file2.txt']
print(paths)
# Initialize a tokenizer
tokenizer = ByteLevelBPETokenizer()
# Customize training
tokenizer.train(files=paths, vocab_size=52000, min_frequency=2, special_tokens=[
    "<s>",
    "<pad>",
    "</s>",
    "<unk>",
    "<mask>",
])

And bug log:

in train(self, files, vocab_size, min_frequency, show_progress, special_tokens)
90 files = [files]
91 print('files list: \n', files)
---> 92 self._tokenizer.train(trainer, files)

Exception: stream did not contain valid UTF-8

my file1.txt and file2.txt contain words like:

xin chào tôi đến từ Việt Nam, tôi gặp vấn đề với tokenizer.

I try to find what self._tokenizer.train() does to fix it myself but project code are complicated. Can you explain what i was wrong?

[n1t0]

Hi @phamdinhkhanh. As the error is saying, we expect UTF-8 content. We don't support UTF-16.

Can you try to encode your training files in UTF-8 and let us know how it goes?

[gaceladri]

I am having the same issue with a downloaded dataset with the "datasets" repo. It's an arrow file and I am struggling a bit on how to encode it as a utf-8. Maybe a good feature request could be to integrate the option to convert it inside tokenizers?

[tillfurger]

Same here and I manually removed the non utf-8 content from the file. Any Ideas?

[BhavyaShah1234]

Put encoding='utf-8' in the script where the files are written and rerun the tokenizers script so the code will look something like:

with open(filename, 'w', encoding='utf-8') as f:
    f.write(text)
    f.close()

[HarshMishra2002]

I get bug when tokenize ByteLevelBPETokenizer() for diacritic language in utf-16 such as 'Viet Nam' language. Bellow are my code initialize tokenizer.

%%time 
from pathlib import Path

from tokenizers import ByteLevelBPETokenizer

paths = ['file1.txt', 'file2.txt']
print(paths)
# Initialize a tokenizer
tokenizer = ByteLevelBPETokenizer()
# Customize training
tokenizer.train(files=paths, vocab_size=52000, min_frequency=2, special_tokens=[
    "<s>",
    "<pad>",
    "</s>",
    "<unk>",
    "<mask>",
])

And bug log:

in train(self, files, vocab_size, min_frequency, show_progress, special_tokens)
90 files = [files]
91 print('files list: \n', files)
---> 92 self._tokenizer.train(trainer, files)
Exception: stream did not contain valid UTF-8

my file1.txt and file2.txt contain words like:

xin chào tôi đến từ Việt Nam, tôi gặp vấn đề với tokenizer.

I try to find what self._tokenizer.train() does to fix it myself but project code are complicated. Can you explain what i was wrong?

I got the same issue.. Simply open your txt files using notepad, then click on "save as"... there you get an option to choose the encoding.. select UT-8 and save the file.. This fixed my error.