tokenize result is empty

[luluxing3]

I try to use provided tokenizer, but got empty result.
The tokenizers version is 0.9.0.
What's wrong with it? Thanks.

[n1t0]

Hi @luluxing3.

The SentencePieceBPETokenizer expects both a vocab (a mapping token->id in the format Dict[str, int]), as well as a list of merges (A list of tuples of string. Each tuple contains two tokens that should be merged together) (Cf the python file here)

[luluxing3]

How to get the merges?

I use hugging face API add_tokens and save the new XLM-Roberta tokenizer.

The new tokenizer contain those files.

I want to use the tokenizers repo to make it more fast. How can I get it? Thanks.

[Narsil]

The code is a bit tricky as XLMRoberta did use some various odd tokens (they are two 'unks' in the vocabulary for instance).

However, there is a XLMRobertaTokenizerFast right now in transformers from source.
So

AutoTokenizer.from_pretrained('xlm-roberta-base', use_fast=True)

works

If you are unable to use transformers from source, you can check out the code we use to do the conversion: https://github.com/huggingface/transformers/blob/7968051aba33ebe47668ef0d8ff2f8888dcf94bd/src/transformers/convert_slow_tokenizer.py#L434

Using those might be the best bet, as transformers added quite a lot of testing to make sure they behave in the same way as the slow tokenizers.

[n1t0]

Closing this, feel free to reopen if needed.