WordLevelTrainer not deterministic

[lucacampanella]

Hi everyone and thanks for the great work.
I'm using the library to create a word level tokenizer. Unfortunately, given the exact same data, the output given by the class WordLevelTrainer is not deterministic. I suspect this happens when the count of two words is exactly the same. Their order in the resulting json (and thus the corresponding token id) is sometimes inverted.

I don't know much Rust, but I believe it would be an easy fix at line 43 of src/models/wordlevel/trainer.rs. If understand correctly, here the hashmap (dictionary) is ordered by values, with ignored keys (_). It would be sufficient to order it by keys in case of same counts.

Reproducibility of the tokenizer given the same data can be very useful in some scenarios.
Let me know if I should look a bit into rust and create a pull request for this or if I understood the code wrong and the source non-determinism is somewhere else.

Thanks a lot for the help!
Luca

Using tokenizers==0.10.3

[n1t0]

Yes you're totally right! Do you want to take a stab at it?

[lucacampanella]

Hi @n1t0 and thanks a lot for the fast reply.
I took a stab at it and created a pull request #718

There is maybe a more concise way of doing it, let me know what you think. The code was tested only in the rust playground and not with the codebase.

[pietrolesci]

Hi @n1t0 ,

I am on 0.10.3 and I can still replicate the non-determinism issue with the following code

from tokenizers.models import WordLevel
from tokenizers.trainers import WordLevelTrainer
from tokenizers import Tokenizer
from tokenizers.pre_tokenizers import Sequence, Digits, Whitespace
from tokenizers.normalizers import BertNormalizer

unk_token = "[UNK]"
tok = Tokenizer(WordLevel(unk_token=unk_token))
trainer = WordLevelTrainer(vocab_size=3, special_tokens=[unk_token])
tok.pre_tokenizer = Sequence([Digits(), Whitespace()])
tok.normalizer = BertNormalizer()

example = "Hello, my name is Bert and I am 20 years old"
tok.train_from_iterator([example], trainer=trainer)
tok.get_vocab()

# run 1: {'[UNK]': 0, 'old': 1, 'is': 2}
# run 2: {'[UNK]': 0, 'is': 1, 'mimmo': 2}

is the fix only available on master?