proof of concept for using dataset of test cases for tokenizer tests

[itazap]

To refactor comparing slow == fast, we need to 'freeze' current working behavior of tokenizers. We move the test strings to the hub (https://huggingface.co/datasets/hf-internal-testing/tokenization_test_data), with expected tokens, encoded_ids, and encoded_ids with special tokens. I got the test strings for all sentence piece based models and I ran all sp-based models on these test --> then uploaded to the dataset.

This allows us to

1. preserve testing expected behavior without relying on comparing fast == slow
2. reduce overwriting of tests per class
3. future tests can be added here!

*Note: will convert other slow == fast cases! this is a POC for test_rust_and_python_full_tokenizers to get some feedback :). then we can easily move tests like test_tokenization_python_rust_equals and many others.

More ideas/ considerations

• need to check if this slow downs running these tests

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ydshieh]

Hi @itazap

To refactor comparing slow == fast, we need to 'freeze' current working behavior of tokenizers.

Could you elaborate what refactorization you plan to achieve, and explain why this approach is "needed" for it?

preserve testing expected behavior without relying on comparing fast == slow

There is still test_rust_and_python_full_tokenizers in the common file. So IIRC, we don't run it with slow tokenizer, but just fast with its expected value?

reduce overwriting of tests per class

This is very nice.

future tests can be added here!

I am not in favor of putting the input and expected values on the hub repository like this. This makes working on test (write it and maintain it like debugging and update) more frictional. It also prevents the community to contribute directly, IIRC.

I am happy if these test values being kept in each test file, and the tests in tests/test_tokenization_common.py use them from there. I think this is still doable, and still achieve what you try to do.