Revert "Bump tokenizers (huggingface#8419)"

This reverts commit a8a1473.
fabiocapsouza · Nov 15, 2020 · 027d1f6 · 027d1f6
1 parent c4cc1c2
commit 027d1f6
Showing 1 changed file with 1 addition and 1 deletion.
diff --git a/setup.py b/setup.py
@@ -129,7 +129,7 @@
     packages=find_packages("src"),
     install_requires=[
         "numpy",
-        "tokenizers == 0.9.3",
+        "tokenizers == 0.9.2",
         # dataclasses for Python versions that don't have it
         "dataclasses;python_version<'3.7'",
         # utilities from PyPA to e.g. compare versions