🔪 Tokenize text on the fly on Colab.
Tokenization is the task of splitting a text into meaningful segments, called tokens. This repository contains python notebooks to run some text tokenizers for quick experimentation purposes. Just click on one of the links in the list below and run the notebook.
Do you believe that this is useful?
Has it saved you time?
Or maybe you simply like it?
If so, support this work with a Star ⭐️.
- Hugging Face's Transformers Library Tokenizers -
- Explosion AI spaCy Library Tokenizers -
- Simone Primarosa - simonepri
See also the list of contributors who participated in this project.
This project is licensed under the MIT License - see the license file for details.