OpenVINO Tokenizers is an OpenVINO extension and a Python library designed to streamline tokenizer conversion for seamless integration into your projects. It supports Python and C++ environments and is compatible with all major platforms: Linux, Windows, and MacOS.
The tutorial consists of the following steps:
- Explain the basics of tokenization
- Install OpenVINO Tokenizers
- Convert tokenizer from HuggingFace Hub using CLI and Python API
- Create a Text Generation pipeline with OpenVINO tokenizer and detokenizer
- Combine an OpenVINO tokenizer with a classification model
This is a self-contained example that relies solely on its own code.
We recommend running the notebook in a virtual environment. You only need a Jupyter server to start.
For details, please refer to Installation Guide.
