Repository navigation
Tokenization
What is it?
In the context of GPT-J, "tokenization" refers to the process of splitting a piece of text into individual tokens that can be processed by the model. Tokenization is a crucial step in preparing text data for input into GPT-J, as the model operates on a sequence of tokens rather than raw text.
What is is the process?
The tokenization process involves breaking down a piece of text into individual words, punctuation marks, and other units of meaning, such as numbers or special characters. The tokens are then assigned numerical values, called embeddings, that represent their meaning and relationship to other tokens in the input text.
GPT-J uses a technique called "byte pair encoding" (BPE) to tokenize text. BPE is a data compression algorithm that iteratively merges the most frequent pairs of consecutive characters or character sequences in the text, creating a set of sub-word units that can be used as tokens.
The advantage of using BPE for tokenization is that it can handle rare and out-of-vocabulary words, as these words can be split into smaller sub-word units that are more likely to occur in the training data. This enables GPT-J to generate text that is more fluent and contextually appropriate.
Byte Pair Encoding Example
Suppose we have a training corpus that includes the following words: "hello", "world", "hell", "ocean", "helicopter". To create a BPE vocabulary, we can perform the following steps:
Initialize the vocabulary with a set of atomic symbols, such as the individual characters in the corpus: {h, e, l, o, w, r, d, c, n, p, t}
Iterate over the corpus, and at each iteration, find the most frequent pair of symbols in the current vocabulary, and merge them into a new symbol:
In the first iteration, the most frequent pair is "l" and "o", which occur together in "hello" and "ocean". We can merge them into a new symbol "lo", and add it to the vocabulary: {h, e, l, o, w, r, d, c, n, p, t, lo}
In the second iteration, the most frequent pair is "h" and "e", which occur together in "hello". We can merge them into a new symbol "he", and add it to the vocabulary: {he, l, o, w, r, d, c, n, p, t, lo}
In the third iteration, the most frequent pair is "he" and "l", which occur together in "hello" and "helicopter". We can merge them into a new symbol "hel", and add it to the vocabulary: {hel, o, w, r, d, c, n, p, t, lo}
In the fourth iteration, the most frequent pair is "lo" and "w", which occur together in "world". We can merge them into a new symbol "low", and add it to the vocabulary: {hel, o, w, r, d, c, n, p, t, low}
Repeat step 2 until a desired vocabulary size or a maximum number of iterations is reached.
After applying the BPE algorithm to our training corpus, we end up with a vocabulary of subword units that can be used to represent the original words in a more efficient way. For example, the word "helicopter" can be represented as "hel" + "icopter", which corresponds to two symbols in the BPE vocabulary, instead of eight characters in the original word.
What actually does the tokenization?
The tokenization is performed by the Hugging Face Tokenizers Library.
The Tokenizers library provides a fast and efficient implementation of this algorithm, which can handle large-scale tokenization tasks on modern hardware.
What programming language is the library written in?
The library is written in Rust, a high-performance systems programming language, and provides bindings for several programming languages, including Python and JavaScript.
How do I install the Hugging Face Tokenizers library?
Install the Tokenizers library by running the following command:
pip install tokenizers
This will download and install the latest version of the Tokenizers library from the Python Package Index.
Once the installation is complete, you can import the Tokenizers library in your Python code using the following statement:
from tokenizers import Tokenizer