How to add extra tokens in tiktoken?

[Stanislas0]

Great work! I want to know how can we add customizable extra tokens to tiktoken. Thank you!

[hauntsaninja]

You can create your own Encoding object by passing in a dict from token-bytes to the integer token value.
See this:

tiktoken/tiktoken/core.py

Line 10 in 4226a6c

class Encoding:

You can see examples of the arguments passed to the constructor here:

tiktoken/tiktoken_ext/openai_public.py

Line 15 in 4226a6c

return {

[hauntsaninja]

If you want to have your own encoding registered so that tiktoken.get_encoding(...) works, you can create a new submodule under the tiktoken_ext namespace. This ability to extend tiktoken's registry is currently undocumented. See

tiktoken/tiktoken/registry.py

Line 22 in 4226a6c

# tiktoken_ext is a namespace package

[Stanislas0]

Thank you so much for the swift reply! I'll have a try.

[Ontopic]

Sorry for "hijacking", this isssue, but feel it is related enough, instead of opening a separate issue.

When looking at https://huggingface.co/docs/tokenizers/api/added-tokens, what would be the best way, to (as much as possible) stay in-sync with the original tokenizer configs), when adding special tokens in tiktoken?

As to match, rstrip, lstrip and normalized, for example. Understandable if not all is possible (yet) in tiktoken, but any further recommendations, as to what is possible currently and how, move current GPT2 based encoders, with added special tokens, to tiktoken, would be appreciated!

[hauntsaninja]

I've added more documentation for this over here: https://github.com/openai/tiktoken#extending-tiktoken