GPT Tokenizers for C#
These two classes can be use to tokenize (and count) tokens for OpenAI's GPT3, GPT4, and GPT 3.5 Turbo Models
The cl100k class also counts tokens for the ADA 002 embedding model
This code has been extracted from a larger project.
You will need to change the hard coded path to the tiktoken files.
More details about how tokens work can be found here : https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb
For GPT3 models like text-davinici-003, use pk50 as the tokenizer
For GPT 3.5, GPT 4, and text-embedding-ada-002, use bl100k as the tokenizer