This project implements token calculation for OpenAI's gpt-4 and gpt-3.5-turbo model, specifically using `cl100k_base` encoding.
-
Updated
Jun 24, 2024 - C#
This project implements token calculation for OpenAI's gpt-4 and gpt-3.5-turbo model, specifically using `cl100k_base` encoding.
Add a description, image, and links to the tiktoken topic page so that developers can more easily learn about it.
To associate your repository with the tiktoken topic, visit your repo's landing page and select "manage topics."