Tiktoken Demo

This is a simple project that demostrates how to use tiktoken library in action for encoding and decoding text using OpenAI's tokenizer (e.g, gpt-3.5, gpt-4o, etc.)

This repo uses uv as package manager.

🔧 Features

• Uses model-specific tokenizer
• Encode any string into OpenAI's tokens
• Decode tokens back in readable text format

💻 Requirements

• Git
• Python
• UV Package manager

🚀 Setup with UV

1. Clone the repository

git clone https://github.com/aayushgupta-dev/tokenizer.git
cd tokenizer

2. Install dependencies

uv add --script .\tokenizer.py "tiktoken"

3. Run the code using UV

uv run .\tokenizer.py

🧪 Sample output

Text:  Hello, how are you today??
Encoded text:  [13225, 11, 1495, 553, 481, 4044, 6961]
Decoded text:  Hello, how are you today??