Token Counter CLI v0.1.0

🎉 Token Counter CLI v0.1.0

A fast, user-friendly CLI tool for counting tokens in text files using tiktoken, with LLM context limit comparison and rich formatting.

✨ Features

• Fast tokenization using OpenAI's tiktoken library
• Rich progress bars for large files
• LLM context limit comparison with latest 2025 models from all major providers
• Beautiful color-coded output with provider grouping and readable number formatting
• Flexible file handling with wildcard discovery and recursive scanning
• Multiple encoding support with interactive selection
• Comprehensive configuration via JSON files

📦 Installation

pip install token-counter-cli

🚀 Quick Start

# Count tokens in a file
token-counter my_document.txt

# Check against LLM limits with beautiful output
token-counter my_document.txt --check-limits

# Process directory recursively with auto-discovery
token-counter my_project/ --recursive --add-extensions "*"

# Interactive encoding selection
token-counter my_file.txt --select-encoding

🌈 What's New in v0.1.0

• Enhanced color system: Green (≤80%), yellow (80-100%), red (>100%) usage indicators
• Provider grouping: LLM limits organized by OpenAI, Anthropic, Google, Meta, xAI, Mistral, Cohere
• Readable numbers: Comma separators and K/M notation (e.g., "1.2M tokens")
• Improved formatting: "122K of 1M tokens" with color-coded context limits
• Latest LLM models: Updated with 2025 models including GPT-4.1, Claude 4, Gemini 2.5, Llama 4

📚 Links

• PyPI Package: https://pypi.org/project/token-counter-cli/
• Documentation: See README.md for full usage guide
• Issues & Features: https://github.com/puya/token-counter/issues

Enjoy counting tokens! 🚀