v1.4.0

What's New

Token Counter Tool

New standalone tool at /tokens for counting tokens across multiple LLM tokenization strategies. Users select a model from a grouped dropdown (default: GPT-4o) and instantly see the token count with a side-by-side comparison table for all other tokenizers.

Supported models:

• OpenAI — GPT-4o, GPT-4, GPT-3.5-turbo (exact BPE via tiktoken with token preview)
• Anthropic — Claude 3.5 / 4 (heuristic estimate)
• Meta — Llama 3 / 3.1 / 3.2 (heuristic estimate)
• Google — Gemini 1.5 / 2 (heuristic estimate)
• Mistral AI — Mistral / Mixtral (heuristic estimate)
• Classic — Whitespace, Word, Sentence, Character

Exact BPE strategies include an expandable token preview showing how text is split into individual tokens.

HTML/Markdown Viewer Enhancements

• Added GFM (GitHub Flavored Markdown) extensions: tables, strikethrough, task lists, autolinks
• Added Mermaid diagram rendering support
• Added maximize/fullscreen preview mode

UI Improvements

• Standalone input textareas are now vertically resizable
• Wider tool layouts for better use of screen space

Changelog

• cc43cca feat: add Token Counter tool with multi-model tokenization strategies
• 1bd0a21 fix: make standalone input textareas vertically resizable and widen tool layouts
• d24c39f feat: enhance HTML/Markdown viewer with GFM extensions, mermaid support, and maximize preview
• 05206dd docs: update wiki

Full Changelog: v1.3.0...v1.4.0