fix(codex): restore o200k token estimates

[mulfyx]

Summary

• restore o200k_base text tokenization for Codex /v1/messages/count_tokens
• keep image, encrypted-reasoning, and protocol-framing costs as explicit local estimates
• use the same tokenizer for standalone-search truncation so its token budget remains valid
• document which provider counters are tokenizer-backed and which remain heuristic

Problem

The Rust implementation currently treats each uninterrupted run of letters, digits, _, or - as one token. Long identifiers, CJK text without whitespace, and base64-like text can therefore collapse to a single estimated token. The endpoint then severely under-reports request size.

This is also a regression from the proxy's original TypeScript implementation, which encoded text with gpt-tokenizer/model/gpt-4o. The current documentation continued to describe the Codex path as o200k_base-backed even though the Rust implementation had become a word-run heuristic.

Measured end to end against Codex terminal usage for the same translated requests:

Fixture	Previous /count_tokens	Actual input tokens	Error
English prose	976	978	-0.2%
Long identifier	17	531	-96.8% (31.2x undercount)
Minified code	2,224	1,649	+34.9%
CJK text	17	2,705	-99.4% (159.1x undercount)
Base64-like text	17	2,257	-99.2% (132.8x undercount)

With this change:

Fixture	New /count_tokens	Actual input tokens	Error
English prose	984	978	+0.6%
Long identifier	537	531	+1.1%
Minified code	1,655	1,649	+0.4%
CJK text	2,711	2,705	+0.2%
Base64-like text	2,263	2,257	+0.3%

The remaining six-token offset is the proxy's protocol-framing estimate. /count_tokens remains a local preflight estimate, not an upstream billing total.

Implementation

• add tiktoken-rs with default features disabled
• count ordinary Codex text with the shared o200k_base tokenizer
• preserve the existing explicit estimates for images, encrypted reasoning, and request framing
• truncate standalone-search content by encoded-token prefixes while backing up to a valid UTF-8 boundary
• add regression coverage for long identifiers, CJK, base64-like text, minified code, and UTF-8-safe token-budget truncation
• correct the HTTP API and architecture documentation

Why this is a bug

• The current Rust counter collapses whole alphanumeric runs into one token.
• The original proxy implementation used the GPT-4o tokenizer instead of this heuristic.
• The existing HTTP API documentation already promises an o200k_base estimate for the Codex path.
• Anthropic defines POST /v1/messages/count_tokens as a preflight estimate over the message input, including structured content such as tools and images, and notes that the estimate can differ slightly from final usage in its token-counting guide.
• OpenAI's tokenizer mapping assigns the gpt-5 family to o200k_base. The OpenAI Cookbook token-counting example also lists tiktoken-rs among Rust implementations.

Validation

• cargo fmt --all -- --check
• cargo check --all-targets
• cargo test --all-targets
• cargo clippy --all-targets -- -D warnings
• live /count_tokens calls and matching terminal Codex requests across all fixtures above
• verified that /count_tokens remains local and does not make an upstream request

[raine]

Seems to make sense. Thanks again