v0.8.0 — token report against the embedding window

Feature release. A model embeds only the first window tokens of a text; the rest can be stored by a caller but is invisible to vector search. How many characters that is depends on the text — for one token window it varies by more than a factor of two — so a caller splitting long records cannot derive it. 0.8.0 lets the server report it. Nothing changes for a caller who does not ask.

The report

POST /embed          { "texts": [...], "token_info": true }   # adds "token_info" to the usual response
POST /count_tokens   { "texts": [...] }                       # the same report, without running the model
Entry:               { "count": <int>, "window": <int>, "truncated": <bool>, "window_end_char": <int> }

• count and window both include the tokenizer's special tokens.
• window_end_char is a character offset: text[:window_end_char] is exactly what the vector represents, and it equals len(text) when nothing was cut.
• /count_tokens has the same validation, 100-text batch limit and bearer middleware as /embed.
• api_openai and mlx_bge_m3 cannot see their own tokens and report null. Read it as unknown, never as "fits".

Verified end to end on jina-v5-nano: embedding text[:window_end_char] gives the same vector as the whole text (cosine 1.0), while cutting 8 characters earlier does not (0.97).

Cost

jina-v5-nano, 8 texts per call	counting as a share of one /embed call
300 characters	0.4%
3,000 characters	0.5%
16,000 characters	1.6%

Counting is paid only when requested. It uses a separate tokenizer instance with padding and truncation off, loaded on the first request that needs it: about 29 MiB resident for jina-v5-nano, and nothing for a server that is never asked.

Compatibility

Additive. Without "token_info": true, /embed returns the same response as in 0.7.0, and no existing route changed. A client should treat a missing /count_tokens (404 from an older server) as unknown.

A bug found before release and fixed in it: jina-v5-nano's tokenizer.json enables batch padding, which made every text in a mixed batch report the longest member's length. The counting tokenizer clears padding and truncation as loaded.

24 regression tests; CI now installs tokenizers with the dev extra so they run there instead of skipping.