sidekick v0.14.0

sidekick v0.14.0 adds /tokenize and /detokenize, so a client can check that a text fits a model before embedding it. It lets a client turn off the prompt sidekick puts before each embedding input (input_type: "none"), and lists those prompts in /v1/models. It also publishes each model's benchmark scores (MTEB and model-card results) beside its fidelity grades. Nothing an existing request does has changed.

Upgrading

• HTTP API: additions only.
  • POST /tokenize and POST /detokenize, in vLLM's shape (D44).
  • input_type: "none" on /v1/embeddings, /v2/embed and /tokenize.
  • prompts in /v1/models for each embedder.
  • Every existing request gets the same status, message and vector as in 0.13.0. The one difference is the error text for an unsupported input_type, which now mentions none.
• C ABI: additions only. sk_embed accepts purpose 2 (no prompt), and sk_model_info reports prompts. SK_ABI_VERSION is unchanged. A library that reports prompts accepts purpose 2, and an older one refuses it when sk_embed is called, so check sk_model_info first.
• Rust API: EmbedPurpose gains a Raw variant. Code that matches every variant explicitly needs an arm for it. EmbedInput::tokenize, EmbedInput::check_whole, InputBuilder::tokenize and InputBuilder::check_text are new.
• Config keys and manifests: unchanged.
• Memory: an embedder's backend and /tokenize share one copy of the model's tokenizer.
• The release archives are the same four as 0.13.0, with the same bundled ONNX Runtime versions.

Prompts, and turning them off (D9, amended)

Most supported embedders were trained with a prompt before their input, such as e5's query: and passage: or EmbeddingGemma's title: none | text: . OpenAI's API has no field for one, so sidekick adds the model's own retrieval prompts from its manifest: the query prompt for input_type: "query", the document prompt otherwise, including when input_type is absent. The README said little of this, and nothing turned it off, so a client that sent its own prompt got two. On EmbeddingGemma, MTEB's classification, clustering and similarity prompts sent that way gave vectors at cosine 0.906–0.926 to the model's.

• input_type: "none" embeds the text exactly as sent, keeping the tokenizer's special tokens. Sent with the model's own prompt typed in, it reproduces sentence-transformers' prompted vectors: cosine 1.00000 on EmbeddingGemma's classification, clustering and STS prompts, and 0.99998 on an F2LLM instruction.
• Disclosure:
  • /v1/models and sk_model_info list each embedder's prompts.
  • MODELS.md has a generated table of them.
  • The README states the default.
• Under truncate: "START", sidekick keeps its own prompt whole but can't tell a client's prompt from the text, so it may cut it. With none, use END, or keep inputs within the limit.
• /v2/embed still refuses Cohere's classification and clustering, since no model declares prompts for them. Send none with your own prompt instead.

/tokenize and /detokenize (D44)

• POST /tokenize takes {model, prompt, add_special_tokens?, return_token_strs?, input_type?, max_tokens?} and returns {count, max_model_len, tokens, token_strs?, fits}.
• POST /detokenize takes {model, tokens} and returns {prompt}. Special tokens are kept, as in vLLM.
• count is the embedding path's own count: prompt, text and special tokens, never truncated, even past a tokenizer's own limit.
• max_model_len is the limit /v1/embeddings enforces: /v1/models' max_seq_len, or a lower limit set by the model's tokenizer.json.
• Use fits to decide whether a text will be accepted, not count <= max_model_len. fits is the route's own verdict, from the same function /v1/embeddings (under truncate: NONE) or /v1/classify runs. The routes also bound a text's size in bytes, which ordinary text can break. 42 KB of Thai prose counts as 122 tokens on bge-small-en-v1.5, because its tokenizer reads each long run without spaces as one unknown token, and /v1/embeddings refuses it.
• Supported models: embedders and text-classification models. Rerankers, the zero-shot classifier formats and the Foundation Models chat model get a 400 that says why.
• Cost:
  • At most 4 MiB of text is tokenized at once, across requests.
  • Waiting for room counts against the request timeout.
  • Measured with 24 concurrent 2 MiB requests on LFM2.5-Embedding's tokenizer (1.6 million tokens each), peak memory was 2.1 GB. Most of that is macOS's cache of freed large blocks: with MallocLargeCache=0 in the daemon's environment it was 0.45 GB.

Benchmark scores beside fidelity grades

MODELS.md and the comparison page now say how good each model is at a task, next to how faithfully each build reproduces it.

• MTEB scores for the embedders, by task category, on MTEB(eng, v2), MTEB(Multilingual, v2), MTEB(eng, v1) and NanoBEIR-multilingual. tools/fetch_benchmarks.py reads them from the mteb/results dataset at a pinned commit and averages them as the leaderboard does. Every complete mean matches the live leaderboard.
• Model-card scores for the reranker, the NLI models and the other classifiers, in docs/data/benchmarks.toml, each with a source pinned to a revision.
• Provenance is enforced. tools/models_table.py --check, which CI runs, refuses:
  • a score without its benchmark version, metric, source or read date;
  • a mean over fewer than all of its tasks;
  • a source URL not pinned to a revision.
• Marked on the page:
  • scores on tasks the model was trained on;
  • self-reported scores;
  • the context each MTEB run used, against each build's max_seq_len;
  • whether a client can reproduce the run's prompts.
• The comparison page has a "Benchmark score" filter that adds a sortable score column, and shows every build's maximum tokens.
• Read them with care:
  • Compare scores only within one benchmark.
  • A score carries over to a build graded A, and may not to C, D or int8.
  • A few hundred labelled examples from your own data decide.

What's Changed

• feat(docs): publish benchmark scores beside each build's fidelity grade by @ironblock in #121
• feat(server): /tokenize and /detokenize for encoder models by @ironblock in #122
• feat(embeddings): input_type "none" embeds the text as sent; disclose prompts by @ironblock in #125
• chore(release): 0.14.0 by @ironblock in #126

Full Changelog: v0.13.0...v0.14.0