sidekick v0.15.0

sidekick v0.15.0 lets a client choose any of an embedder's own named prompts (prompt_name, as in Text Embeddings Inference), and returns an NLI model's full class probabilities on request (nli_probs), so a client can tell "contradicted" from "not mentioned". It caps Core ML models served on the Neural Engine at 2,048 tokens, the longest length measured to work there, and its converter now stores a model's weights once for all of its length buckets, about a third of the disk. It also publishes how sidekick's builds score on MTEB, Apple M4 Max results for the ONNX builds, and a first verification corpus with a pilot scoring of fact-checking models. Existing requests behave as in 0.14.0, apart from the ANE cap, which changes no published model.

Upgrading

• HTTP API: additions, plus one new limit.
  • prompt_name on /v1/embeddings, /v2/embed and /tokenize, and named_prompts for each embedder in /v1/models (D45). Sending prompt_name with one of sidekick's own input_type values (query, document, passage, none) is a 400; on /v2/embed, a prompt_name overrides Cohere's input_type values.
  • nli_probs: true on /v1/classify for nli-format models (D47). /v1/models lists the model's nli_classes, and lists nli_probs in extensions where it's available. Without the field, responses are byte for byte as before.
  • The ANE length cap (D48): a Core ML model served with compute_units cpu_and_ne or all serves at most 2,048 tokens, or its manifest's ane_max_seq_len. No published build goes past 2,048, so none changes. A model you converted yourself with longer buckets loses them on the ANE: its effective max_seq_len drops, and longer inputs get D43's 400. /v1/models reports each capped model's seq_cap, and /health lists them under seq_caps.
• C ABI: additions only. sk_embed_named embeds with a named prompt, and sk_model_info reports named_prompts. SK_ABI_VERSION is unchanged: a library whose sk_model_info reports named_prompts exports sk_embed_named, so resolve it with dlsym if the host must also run against older libraries.
• Rust API: one breaking change. EmbedPurpose gains Named(&str), so it now has a lifetime parameter, and it's marked #[non_exhaustive] so later purposes won't break matches again. Code that names the type or matches every variant needs updating. Classifier::run_nli and ScanOptions::no_neural_engine are new.
• Config keys: ignore_ane_seq_cap = true, or sidekickd --ignore-ane-seq-cap, lifts the ANE cap for every model, for measurement. Everything else is unchanged.
• Manifests: three new optional keys.
  • [prompts] holds the checkpoint's named prompts. The committed manifests for EmbeddingGemma-300m, embeddinggemma-2, F2LLM-v2-160M, LFM2.5-Embedding-350M and Qwen3-Embedding-0.6B carry them, and older daemons ignore the table.
  • [classify.nli] classes lists an nli model's classes in output order. nli_probs needs it: a manifest installed before 0.15.0 has no such line, so nli_probs isn't offered for that model (extensions doesn't list it) until you add the line from the model's committed example manifest. The committed nli manifests now carry it, and older daemons skip an nli model whose manifest has it, so don't install this release's nli manifests under a 0.14.0 daemon.
  • ane_max_seq_len raises or lowers the ANE cap for one model.
• Converter: multi-bucket Core ML conversions share one weights file per chunk across buckets by default (D49); --no-shared-weights converts as before. A shared build's weight files hash differently from a separate build's, though its outputs are bit-identical.
• The release archives are the same four as 0.14.0, with the same bundled ONNX Runtime versions.

Named prompts (D45)

Many embedders publish more prompts than sidekick's two input_type ones. EmbeddingGemma names prompts for classification, clustering, sentence similarity and code retrieval, among others. Before this release, a client had to type that text into its input with input_type: "none", where truncate: "START" could cut it.

• prompt_name picks one of the model's own prompts by name, matched exactly, case included, as Text Embeddings Inference and sentence-transformers do. A name the model doesn't have is a 400 that lists the ones it has.
• A named prompt is sidekick's prompt, so truncate: "START" keeps it whole. Measured on EmbeddingGemma, an over-length input with its prompt typed into the text gave cosine 0.79 under START, against the same prompt sent by name.
• Disclosure: /v1/models and sk_model_info list each embedder's named_prompts, and MODELS.md has a generated table of them.
• Instruction models (Qwen3-Embedding, F2LLM) take a free-text instruction rather than a fixed set. Their checkpoints name only query and document. For other tasks, send the whole instruction with input_type: "none" and truncate with END, or keep inputs within the limit. docs/INTEGRATING.md gives each model's format.

NLI class probabilities (D47)

An nli model's probs reduce each pair to one value (entailment against contradiction), which can't tell a contradicted claim from one the evidence doesn't mention. With nli_probs: true, each result also carries, per candidate label, the pair's softmax over the model's own classes, for example {"contradiction": 0.97, "entailment": 0.01, "neutral": 0.02}.

• The classes come from the manifest's [classify.nli] classes, the checkpoint's id2label in output order and lowercased. The registry checks it against the model's outputs, so a list in the wrong order is refused.
• Two-way models list their own two classes, such as entailment and not_entailment; they have no neutral.
• Cost: one prediction per pair serves both probs and nli_probs.

Length: the long-context plan (D46) and the ANE cap (D48)

D46 sets how far each build should go towards its model's trained context: the trained context or 2,048 tokens, whichever is shorter, and past 2,048 only where a backend has been measured to serve it well. It changes no manifest. It records Qwen3-Embedding-0.6B's long Core ML buckets, measured on an M1 Max:

• The GPU: 152, 350 and 940 ms at 1,024, 2,048 and 4,096 tokens, all grade A.
• The ANE: grades A at 2,048. At 4,096 its first compile took 29 minutes, and 7 of 51 predictions then failed.

D48 turns that into a runtime limit, as D33 does for the CPU:

• Which models: a Core ML model served on the ANE (cpu_and_ne, or all on a Mac with a Neural Engine) loads no bucket past 2,048 tokens unless its manifest's ane_max_seq_len says otherwise.
• Unaffected: cpu_and_gpu is uncapped, and cpu_only keeps D33's 1,024. On an Intel Mac, cpu_and_ne runs on the CPU, so it gets D33's cap instead.
• To serve long inputs, serve the model on the GPU ([models."<id>"] compute_units = "cpu_and_gpu"). The longer Qwen3-Embedding Core ML build itself will ship in a later release, once its full bucket list is graded on every path.

Shared weights across buckets (D49)

A Core ML build has one program per length bucket, and until now each carried its own copy of the model's weights. The converter now re-exports every bucket of a chunk against one shared weights file, and installs it once, hard-linked into each bucket.

model	separate buckets	shared
bge-small-en-v1.5 (3 buckets)	199.5 MB	67.4 MB
F2LLM-v2-160M (3 buckets)	956 MB	320 MB
Qwen3-Embedding-0.6B, chunked (2 buckets)	2.4 GB	1.2 GB

• Gated against the separate build:
  • every compiled weights file in a chunk must be the same file;
  • every program must match the separate build apart from weight offsets;
  • outputs must match bit for bit on the CPU, GPU and ANE.
    No parity grade changed.
• Sharing is per chunk, never across chunks, because the ANE's weight limit (D32) applies to the whole file a program references.
• What isn't shared: Core ML's compile cache still keeps a copy per bucket, and each loaded bucket maps its own weights, so memory and first-load compile times don't change.
• A copied model directory (cp -R) loses its links. python -m sidekick_convert.shared_weights link <model-dir> restores them.
• The converter's own Core ML cache: the converter now also empties it as each gate finishes. F2LLM's conversion peaked at 0.3 GB of cache instead of 3.9 GB.
• Published builds: the per-bucket sizes in MODELS.md describe separate builds until each model is reconverted.

Measurements and data

• MTEB on sidekick's builds. tools/mteb_transfer.py runs MTEB tasks through a running sidekickd and compares every vector with an fp32 reference, so you can see whether a model's published score carries over to the build you run. Builds graded A reproduce the fp32 run. Where a published score differs from the fp32 run, it's because of how the published run was made, not the conversion. The results are in docs/data/mteb_transfer.json and MODELS.md.
• Apple M4 Max, ONNX builds. All seven ONNX embedders grade A on an M4 Max. About 44% of the CPU time there runs in KleidiAI's SME2 kernels. The best thread count differs from the M1's: Qwen3-Embedding's long inputs are fastest at 2 threads.
• A verification corpus (#133). 54 reviewed triples test whether a checker can tell "supported", "contradicted" and "not mentioned" apart in real tool output, at 1k–32k tokens. Each triple puts a claim beside three nearly identical evidence haystacks. The obstacles include distractors, repeated mentions, superseded values, precedence and version scope. The tools and evidence are pinned and scrubbed, and generation is deterministic.
• A pilot scoring of fact-checkers on it is in MODELS.md and docs/data/verification-scores.json. Every checker had to pass known-answer controls first. The report states each checker's ceiling, the smallest difference it can detect (about 18 points), and a strong second reader that sees exactly what each checker sees.
  • A strong LLM judge got 83% of triples fully right.
  • sidekick's NLI models, MiniCheck, HHEM and Apple's Foundation Models all failed their controls, scoring 0–22%. Their results are reported as instrument-limited.
  • An audit verified how each checker is called: labels, premise order and a textbook MNLI check. It found no construction fault, and the causes of the failures are recorded as candidates, not conclusions.
  • It's a pilot: it ranks nothing until the corpus grows.

What's Changed

• feat(embeddings): named prompts by prompt_name, as in Text Embeddings Inference by @ironblock in #130
• feat(tools): measure whether published MTEB scores carry over to sidekick's builds by @ironblock in #135
• docs(models): publish onnx-cpu results from an Apple M4 Max by @ironblock in #138
• docs: D46, how far each build goes toward its model's context by @ironblock in #143
• feat(classify): NLI class probabilities on request (nli_probs) by @ironblock in #148
• feat(registry): cap ANE-served Core ML models at 2,048 tokens (D48) by @ironblock in #149
• feat(corpus): verification triples from hermetic captures by @ironblock in #146
• fix(corpus): no capture complete minus its needle in NOT MENTIONED; review-4 by @ironblock in #151
• feat(convert): one weights file per chunk, shared by every bucket (D49) by @ironblock in #153
• feat(tasks): scoring harness for the verification corpus (#133) by @ironblock in #152
• fix(tasks): attribute the failed verification checkers by audit, not by assertion by @ironblock in #154
• chore(release): 0.15.0 by @ironblock in #155

Full Changelog: v0.14.0...v0.15.0