sidekick v0.17.0

sidekick v0.17.0 serves open chat models on the Neural Engine beside Apple Foundation Models, through the same /v1/chat/completions (D53). The first is MiniCPM5-2B, as minicpm5-2b: a build with rotations folded into its weights and GPTQ onto 8-bit palettes, run as a stateful Core ML decoder that keeps the GPU free for a larger model. It grades B against the original under a new rubric for chat models (D54, D55) and decodes at about 18.5 tokens/s on an M1 Max. Its graded build is published on Hugging Face as sidekickd/minicpm5-2b-coreml, and sidekickd fetch minicpm5-2b installs it. The chat route also changes for Foundation Models: one JSON Schema layer checks response_format for every chat backend and refuses what no backend can honor, guided replies keep the schema's property order, seed and top_p are honored, and temperature: 0 is greedy. Embedding, classification and reranking requests behave as in 0.16.0.

Upgrading

• /health's chat section is reshaped, a breaking change for anything that reads it. The single object (chat.model, chat.availability, chat.context_limit, chat.variant, chat.variant_id, chat.model_capabilities, chat.fm_sdk) moves to chat.models["apple-fm"] with the same fields, plus backend, request_timeout_secs and request_timeout_source. chat.model is gone: the key is the model's id. chat.lane is new, and null unless an open model is installed. An open model's entry adds, among others, its state, load_error, queue_depth and lane_queue_depth, request and load counts, its decoder's calls and stalls, its Neural Engine share (ane) and its grammar engine's state (guided). A new top-level request_timeout_overrides lists the per-model timeouts applied and those that name no model served. docs/API.md, "Listing models and health", describes each field.
• response_format schemas are checked against one subset, and the rest is a 400 (#179). A keyword outside it is a 400 unsupported_schema whose message names the keyword and its JSON pointer, on apple-fm as on open models. Schemas the Foundation Models bridge accepted while dropping parts of them are now refused: among others minimum, maximum, pattern, format, minLength, $ref and $defs, oneOf and allOf, a union of types, enum of non-strings, an open object, and an array without items. The subset is in docs/API.md, "Structured output" (D8 and D53).
• Guided replies on apple-fm follow the schema's property order (#180). Foundation Models now generates properties in the order the schema lists them (the bridge sorted them by name before), and the reply is re-serialized as compact JSON on one line with its keys in that order ({"name":"Ada","age":36}), so identical requests at temperature 0 give identical bytes. The generation order can change what the model writes: put the fields it should decide first first. A nullable property ("type": [T, "null"], or anyOf with {"type": "null"}) is now supported as an object property's value, and a required nullable property the model leaves out comes back as null.
• Sampling on apple-fm. temperature: 0, or top_p: 0, now selects Foundation Models' greedy mode, so a request repeats byte for byte on the same Mac and macOS; it used to set a temperature of 0 on the default sampling. top_p and seed, which were ignored, are honored through Foundation Models' random sampling mode (seed alone samples the whole distribution, top_p 1). A request with none of the three samples as before (D22 amended; docs/API.md, "Sampling").
• Range checks: temperature outside 0–2 and top_p outside 0–1 are a 400 invalid_request_error, on every chat model.
• Guardrail refusals keep their status and code, a 400 content_filter, but the message now names the model and says whether output was guided: "apple-fm's guardrails refused this request, whose output was plain text: …", where it was "The on-device model's guardrails rejected this request: …". A refused guided request also carries OpenAI's param: "response_format", since guided output can be refused where the same prompt as plain text isn't (#178); no other error sets param. A refusal after text has streamed now sets the final chunk's constrained to whether output was guided. The daemon never retries a refusal or answers it with another model.
• Installing an open model changes how apple-fm runs. With none installed, apple-fm runs as before. With one, apple-fm and the open models share one queue on the Neural Engine, the ANE chat lane, because a Foundation Models request overlapping a generation slows both about 25×: requests run one at a time in arrival order; past queue_depth waiting (default 4), a request is a 503 backend_busy with Retry-After: 5; time in the queue counts against the request timeout; and a request that timed out keeps its turn until Foundation Models has stopped generating, which for a non-streamed request can take tens of seconds (#197).
• A streamed chat request whose client disconnects gives up its place in a queue or a load wait, on every chat model, apple-fm included.
• HTTP API: other changes are additions. /v1/models lists each chat model's backend (foundation_models for apple-fm, coreml for an open model), context_length (prompt and completion together), honored_fields, guided_streaming and guardrails, and for an open model its compute_units, graded_os and its decoder's compile_cache. A request to an open model whose decoder hung is a 503 backend_unavailable with the reason. JSON objects the daemon assembles itself, such as /health, now keep their keys in the order they are written instead of sorted by name.
• Config keys, all optional: ane_budget_mib (default 2,816), the Neural Engine footprint past which loading an open model logs a warning; [models."<id>"] request_timeout_secs, one model's request timeout over the global one, for any model (chat, embedder or classifier; at least 1); and, for open chat models, [models."<id>"] queue_depth (default 4) and schema_in_prompt (default true). model_idle_ttl_secs also unloads an idle open model. docs/CONFIG.md lists them.
• Manifests: a third kind, chat.toml, for an open chat model (task = "text-generation", interface = "kv-state-1"): its decoder's chunk programs, the embedding and RoPE tables the host reads, the tokenizer, the checkpoint's chat template, sampling defaults and a default request timeout (docs/design/open-llm.md, "The manifest"). Unknown keys are refused, so a manifest that asks for something the daemon can't honor is skipped, with the reason in /health's skipped_models. Open models are skipped too in a build without Core ML (the Linux archives), on a Mac without a Neural Engine, and when the config's compute_units for them is anything but cpu_and_ne. The id apple-fm is reserved. manifest.toml and classifier.toml are unchanged.
• Command line: no new flags. sidekickd fetch installs a chat build whole (--buckets doesn't apply), and sidekickd warm and the startup warm-up (D52) compile an open model's decoder as they compile any Core ML bucket.
• C ABI: unchanged. SK_ABI_VERSION is the same; libsidekick.dylib serves no chat.
• Rust API: additions, with source-level changes. sidekick_core::ChatRequest gains top_p, seed, deadline and hold, and its schema is a validated sidekick_core::JsonSchema instead of a serde_json::Value. sidekick_core::Error gains Busy, LoadTimeout and UnsupportedSchema, which an exhaustive match must handle. The stop-sequence shaper moves from sidekick_fm::shaping to sidekick_core::shaping. New: the sidekick-llm crate; sidekick_core::chat_manifest, sidekick_core::schema, ResolvedChat, WorkHold, FOUNDATION_MODELS_ID and ModelRegistry's chat, chat_ids, chats, skip_chat and skip_chats; sidekick_coreml::KvDecoder and its program contract; and sidekick_embed::warm's load_with, warm_with, refuse and refusal. The workspace enables serde_json's preserve_order feature, which Cargo's feature unification turns on for every crate in a build that includes sidekick's: a serde_json::Map there keeps insertion order instead of sorting its keys.
• The release archives are the same four as 0.16.0, with the same bundled ONNX Runtime versions, and the macOS archives' sidekickd keeps its signing identifier, io.github.ironblock.sidekickd. sidekickd now statically links llguidance (MIT) and minijinja (Apache-2.0); no new library ships beside it. THIRD_PARTY_NOTICES.md adds the MiniCPM5-2B and SmolLM2-135M-Instruct chat templates and the Wikipedia excerpts that the test fixtures, the converter's calibration and the chat benchmark use; none of them is compiled into sidekickd.

Open chat models on the Neural Engine (D53)

sidekick is for the work beside a large model that owns the GPU. Its chat route served only Apple's Foundation Models, whose guided generation drops optional fields (#177) and whose guardrails refuse guided output on ordinary text (#178). The chat benchmark (docs/CHAT-BENCHMARK.md) found MiniCPM5-2B level with Foundation Models on titles, fixed-vocabulary tags and instructions, well ahead on extraction, and refusing nothing. This release runs it on the Neural Engine.

• Choosing a model: a request goes to the chat model its model names, exactly: there is no default, no alias and no fallback from one model to another. /v1/models lists what each chat model honors, so a client chooses.
• The build: four chunk programs of 14, 14, 10 and 4 layers, each one 16-token function that keeps its layers' KV cache in Core ML state, over a 2,048-position cache, so the context is 2,033 tokens, prompt and completion together. 25 files, 2.63 GiB to download.
• The grade: B on the Neural Engine under D54, by each family's mean KL divergence from fp32,...

sidekick v0.16.0

sidekick v0.16.0 publishes converted builds on Hugging Face and installs them with one command: sidekickd fetch bge-small-en-v1.5 downloads a graded build, checks every file against sidekick's checksum ledger, and installs it ready to serve. The first published builds are bge-small-en-v1.5 and Qwen3-Embedding-0.6B, each in Core ML and ONNX; Qwen3-Embedding's Core ML build now takes inputs up to 2,048 tokens. The daemon also compiles Core ML models ahead of requests, so a model's first request after an install, upgrade or macOS update no longer waits for that compile, and the macOS archives' sidekickd keeps one code-signing identity, so upgrading no longer recompiles every Neural Engine model. The docs are reorganised around one page per task, with a shorter README. Existing requests behave as in 0.15.0.

Upgrading

• The first start after this upgrade compiles every Neural Engine model once more. From this release, the macOS archives' sidekickd is signed with a fixed identifier, io.github.ironblock.sidekickd. Core ML keys each compiled ANE program to the identifier of the process that loads it; until now that was cargo's hashed name, which changed with every version, so every upgrade recompiled every ANE bucket. It changes once more now, and not on later upgrades. If you build sidekickd yourself, run tools/sign_sidekickd.sh target/release/sidekickd after cargo build (docs/INSTALL.md).
• The daemon now compiles Core ML models at startup, in the background (D52). A child process loads each bucket the daemon will serve, one at a time, so Core ML compiles it before a request needs it; nothing stays loaded, so memory and idle unloading are unchanged. Requests are served meanwhile: one that needs the bucket being compiled waits for that compile, then loads it from the cache. To keep the old behaviour (compile on first request), set warm_on_start = false or pass --no-warm; to leave out one model, [models."<id>"] warm = false. warm_budget_secs (default 1,800, 0 for no limit) stops starting new compiles after that long.
• HTTP API: additions only. /v1/models lists each Core ML bucket's compile_cache state (warm, warming, cold or failed), and /health has a warmup object with the warm-up's progress and each bucket's load times. A model capped by its manifest's new cpu_max_seq_len says so in seq_cap.reason.
• Command line: new subcommands. sidekickd fetch and sidekickd warm, below. A bare sidekickd still serves.
• Config keys: warm_on_start, warm_budget_secs and [models."<id>"] warm, all optional. docs/CONFIG.md now lists every key, flag, environment variable and per-OS default, and a test keeps it complete.
• Manifests: one new optional key. cpu_max_seq_len lowers a Core ML model's CPU length cap (D33's 1,024) for a model whose buckets already differ on the CPU below it. Qwen3-Embedding-0.6B's manifest sets 512. A daemon before 0.16.0 ignores the key and serves that model's CPU path to 1,024 tokens, where its buckets disagree, so serve the published Qwen3-Embedding Core ML build with 0.16.0 or later.
• Qwen3-Embedding-0.6B's Core ML build goes to 2,048 tokens. The committed manifest lists buckets 128, 256, 512, 1,024 and 2,048, and the converter now converts a manifest's own buckets. On the ANE and the GPU, inputs past 512 tokens are now embedded whole; the 512-token build refused them (D43's 400) or cut them (truncate), so their vectors differ. On the CPU it serves up to 512 tokens.
• C ABI: unchanged. SK_ABI_VERSION is the same. A host embedding libsidekick.dylib is cached by Core ML under its own process's identity; docs/INTEGRATING.md explains what that means for a host's upgrades.
• Rust API: additions, with one source-level change. sidekick_embed::warm is new. ModelManifest and ClassifierManifest gain a public cpu_max_seq_len field, so code that builds either with a struct literal must name it. sidekick_core::cpu_seq_cap takes the manifest's cpu_max_seq_len.
• The release archives are the same four as 0.15.0, with the same bundled ONNX Runtime versions. Their README.md and INTEGRATING.md now link to this release's files on GitHub instead of relative paths that didn't resolve once unpacked.
• Docs have moved. Links into the old README's sections now land on its top. Each topic has its own page: INSTALL, RUNNING, CONFIG, API, BACKENDS and DEVELOPMENT, beside the existing ones (below).
• If you run sidekickd as a LaunchAgent, the documented plist now sends standard output, where the daemon logs, to ~/Library/Logs as well as standard error. The old one logged standard error only, under /tmp (docs/INSTALL.md).

Published builds on Hugging Face (D50, D51)

Converting a model for the Neural Engine takes a pinned toolchain, gigabytes of downloads and, for a large model, hours. sidekick now publishes the builds it has graded, under the sidekickd organisation, one repository per model and backend.

model	Core ML	ONNX
bge-small-en-v1.5	sidekickd/bge-small-en-v1.5-coreml: fp16, 128–512 tokens; ANE, GPU and CPU all grade A	sidekickd/bge-small-en-v1.5-onnx: fp32; grade A
Qwen3-Embedding-0.6B	sidekickd/qwen3-embedding-0.6b-coreml: fp16, 128–2,048 tokens; ANE A, GPU A, CPU B (to 512 tokens)	sidekickd/qwen3-embedding-0.6b-onnx: fp32 (default) and fp16-weights, 4,096 tokens; both grade A

• A repository holds one directory per build, exactly as sidekick installs it, plus a checksum file per build, the model's licence text (and its NOTICE, where upstream has one), and a model card. Core ML builds are compiled programs (.mlmodelc), and their buckets share one weights file per chunk (D49), so Qwen3-Embedding's five buckets download as 1.2 GB, not 6.1 GB.
• The model card holds only what is particular to the model: the builds and their grades on each compute path, the licence and training data, the original model's scores, and every file's sha256. Everything else (how fetch works, what a grade means, how to reproduce a build) links to this release's docs. docs/PUBLISHING.md describes the card.
• Which models can be published depends on their licences and their training data's, as recorded in docs/data/licences.toml for all 28 models in the registry. A model whose training data carries non-commercial terms, such as MS MARCO for Qwen3-Embedding, is published with a notice at the top of its card, not behind a gate: whether such terms bind a model trained on the data is unsettled, the user decides whether their use is permitted, and the model's own licence is unchanged. Only terms that must be passed on to every recipient (the Gemma Terms of Use) put a repository behind Hugging Face's access gate; a model whose licence doesn't allow redistribution (LFM2.5-Embedding) isn't published. docs/PUBLISHING.md gives the policy.
• The checksum ledger, docs/data/ledger.json (tools/ledger.py), records every build sidekick has converted, published or not: every file's sha256 and size, what produced it (converter commit, coremltools, PyTorch, OS, Xcode, chip), the grades of the reports that graded those exact files, and where it is published. MODELS.md's "Published builds" table is generated from it.
• The comparison page (https://ironblock.github.io/sidekick/) gains Licence and Training data columns, a licence filter, and a "Leave out" filter for, among others, models with non-commercial training data.

sidekickd fetch (D51)

sidekickd fetch bge-small-en-v1.5                       # Core ML on macOS, its published build
sidekickd fetch qwen3-embedding-0.6b --backend onnx     # fp32; --variant fp16-weights for half the download
sidekickd fetch qwen3-embedding-0.6b --buckets 128,512  # only the buckets you'll use

• Download: with the Hugging Face CLI (hf, 1.0 or later) installed, fetch runs it, with your login, Xet transfers and HF_* settings. Without it, fetch downloads anonymously with the system's curl into the same cache, in the layout hf uses, so either can reuse the other's files. --no-hf uses curl even when hf is installed.
• Verification: every file is checked against the Hub's hash, then against the checksum ledger built into sidekickd. The installed files must match one recorded build of that model, backend and variant, or nothing is installed. The repository, revision and variant default to the build's latest published record; --repo and --revision fetch from elsewhere, verified the same way.
• Install: the files are hard-linked from the download cache into the models directory, never symlinked (Core ML can lose the Neural Engine on some builds through a symlink, docs/design/coreml-cache.md), so the files on disk are stored once. A gated repository's terms are shown, and fetch stops until you pass --accept-terms <model>.
• Then, fetch compiles the Core ML buckets it installed (D52; --no-warm skips this). The daemon reads its models directory at startup, so restart it to serve the new model.

Compiling ahead of requests (D52)

A Core ML bucket's first load in a process compiles it for the device: a few seconds on the ANE for a small model, a minute or more for a long bucket of a large one. The cache that holds the result is keyed to the executable, its code-signing identity, the home directory, the model's path and its files, and the macOS build, so a new install, a moved model or a macOS update starts cold.

• When: at startup in the background, after sidekickd fetch, and on demand: sidekickd warm [<model>…] [--buckets …] [--budget-secs N] [--list].
• What: exactly the buckets the d...

sidekick v0.15.0

sidekick v0.15.0 lets a client choose any of an embedder's own named prompts (prompt_name, as in Text Embeddings Inference), and returns an NLI model's full class probabilities on request (nli_probs), so a client can tell "contradicted" from "not mentioned". It caps Core ML models served on the Neural Engine at 2,048 tokens, the longest length measured to work there, and its converter now stores a model's weights once for all of its length buckets, about a third of the disk. It also publishes how sidekick's builds score on MTEB, Apple M4 Max results for the ONNX builds, and a first verification corpus with a pilot scoring of fact-checking models. Existing requests behave as in 0.14.0, apart from the ANE cap, which changes no published model.

Upgrading

• HTTP API: additions, plus one new limit.
  • prompt_name on /v1/embeddings, /v2/embed and /tokenize, and named_prompts for each embedder in /v1/models (D45). Sending prompt_name with one of sidekick's own input_type values (query, document, passage, none) is a 400; on /v2/embed, a prompt_name overrides Cohere's input_type values.
  • nli_probs: true on /v1/classify for nli-format models (D47). /v1/models lists the model's nli_classes, and lists nli_probs in extensions where it's available. Without the field, responses are byte for byte as before.
  • The ANE length cap (D48): a Core ML model served with compute_units cpu_and_ne or all serves at most 2,048 tokens, or its manifest's ane_max_seq_len. No published build goes past 2,048, so none changes. A model you converted yourself with longer buckets loses them on the ANE: its effective max_seq_len drops, and longer inputs get D43's 400. /v1/models reports each capped model's seq_cap, and /health lists them under seq_caps.
• C ABI: additions only. sk_embed_named embeds with a named prompt, and sk_model_info reports named_prompts. SK_ABI_VERSION is unchanged: a library whose sk_model_info reports named_prompts exports sk_embed_named, so resolve it with dlsym if the host must also run against older libraries.
• Rust API: one breaking change. EmbedPurpose gains Named(&str), so it now has a lifetime parameter, and it's marked #[non_exhaustive] so later purposes won't break matches again. Code that names the type or matches every variant needs updating. Classifier::run_nli and ScanOptions::no_neural_engine are new.
• Config keys: ignore_ane_seq_cap = true, or sidekickd --ignore-ane-seq-cap, lifts the ANE cap for every model, for measurement. Everything else is unchanged.
• Manifests: three new optional keys.
  • [prompts] holds the checkpoint's named prompts. The committed manifests for EmbeddingGemma-300m, embeddinggemma-2, F2LLM-v2-160M, LFM2.5-Embedding-350M and Qwen3-Embedding-0.6B carry them, and older daemons ignore the table.
  • [classify.nli] classes lists an nli model's classes in output order. nli_probs needs it: a manifest installed before 0.15.0 has no such line, so nli_probs isn't offered for that model (extensions doesn't list it) until you add the line from the model's committed example manifest. The committed nli manifests now carry it, and older daemons skip an nli model whose manifest has it, so don't install this release's nli manifests under a 0.14.0 daemon.
  • ane_max_seq_len raises or lowers the ANE cap for one model.
• Converter: multi-bucket Core ML conversions share one weights file per chunk across buckets by default (D49); --no-shared-weights converts as before. A shared build's weight files hash differently from a separate build's, though its outputs are bit-identical.
• The release archives are the same four as 0.14.0, with the same bundled ONNX Runtime versions.

Named prompts (D45)

Many embedders publish more prompts than sidekick's two input_type ones. EmbeddingGemma names prompts for classification, clustering, sentence similarity and code retrieval, among others. Before this release, a client had to type that text into its input with input_type: "none", where truncate: "START" could cut it.

• prompt_name picks one of the model's own prompts by name, matched exactly, case included, as Text Embeddings Inference and sentence-transformers do. A name the model doesn't have is a 400 that lists the ones it has.
• A named prompt is sidekick's prompt, so truncate: "START" keeps it whole. Measured on EmbeddingGemma, an over-length input with its prompt typed into the text gave cosine 0.79 under START, against the same prompt sent by name.
• Disclosure: /v1/models and sk_model_info list each embedder's named_prompts, and MODELS.md has a generated table of them.
• Instruction models (Qwen3-Embedding, F2LLM) take a free-text instruction rather than a fixed set. Their checkpoints name only query and document. For other tasks, send the whole instruction with input_type: "none" and truncate with END, or keep inputs within the limit. docs/INTEGRATING.md gives each model's format.

NLI class probabilities (D47)

An nli model's probs reduce each pair to one value (entailment against contradiction), which can't tell a contradicted claim from one the evidence doesn't mention. With nli_probs: true, each result also carries, per candidate label, the pair's softmax over the model's own classes, for example {"contradiction": 0.97, "entailment": 0.01, "neutral": 0.02}.

• The classes come from the manifest's [classify.nli] classes, the checkpoint's id2label in output order and lowercased. The registry checks it against the model's outputs, so a list in the wrong order is refused.
• Two-way models list their own two classes, such as entailment and not_entailment; they have no neutral.
• Cost: one prediction per pair serves both probs and nli_probs.

Length: the long-context plan (D46) and the ANE cap (D48)

D46 sets how far each build should go towards its model's trained context: the trained context or 2,048 tokens, whichever is shorter, and past 2,048 only where a backend has been measured to serve it well. It changes no manifest. It records Qwen3-Embedding-0.6B's long Core ML buckets, measured on an M1 Max:

• The GPU: 152, 350 and 940 ms at 1,024, 2,048 and 4,096 tokens, all grade A.
• The ANE: grades A at 2,048. At 4,096 its first compile took 29 minutes, and 7 of 51 predictions then failed.

D48 turns that into a runtime limit, as D33 does for the CPU:

• Which models: a Core ML model served on the ANE (cpu_and_ne, or all on a Mac with a Neural Engine) loads no bucket past 2,048 tokens unless its manifest's ane_max_seq_len says otherwise.
• Unaffected: cpu_and_gpu is uncapped, and cpu_only keeps D33's 1,024. On an Intel Mac, cpu_and_ne runs on the CPU, so it gets D33's cap instead.
• To serve long inputs, serve the model on the GPU ([models."<id>"] compute_units = "cpu_and_gpu"). The longer Qwen3-Embedding Core ML build itself will ship in a later release, once its full bucket list is graded on every path.

Shared weights across buckets (D49)

A Core ML build has one program per length bucket, and until now each carried its own copy of the model's weights. The converter now re-exports every bucket of a chunk against one shared weights file, and installs it once, hard-linked into each bucket.

model	separate buckets	shared
bge-small-en-v1.5 (3 buckets)	199.5 MB	67.4 MB
F2LLM-v2-160M (3 buckets)	956 MB	320 MB
Qwen3-Embedding-0.6B, chunked (2 buckets)	2.4 GB	1.2 GB

• Gated against the separate build:
  • every compiled weights file in a chunk must be the same file;
  • every program must match the separate build apart from weight offsets;
  • outputs must match bit for bit on the CPU, GPU and ANE.
    No parity grade changed.
• Sharing is per chunk, never across chunks, because the ANE's weight limit (D32) applies to the whole file a program references.
• What isn't shared: Core ML's compile cache still keeps a copy per bucket, and each loaded bucket maps its own weights, so memory and first-load compile times don't change.
• A copied model directory (cp -R) loses its links. python -m sidekick_convert.shared_weights link <model-dir> restores them.
• The converter's own Core ML cache: the converter now also empties it as each gate finishes. F2LLM's conversion peaked at 0.3 GB of cache instead of 3.9 GB.
• Published builds: the per-bucket sizes in MODELS.md describe separate builds until each model is reconverted.

Measurements and data

• MTEB on sidekick's builds. tools/mteb_transfer.py runs MTEB tasks through a running sidekickd and compares every vector with an fp32 reference, so you can see whether a model's published score carries over to the build you run. Builds graded A reproduce the fp32 run. Where a published score differs from the fp32 run, it's because of how the published run was made, not the conversion. The results are in docs/data/mteb_transfer.json and MODELS.md.
• Apple M4 Max, ONNX builds. All seven ONNX embedders grade A on an M4 Max. About 44% of the CPU time there runs in KleidiAI's SME2 kernels. The best thread count differs from the M1's: Qwen3-Embedding's long inputs are fastest at 2 threads.
• A verification corpus (#133). 54 reviewed triples test whether a checker can tell "supported", "contradicted" and "not mentioned" apart in real tool output, at 1k–32k tokens. Each triple puts a claim beside three nearly identical evidence haystacks. The obstacles include distractors, repeated mentions, superseded values, precedence and version scope. The tools and evidence are pinned and scrubbed, and generation is deterministic.
• A pilot scoring of fact-checkers on it is in MODELS.md and docs/data/verification-scores.json. Every checker had to pass known-answer controls first. The report states each checker's ceiling, the smalles...

sidekick v0.14.0

sidekick v0.14.0 adds /tokenize and /detokenize, so a client can check that a text fits a model before embedding it. It lets a client turn off the prompt sidekick puts before each embedding input (input_type: "none"), and lists those prompts in /v1/models. It also publishes each model's benchmark scores (MTEB and model-card results) beside its fidelity grades. Nothing an existing request does has changed.

Upgrading

• HTTP API: additions only.
  • POST /tokenize and POST /detokenize, in vLLM's shape (D44).
  • input_type: "none" on /v1/embeddings, /v2/embed and /tokenize.
  • prompts in /v1/models for each embedder.
  • Every existing request gets the same status, message and vector as in 0.13.0. The one difference is the error text for an unsupported input_type, which now mentions none.
• C ABI: additions only. sk_embed accepts purpose 2 (no prompt), and sk_model_info reports prompts. SK_ABI_VERSION is unchanged. A library that reports prompts accepts purpose 2, and an older one refuses it when sk_embed is called, so check sk_model_info first.
• Rust API: EmbedPurpose gains a Raw variant. Code that matches every variant explicitly needs an arm for it. EmbedInput::tokenize, EmbedInput::check_whole, InputBuilder::tokenize and InputBuilder::check_text are new.
• Config keys and manifests: unchanged.
• Memory: an embedder's backend and /tokenize share one copy of the model's tokenizer.
• The release archives are the same four as 0.13.0, with the same bundled ONNX Runtime versions.

Prompts, and turning them off (D9, amended)

Most supported embedders were trained with a prompt before their input, such as e5's query: and passage: or EmbeddingGemma's title: none | text: . OpenAI's API has no field for one, so sidekick adds the model's own retrieval prompts from its manifest: the query prompt for input_type: "query", the document prompt otherwise, including when input_type is absent. The README said little of this, and nothing turned it off, so a client that sent its own prompt got two. On EmbeddingGemma, MTEB's classification, clustering and similarity prompts sent that way gave vectors at cosine 0.906–0.926 to the model's.

• input_type: "none" embeds the text exactly as sent, keeping the tokenizer's special tokens. Sent with the model's own prompt typed in, it reproduces sentence-transformers' prompted vectors: cosine 1.00000 on EmbeddingGemma's classification, clustering and STS prompts, and 0.99998 on an F2LLM instruction.
• Disclosure:
  • /v1/models and sk_model_info list each embedder's prompts.
  • MODELS.md has a generated table of them.
  • The README states the default.
• Under truncate: "START", sidekick keeps its own prompt whole but can't tell a client's prompt from the text, so it may cut it. With none, use END, or keep inputs within the limit.
• /v2/embed still refuses Cohere's classification and clustering, since no model declares prompts for them. Send none with your own prompt instead.

/tokenize and /detokenize (D44)

• POST /tokenize takes {model, prompt, add_special_tokens?, return_token_strs?, input_type?, max_tokens?} and returns {count, max_model_len, tokens, token_strs?, fits}.
• POST /detokenize takes {model, tokens} and returns {prompt}. Special tokens are kept, as in vLLM.
• count is the embedding path's own count: prompt, text and special tokens, never truncated, even past a tokenizer's own limit.
• max_model_len is the limit /v1/embeddings enforces: /v1/models' max_seq_len, or a lower limit set by the model's tokenizer.json.
• Use fits to decide whether a text will be accepted, not count <= max_model_len. fits is the route's own verdict, from the same function /v1/embeddings (under truncate: NONE) or /v1/classify runs. The routes also bound a text's size in bytes, which ordinary text can break. 42 KB of Thai prose counts as 122 tokens on bge-small-en-v1.5, because its tokenizer reads each long run without spaces as one unknown token, and /v1/embeddings refuses it.
• Supported models: embedders and text-classification models. Rerankers, the zero-shot classifier formats and the Foundation Models chat model get a 400 that says why.
• Cost:
  • At most 4 MiB of text is tokenized at once, across requests.
  • Waiting for room counts against the request timeout.
  • Measured with 24 concurrent 2 MiB requests on LFM2.5-Embedding's tokenizer (1.6 million tokens each), peak memory was 2.1 GB. Most of that is macOS's cache of freed large blocks: with MallocLargeCache=0 in the daemon's environment it was 0.45 GB.

Benchmark scores beside fidelity grades

MODELS.md and the comparison page now say how good each model is at a task, next to how faithfully each build reproduces it.

• MTEB scores for the embedders, by task category, on MTEB(eng, v2), MTEB(Multilingual, v2), MTEB(eng, v1) and NanoBEIR-multilingual. tools/fetch_benchmarks.py reads them from the mteb/results dataset at a pinned commit and averages them as the leaderboard does. Every complete mean matches the live leaderboard.
• Model-card scores for the reranker, the NLI models and the other classifiers, in docs/data/benchmarks.toml, each with a source pinned to a revision.
• Provenance is enforced. tools/models_table.py --check, which CI runs, refuses:
  • a score without its benchmark version, metric, source or read date;
  • a mean over fewer than all of its tasks;
  • a source URL not pinned to a revision.
• Marked on the page:
  • scores on tasks the model was trained on;
  • self-reported scores;
  • the context each MTEB run used, against each build's max_seq_len;
  • whether a client can reproduce the run's prompts.
• The comparison page has a "Benchmark score" filter that adds a sortable score column, and shows every build's maximum tokens.
• Read them with care:
  • Compare scores only within one benchmark.
  • A score carries over to a build graded A, and may not to C, D or int8.
  • A few hundred labelled examples from your own data decide.

What's Changed

• feat(docs): publish benchmark scores beside each build's fidelity grade by @ironblock in #121
• feat(server): /tokenize and /detokenize for encoder models by @ironblock in #122
• feat(embeddings): input_type "none" embeds the text as sent; disclose prompts by @ironblock in #125
• chore(release): 0.14.0 by @ironblock in #126

Full Changelog: v0.13.0...v0.14.0

sidekick v0.13.0

sidekick v0.13.0 refuses over-length input to /v1/embeddings by default (D43). A request without truncate now gets a 400 naming the input, its token count and the model's limit, where 0.12 truncated the text silently and returned a normal vector. Send truncate: "END" to keep the old behaviour. This is the only change from 0.12.1, which added the field so clients could opt in first.

Upgrading

• HTTP API: one breaking change. On /v1/embeddings, a request without truncate (or with null) is now NONE: an input longer than the model's limit is a 400 invalid_request_error, and the whole request is refused before any inference runs. To keep 0.12's behaviour, send truncate: "END" (any case) or true; the vectors are byte-identical to 0.12's. OpenAI's Python SDK sends the field through extra_body.
• Each model's limit is max_seq_len in /v1/models (since 0.12.1), in tokens, counting special tokens and the model's prompt prefix.
• Unchanged, still truncating by default:
  • /v2/embed, whose truncate defaults to END as in Cohere's v2 contract;
  • the C ABI's sk_embed, which has no truncate parameter;
  • the Rust API's Embedder::embed (EmbedLimits::default() stays END);
  • /v2/rerank, and the laya, gliner2, fev and nli classifier formats, which truncate by design.
• The daemon and the dylib now differ for an over-length input. A host that uses the daemon and falls back to libsidekick.dylib should send truncate: "END" to the daemon to get the same vectors on both, or handle both behaviours.
• Error text: the over-length message now tells the client to send truncate: END or START, on /v2/embed's NONE too.
• Config keys, the C ABI and the Rust API: unchanged.
• The release archives are the same four as 0.12.1, with the same bundled ONNX Runtime versions.

Why (D43)

No decision ever chose silent truncation: /v1/embeddings passed through the embedders' library default. A consumer embedding a 2,355-document corpus through Qwen3-Embedding-0.6B at 512 tokens found 36% of its documents over the limit, with vectors that agreed with full-length ones only down to cosine 0.855, while every gate graded A. D10 already refuses silent truncation of an embedding's dimensions, and the APIs sidekick's are compared with refuse over-length input too: OpenAI's /v1/embeddings answers a 400, and Hugging Face Text Embeddings Inference returns an error unless the request sends truncate: true. Within sidekick, /v1/rerank and text-classification models on /v1/classify already refuse unless the request asks to truncate.

The error says how to proceed:

input 1: 600 tokens, over model `bge-small-en-v1.5`'s limit of 512 tokens, and truncate is NONE (send truncate: END or START to truncate it)

What's Changed

• feat(embeddings)!: refuse over-length /v1/embeddings input by default by @ironblock in #118
• chore(release): 0.13.0 by @ironblock in #120

Full Changelog: v0.12.1...v0.13.0

sidekick v0.12.1

sidekick v0.12.1 lets a client refuse over-length input to /v1/embeddings instead of having it silently truncated, lists each model's input limit in /v1/models, and raises Qwen3-Embedding-0.6B's ONNX build from 512 to 4,096 tokens, graded A on a new long-input parity corpus. It prepares for 0.13.0, where over-length input to /v1/embeddings becomes a 400 by default.

Upgrading

• HTTP API: additions, one behaviour change.
  • /v1/embeddings takes an optional truncate: NONE, START or END in any case, or TEI's boolean (true means END, false means NONE). Without it, behaviour is unchanged: over-length input is truncated, keeping the start.
  • Behaviour change: truncate used to be ignored on /v1/embeddings, so a client that sent truncate: false or "none" got a truncated vector and a 200. It now gets a 400 for over-length input, which is what those values ask for. true and "end" behave as before.
  • /v1/models lists each embedder's and classifier's max_seq_len, and each Core ML embedder's buckets, after the CPU cap (D33).
• Plan for 0.13.0: /v1/embeddings will refuse over-length input with a 400 by default, as OpenAI's embeddings API and Hugging Face Text Embeddings Inference do, and as sidekick's /v1/classify and /v1/rerank already do. To keep truncation, send truncate: "END" (or true) now.
• Error messages: an over-length error names the input's index in the batch, its token count, the model and its limit, or max_tokens when the request set it. The whole request is refused, and no inference runs.
• Qwen3-Embedding-0.6B on onnx-cpu now takes up to 4,096 tokens. An input over 512 tokens embeds differently from 0.12.0, which truncated it at 512. The model's Core ML build still stops at 512, so the two builds now give different vectors for long text: re-embed long documents, and don't mix the builds in one index.
• C ABI and config keys: unchanged. sk_embed still truncates.
• Rust API: additions only. sidekick_onnx::MemoryOptions and SHRINK_FROM_TOKENS, OnnxModel::load_with, the ONNX embedder's load_with, and EmbedInput::check_limits. Nothing was removed or changed.
• The release archives are the same four as 0.12.0, with the same bundled ONNX Runtime versions.

Over-length input to /v1/embeddings

/v1/embeddings has always truncated an input longer than the model's limit, with nothing in the response to say so, and the README didn't mention it. A consumer embedding a 2,355-document corpus (median 375 tokens, longest 2,408) through Qwen3-Embedding-0.6B's 512-token ONNX build found 854 documents (36%) truncated. Against full-length vectors, the truncated ones fell as low as cosine 0.855, while the in-limit ones agreed to 0.99999. Every published gate graded the model A, because no gate tested past about 520 tokens.

• truncate: "NONE" refuses any input over the limit with a 400, including on a model whose own tokenizer.json truncates (LFM2.5-Embedding-350M, at 512).
• The README now documents the default, the limit, and where to read it.

Qwen3-Embedding-0.6B at 4,096 tokens on onnx-cpu (D40)

• Grades: A on the main corpus (worst 1 − cos 1.7e-11, unchanged) and on a new long-input corpus at 600, 1,500, 2,400 and 4,096 tokens (1 − cos ≤ 1.2e-11), with no rank flips.
• The long-input corpus, fixtures/parity/corpus_long.toml, has its own hash and references. Existing models' references are unchanged, and the parity suite grades it for models whose limit reaches 600 tokens.
• Cost at length (M1 Max, 8 threads): about 0.9 s per input at 512 tokens, 7 s at 2,400 and 13–14 s at 4,096. The process's footprint is about 1.8 GB above its pre-load size once loaded, and peaks at about 2.6 GB above it at 2,400 tokens and 4.2 GB at 4,096. Long inputs suit background indexing, not interactive requests.
• Memory is given back: a run of 1,024 tokens or more now ends by releasing ONNX Runtime's arena growth, so after later inputs the footprint returns near its loaded size instead of staying at the peak. This applies to every ONNX model; shorter runs are unchanged.
• Timeouts: four 4,096-token inputs in one request can exceed the default 60 s request_timeout_secs, and the work continues after the 504. Send one or two long inputs per request, or raise the timeout.

Fixes

• START truncation on a model whose tokenizer.json truncates (LFM2.5) kept the start of the text instead of its end (#114). On a text of two paragraphs, the vector's cosine to the second went from 0.139 to 0.991.
• The static embedder tokenizes every input before embedding any, so one bad input refuses the whole request, as on Core ML and ONNX.

What's Changed

• feat(embeddings): truncate on /v1/embeddings, max_seq_len and buckets on /v1/models by @ironblock in #113
• feat(onnx,parity): Qwen3-Embedding-0.6B's ONNX build at 4,096 tokens, graded on a long-input corpus by @ironblock in #115
• chore(release): 0.12.1 by @ironblock in #119

Full Changelog: v0.12.0...v0.12.1

sidekick v0.12.0

sidekick v0.12.0 serves chunked Core ML embedders, so Qwen3-Embedding-0.6B now runs on the Neural Engine, where it grades A. It refuses an all-zero or non-finite embedding instead of serving it, after Core ML's CPU path on an M4 Max was measured returning zeros. It also logs which config file it read, and computes every published grade from a single definition.

Upgrading

• HTTP API and C ABI: unchanged in shape. Two kinds of request now get an error where they used to get a result:
  • an embedding whose output is all zeros or non-finite now gets a 500 internal_error instead of a 200 (see below);
  • an input that tokenizes to nothing, which only a tokenizer that adds no special tokens can produce, now gets a 400 on every backend, where ONNX models answered 500.
• Config keys: unchanged. The default config file on macOS is ~/Library/Application Support/sidekick/config.toml. On Linux it's ~/.config/sidekick/config.toml. --help and the README used to give only the Linux path, so a config written there on a Mac was ignored. The sidekickd starting log line now names the file it read, or the path it looked for.
• Manifests: a Core ML embedder may now carry [chunking] and a {chunk} artifact placeholder, as classifiers already could (D37). Static and ONNX embedders still reject both.
• Rust API: additions only. sidekick_embed::degenerate reports whether an embedding is unservable. sidekick_server::Config::resolve_path says which config file load reads. Nothing was removed or changed.
• Qwen3-Embedding-0.6B's served path is now Core ML on the ANE. MODELS.md and the comparison page list its chunked Core ML build, where they listed onnx-cpu. onnx-cpu stays its route on Linux and on Intel Macs, which have no ANE.
• The release archives are the same four as 0.11.0, with the same bundled ONNX Runtime versions.

Chunked embedders: Qwen3-Embedding-0.6B on the Neural Engine (D37)

Qwen3-Embedding-0.6B carries 1.11 GiB of fp16 weights per bucket. That is over the 1 GiB a single program could carry onto the ANE on an M1 Max (D32), so the model had no Core ML build. The embedder converter now splits each bucket into two programs of about 570 MB, and sidekickd serves the chain. At load, it checks that the chunks connect: the inputs, hidden_out to hidden_in, and the last chunk's output.

• Splitting changes no arithmetic. In fp32, the chunks equal the unsplit model exactly, and the GPU chain is bit-identical to the unsplit GPU program in every bucket. F2LLM-v2-160M, which isn't chunked, reconverts byte-identically.
• Grades on an M1 Max under macOS 27.0: A on the ANE (worst cosine 0.999986), A on the GPU (0.999997), and B on the CPU (0.999684). The compute plan puts 2,129 of its 2,135 operations on the ANE.
• Grades on an M4 Max under macOS 26.5.2: the ANE and GPU grades are the same. The CPU's is not; see below.

All-zero embeddings are refused

On an M4 Max under macOS 26.5.2, Core ML's CPU engine returns all-zero embeddings from Qwen3-Embedding-0.6B for some inputs, chunked or not. Truncating the model shows the depth involved: at 23 of its 28 layers every output is non-zero, and from 24 layers some are zero. Such an output is finite, so an unchunked build served with cpu_only under 0.11.0 would have answered with a 200. The same program compiled for fp32 is healthy on that machine, and so is onnx-cpu (A, 29 ms per input). The M1 Max under macOS 27.0 is unaffected.

• The Core ML and ONNX embedders now return an error when an output has a non-finite value, or when every value is zero. The error names the model, the bucket and the compute units. When the CPU returns zeros, the error also suggests other compute units. Static embedders are unaffected.
• On an M4-class Mac, serve Qwen3-Embedding-0.6B on its default, the ANE, or on the GPU, not with a cpu_only override.
• The converter now reports "all-zero output" for such cases, where it used to report a NaN parity cosine.
• ane_check reports non-finite and all-zero output as such, instead of blaming the attention mask.

The ANE weight limit depends on the chip or the OS

The same M4 Max under macOS 26.5.2 runs Qwen3-Embedding-0.6B's unsplit 1.110 GiB program on the ANE. Its compute plan puts 2,124 of 2,129 ops there, and the program takes 15 ms per input against 43 ms on the CPU. The M1 Max under macOS 27.0 puts none of it there. The two machines differ in both chip and OS, so it is open which of them moved the limit. The 1 GiB constant (D32) stays, as the conservative choice. Chunking keeps a chunkable model's programs under it, losslessly and at under 2% on the GPU. Deciding at load from the machine's own compute plan is tracked as #102.

Grades from one definition

• One table: the grade thresholds are [grades] in fixtures/parity/expectations.toml. The parity suite and tools/models_table.py both read it.
• Computed letters: every published letter is computed from its numbers, and CI fails on any row whose stored letter disagrees.
• Shared text: docs/MODELS.md and the comparison page share one formatter and generated legends.
• No grade changed. The check uncovered one mismatch, which was fixed by re-grading: nlptown-sentiment on an Intel Mac's Core ML CPU path is A under the current Intel bucket gate, with the same artifacts and measurements.

Also

• The converter checks early that coremltools has its native libraries. coremltools 9.0 has no Python 3.14 wheel. The package pip installs there used to fail only after a full conversion. CONVERTING.md states the supported Python versions.
• The converter's errors: the "can't be chunked" error names the backbone class. The over-the-limit error offers --chunks auto only for a backbone that can chunk.
• New and corrected docs:
  • CONVERTING.md cites the community ANE reference (hollance/neural-engine). It says where Core ML's diagnostics are logged, and notes that GPU grades assume Core ML's default fp32 accumulation.
  • It also notes that the onnxruntime used for exports needn't match the daemon's bundled runtime.
• tools/bench_threads.py, the sweep behind MODELS.md's thread table, is committed.

What's Changed

• fix(convert): honest chunking errors and an early coremltools check by @ironblock in #93
• fix(server): document the real default config path, and log the one read by @ironblock in #97
• One definition of the grades: computed letters, shared scales, one formatter by @ironblock in #98
• docs(converting): the export's onnxruntime needn't match the daemon's by @ironblock in #99
• feat: chunked Core ML embedders, and Qwen3-Embedding-0.6B on the ANE (D37) by @ironblock in #100
• fix(embed): refuse an all-zero or non-finite embedding instead of serving it by @ironblock in #101
• docs: the M4 Max's ANE limit and its zero-output CPU path, and the thread sweep tool by @ironblock in #107
• docs(converting): the community ANE reference, Core ML's log, GPU accumulation by @ironblock in #105
• chore(release): 0.12.0 by @ironblock in #108

Full Changelog: v0.11.0...v0.12.0

sidekick v0.11.0

sidekick v0.11.0 makes ONNX Runtime use its AVX-512 kernels on Intel Macs (D42), making embeddings on a Mac Pro's CPU 1.2–1.7× faster. It also converts ModernBERT NLI models, adds ModernBERT-large-zeroshot-v2.0 with a licence flag, and publishes NLI grades for an Intel Mac's GPU.

Upgrading

• HTTP API, C ABI and config keys: unchanged.
• A new environment variable, SIDEKICK_ONNX_AVX512=0, turns the AVX-512 change off. It only has an effect on an Intel Mac with AVX-512.
• Rust API: additions only. The new module sidekick_onnx::avx512 has enabled and enable_on_this_thread. Nothing was removed or changed.
• Embeddings on AVX-512 Intel Macs change in the last digits. The AVX-512 kernels round differently from the AVX2 ones, so onnx-cpu vectors from 0.11.0 and 0.10.0 on the same Mac agree to within 1 − cos ≈ 2e-11, not bit for bit. Stored vectors don't need re-embedding. Other machines are unaffected.
• The release archives are the same four as 0.10.0, with the same bundled ONNX Runtime versions: 1.28.3, or 1.23.2 on Intel macOS.

AVX-512 on Intel Macs (D42)

ONNX Runtime uses its AVX-512 kernels only when the CPU register XCR0 shows the operating system saves AVX-512 state. macOS turns that state on lazily, per thread, the first time a thread runs an AVX-512 instruction. Until then XCR0 says AVX-512 is off, so on a Mac Pro or iMac Pro, ONNX Runtime has always run its AVX2 kernels. Newer ONNX Runtime releases check the same way, so a newer or source-built runtime wouldn't have fixed this.

sidekick now runs one AVX-512 instruction on the threads that load ONNX Runtime and create its sessions, so ONNX Runtime selects its AVX-512 kernels. Linux turns AVX-512 on for every thread from boot and needs none of this. Apple silicon is unaffected.

Measured on a Xeon W-3235 Mac Pro, through /v1/embeddings at about 290 tokens:

threads	bge-small	gte-modernbert	embeddinggemma-300m	Qwen3-0.6B
1	1.51×	1.57×	1.60×	1.39×
8 (the default)	1.17×	1.28×	1.29×	1.19×

• Each cell is how many times as long the AVX2 kernels took; at 16 tokens the gain is 2–11% at 8 threads.
• Eight threads is still the fastest setting, so the thread default stands.
• Every exported embedder, EmbeddingGemma 2 included, grades A on onnx-cpu with the AVX-512 kernels.
• docs/MODELS.md's Xeon thread table and grades are re-measured on an idle machine.

ModernBERT NLI models

• tools/convert_nli.py converts ModernBERT-family NLI models. It converts them through sidekick's ModernBERT backbone, with the checkpoint's own classification head.
• MoritzLaurer/ModernBERT-large-zeroshot-v2.0 grades B on the GPU, with buckets to 2,048 tokens, at 19 ms per request and 249 ms per pair at 2,048 tokens on an M1 Max. Its MNLI accuracy, entailment against not, is 0.961 matched and 0.955 mismatched, the best of the NLI models here, on fp32 and every Core ML path. It has no ANE build: the best ANE variant grades C.
• Licence: that model is trained partly on ANLI, which is licensed CC BY-NC 4.0. Whether that restriction applies to the weights is unsettled, so treat it as non-commercial unless you've cleared it. Its manifest and docs/MODELS.md say so, and the comparison page marks it "non-commercial?". The zeroshot-v2.0 -c models, trained only on commercially licensed data, stay the recommended set.

NLI on an Intel Mac's GPU

Graded on the same Mac Pro (AMD GPU, macOS 26.6.2):

• bge-m3-zeroshot-v2.0-c grades B and passes every gate. It costs 234 ms per pair at 1,024 tokens and 425 ms at 2,048.
• deberta-v3-base-zeroshot-v2.0-c, nli-deberta-v3-base and ModernBERT-large-zeroshot-v2.0 grade F on bucket invariance only. This is the Intel GPU behaviour described in 0.9.0. Within a bucket they're at A or B level, and their MNLI predictions match fp32 on every pair.
• roberta-large-mnli also grades F, and should be avoided on an Intel Mac. Its probabilities move by up to 0.08 between buckets, so its scores depend on input length there.

Also

• EmbeddingGemma 2 now has its Intel Mac onnx-cpu grade: A. Its Linux grades are still to come.

What's Changed

• feat(tools): ModernBERT-large-zeroshot-v2.0 (licence-flagged) and the page's licence badge by @ironblock in #86
• docs(data): NLI models graded on an Intel Mac's GPU by @ironblock in #87
• feat(onnx): AVX-512 kernels for ONNX Runtime on Intel Macs (D42) by @ironblock in #88
• chore: release v0.11.0 by @ironblock in #89

Full Changelog: v0.10.0...v0.11.0

sidekick v0.10.0

sidekick v0.10.0 serves NLI cross-encoders as zero-shot classifiers. /v1/classify gains an nli format (D41) that scores exactly as Hugging Face's zero-shot-classification pipeline does. Ten NLI models convert and grade: five classics, an ANE build of one of them, and four modern MIT-licensed models. Every Core ML path matches its fp32 checkpoint's MNLI accuracy.

Upgrading

• HTTP API: additions only. On nli models, /v1/classify accepts a new hypothesis_template extension and multi_label. /v1/models lists hypothesis_template among an nli model's extensions, along with its default template. Other formats behave as before.
• A new classifier manifest format, [classify] format = "nli", with a [classify.nli] table (entailment, contradiction, n_classes, hypothesis_template). A 0.9.0 daemon skips an nli manifest as unparseable and serves the rest. Existing manifests need no change.
• Rust API changes:
  • A new variant, ClassifyFormat::Nli, breaks exhaustive matches on that enum.
  • Struct literals must add new fields: ClassifyParams { multi_label, hypothesis_template }, Prepared { nli } and ClassifySection { nli }.
  • New items: manifest::NliSection, NliPairs, NliPair, nli_value and check_hypothesis_template.
  • Nothing was removed.
• The C ABI and config keys are unchanged. Classifiers stay Core ML only, so the nli format runs on macOS: on Apple silicon, or on an Intel Mac's CPU and GPU.
• The release archives are the same four as 0.9.0, with the same bundled ONNX Runtime versions.

The nli format (D41)

An NLI cross-encoder judges whether a premise entails a hypothesis. As a zero-shot classifier, each candidate label becomes a hypothesis, such as "This example is {label}.", paired with the input text.

• Scoring is Hugging Face's pipeline, exactly:
  • With several labels and multi_label: false, probs is a softmax across labels of the entailment logits.
  • With multi_label: true, or exactly one label, each label is scored on its own as entailment against contradiction. With one label you get P(entailment).
• hypothesis_template must contain exactly one {}. Omit it to use the model's default.
• Truncation cuts only the input text, never the hypothesis. truncation_side: "left" keeps the end, which is where a stack trace or compiler error carries its message. In our end-to-end tests, a trace ending in a PermissionError was judged a success when cut from the right and a failure when cut from the left.
• Each (text, hypothesis) pair is one prediction, in the smallest bucket that fits. usage.prompt_tokens counts every pair.
• A known limit of these models, not of the conversion: in fp32 too, they read "212 passed, 3 failed" as "the tests passed". Keep a pattern for test counts and exit codes beside an NLI detector.

NLI models

Graded on an Apple M1 Max with macOS 27.0, on a shared NLI corpus: 339 requests and 419 pairs, including agent-style texts and long inputs truncated on both sides. Grades give the worst |Δp| against the fp32 checkpoint. MNLI is 308 validation pairs, matched and mismatched; every Core ML path's MNLI accuracy equals fp32.

model	served on	grade	ms per request	MNLI (matched / mismatched)
cross-encoder/nli-MiniLM2-L6-H768	GPU	A	4.0	0.877 / 0.864
cross-encoder/nli-MiniLM2-L6-H768, ANE build (nli-minilm2-l6-h768-ane)	ANE	B	2.4	0.883 / 0.870
cross-encoder/nli-deberta-v3-xsmall	GPU	B	4.2	0.844 / 0.851
cross-encoder/nli-deberta-v3-small	GPU	A	4.0	0.857 / 0.857
cross-encoder/nli-deberta-v3-base	GPU	B	6.7	0.896 / 0.883
FacebookAI/roberta-large-mnli	GPU	A	15.6	0.916 / 0.877
MoritzLaurer/deberta-v3-base-zeroshot-v2.0-c	GPU	A	6.7	0.942 / 0.916 (two-way)
MoritzLaurer/deberta-v3-large-zeroshot-v2.0-c	GPU	A	17	0.948 / 0.935 (two-way)
MoritzLaurer/roberta-large-zeroshot-v2.0-c	GPU	B	15.6	0.942 / 0.942 (two-way)
MoritzLaurer/bge-m3-zeroshot-v2.0-c, to 2,048 tokens	GPU	A	15.5 (208 at 2,048 tokens)	0.942 / 0.922 (two-way)

• Which to use. deberta-v3-base-zeroshot-v2.0-c is the best general choice on the GPU. For a detector running beside a GPU-served model, nli-minilm2-l6-h768-ane stays on the Neural Engine. For long inputs or other languages, use bge-m3-zeroshot-v2.0-c. Its tokenizer config caps input at 512 tokens, but the model takes 8,192 and sidekick serves it to 2,048.
• The ANE build uses three rewrites that are exact in fp32: a matmul softmax, erf GELU, and a rescale of each layer's output-linear inputs. Together they make the ANE bucket-invariant and take it from F to B. roberta-large-mnli reaches only C with the same rewrites, so it has no ANE build. The DeBERTa models are slower and less accurate on the ANE than on the GPU.
• Licensing. The -c models are trained only on commercially licensed data. Some other popular NLI models are trained partly on ANLI, which is CC BY-NC 4.0. tools/convert_nli.py converts BERT, RoBERTa, XLM-RoBERTa and DeBERTa-v3 cross-encoders, so whether to use such a model is your call.
• On an Intel Mac (Xeon W-3235, GPU), nli-minilm2-l6-h768 and nli-deberta-v3-small grade F, on bucket invariance only. This is the Intel GPU behaviour described in 0.9.0. Within a bucket they're accurate, and MNLI decisions agree with fp32 on every pair. nli-minilm2-l6-h768 takes 11.5 ms per hypothesis there with compute_units = "cpu_and_gpu".

What's Changed

• feat(classify): an NLI zero-shot format (D41) by @ironblock in #82
• feat(tools): five classic NLI cross-encoders for the nli format by @ironblock in #83
• feat(tools): modern NLI models (zeroshot-v2.0-c), and NLI graded on an Intel Mac by @ironblock in #84
• chore: release v0.10.0 by @ironblock in #85

Full Changelog: v0.9.0...v0.10.0

sidekick v0.9.0

sidekick v0.9.0 adds EmbeddingGemma 2, Google's Gemma 4 based embedder, which grades A on the Neural Engine, the GPU and ONNX Runtime's CPU path. A build now says which model backends it has. Grading Core ML on Intel Macs got a cause, a per-architecture gate and a clear list of what is supported there. Every graded model is now on a sortable, filterable comparison page, built from the same data as docs/MODELS.md.

Upgrading

• No breaking changes. Manifests, config files, the HTTP API and the C ABI from 0.8.0 work unchanged.
• New response fields, both additive:
  • /health has a top-level backends array listing the model backends compiled into this build, e.g. ["coreml", "onnx", "static"]. static is always present, coreml needs the coreml feature on macOS, and onnx needs the onnx feature.
  • sidekickd --version ends with the same list, e.g. (backends: coreml, onnx, static).
• Rust API: additions only.
  • A new crate, sidekick-onnx, holds ONNX Runtime's lookup, lazy load, sessions and the inference-thread rule. ort is a dependency of that crate alone, behind its runtime feature, which sidekick-embed's onnx feature turns on.
  • sidekick_embed::onnx re-exports everything that moved under the same names, so code written against 0.8.0 compiles unchanged.
  • New: sidekick_embed::compiled_backends().
• Builds: the coreml and onnx features and the build commands are unchanged. A build without onnx has no ort in its dependency tree, and CI checks that for sidekickd and libsidekick, as it already checked that Linux builds have no Core ML.
• The release archives are the same four as 0.8.0, with the same bundled ONNX Runtime versions: 1.28.3, and 1.23.2 on Intel macOS.

EmbeddingGemma 2

google/embeddinggemma-2 (Apache-2.0, ungated) is the text tower of a Gemma 4 based embedder: 271M parameters loaded text-only, 768 dimensions with Matryoshka 512/256/128, mean pooling, and the same task prompts as EmbeddingGemma 1. sidekick serves buckets up to 2,048 tokens; the model takes 8,192.

path	grade	worst-case cosine	ms per input
Core ML Neural Engine (served)	A	0.999995	10.1
Core ML GPU	A	0.9999993	14.1
Core ML CPU	F (bucket invariance only)	0.99993	28.0
ONNX Runtime CPU (1.28.3)	A	1 − 2e-12	31

Graded on an Apple M1 Max with macOS 27.0, on the parity suite's 51 cases, against sentence-transformers 6.1 in fp32. Grades on x86-64 will follow.

• Converting it needs a newer environment. tools/convert_embeddinggemma2.py needs transformers ≥ 5.19 and sentence-transformers ≥ 6.1, plus Pillow and torchvision, which its processor imports even for text-only loading. Keep that in a separate environment; the other converters stay on transformers 4.57.6 (docs/CONVERTING.md).
• Its sliding window is 512 tokens each side, inclusive and not halved, unlike EmbeddingGemma 1's. Every layer sees the whole input up to 513 tokens. The 2,048-token test cases exercise the band.
• fp16. The model's card warns that fp16 gives NaN or degraded embeddings. In sidekick's measurement the residual stream and attention scores fit fp16 comfortably. What overflowed was RMSNorm squaring its input: norm inputs reach about 1,600. Power-of-two pre-scales on every RMSNorm, GELU computed from tanh, and a pooling scale per bucket fix it. Without them, Core ML's CPU path lost 5%.
• Core ML's CPU path fails only the bucket-invariance gate. Its softmax rounds differently once the key axis passes 512 keys, so an embedding moves by about 6e-5 between buckets, while its accuracy is 0.99993. The fix for the CPU broke the Neural Engine, so the model is served on the Neural Engine.
• The ONNX export passes the model prepared attention masks, because transformers 5.19's mask functions don't trace. On a 1,970-token input where the window is live, it matches the checkpoint's own forward pass at cosine 1.0000000.

Core ML on Intel Macs

v0.8.0 noted that Core ML's CPU path on Intel isn't bucket-invariant. This release finds the cause, grades five more models, and settles what to recommend. Everything below was measured on one Mac Pro (Xeon W-3235, AMD GPU, macOS 26.6.2).

• The cause. Core ML's CPU attention-score matmul (Q×Kᵀ) on Intel computes the last, partial tile of its output by a different kernel path. For inputs that nearly fill their bucket, a few scores then round one fp16 ulp differently from the same scores in a wider bucket. It's deterministic and costs no accuracy. Apple silicon is bit-exact. tools/repro_cpu_matmul_tail.py reproduces it.

• The parity suite's CPU bucket gate is now per architecture. On x86-64 macOS it is cosine 0.99999 for embedders and Δp 5e-4 for classifiers and rerankers; the classifier value is provisional, since two classifiers measure it. Apple silicon stays exact, and the GPU and ANE gates are unchanged.

• Grades on Intel:

  model	Core ML CPU	Core ML GPU
  bge-small-en-v1.5	A	A
  nlptown-sentiment	A under the new gate	A
  ms-marco-minilm-l6-v2	B	A
  lumma-fev-0.1b	not graded (over 4 hours)	A
  gliner2.5-decide	not graded	F (bucket invariance)
  laya-typed-decisions	not graded	F (bucket invariance)
  agent-jev	not graded	not graded

• Compute units stay per model. On this Mac the GPU is 6–13× faster than the CPU on the small models, and as accurate or better. It is no more bucket-invariant, though: Core ML logs that MPSGraph doesn't support padded tensors on Intel Macs. [models."<id>"] compute_units = "cpu_and_gpu" selects it.

• Unsupported on Intel Macs:

  • gliner2.5-decide and laya-typed-decisions, as measured: they fail on the GPU and are impractically slow on the CPU;
  • laya-en and julia-1, by model family.

  The decision is recorded as an amendment to D38.

A comparison page for every graded model

ironblock.github.io/sidekick lists every graded model by path, precision, chip and runtime: its grade, measured error, latency and memory. It is sortable and filterable, and a "best path" mode picks one path per model and chip by accuracy, speed or size. A "keep off the GPU" option excludes GPU paths, for running beside a GPU-served model.

The page and the data tables in docs/MODELS.md are both built from one results file, docs/data/models.json, by tools/models_table.py. CI fails if the two disagree. Regenerating the tables corrected some drift: EmbeddingGemma 1's ONNX footprint is 2.7–3.4 GB over three runs, not 3.0–3.5.

What's Changed

• feat(docs): a model comparison page on GitHub Pages, from one results file by @ironblock in #75
• docs: link the model comparison page from the README and MODELS.md by @ironblock in #76
• feat(docs): generate MODELS.md's data tables from the results file by @ironblock in #77
• refactor: ONNX Runtime in its own crate, and report the backends a build serves by @ironblock in #78
• feat: google/embeddinggemma-2 on Core ML (ANE A) and ONNX Runtime (A) by @ironblock in #79
• test: reproduce Core ML's Intel CPU bucket drift; grade four more models on Intel by @ironblock in #80
• chore: release v0.9.0 by @ironblock in #81

Full Changelog: v0.8.0...v0.9.0