sidekick v0.9.0

sidekick v0.9.0 adds EmbeddingGemma 2, Google's Gemma 4 based embedder, which grades A on the Neural Engine, the GPU and ONNX Runtime's CPU path. A build now says which model backends it has. Grading Core ML on Intel Macs got a cause, a per-architecture gate and a clear list of what is supported there. Every graded model is now on a sortable, filterable comparison page, built from the same data as docs/MODELS.md.

Upgrading

• No breaking changes. Manifests, config files, the HTTP API and the C ABI from 0.8.0 work unchanged.
• New response fields, both additive:
  • /health has a top-level backends array listing the model backends compiled into this build, e.g. ["coreml", "onnx", "static"]. static is always present, coreml needs the coreml feature on macOS, and onnx needs the onnx feature.
  • sidekickd --version ends with the same list, e.g. (backends: coreml, onnx, static).
• Rust API: additions only.
  • A new crate, sidekick-onnx, holds ONNX Runtime's lookup, lazy load, sessions and the inference-thread rule. ort is a dependency of that crate alone, behind its runtime feature, which sidekick-embed's onnx feature turns on.
  • sidekick_embed::onnx re-exports everything that moved under the same names, so code written against 0.8.0 compiles unchanged.
  • New: sidekick_embed::compiled_backends().
• Builds: the coreml and onnx features and the build commands are unchanged. A build without onnx has no ort in its dependency tree, and CI checks that for sidekickd and libsidekick, as it already checked that Linux builds have no Core ML.
• The release archives are the same four as 0.8.0, with the same bundled ONNX Runtime versions: 1.28.3, and 1.23.2 on Intel macOS.

EmbeddingGemma 2

google/embeddinggemma-2 (Apache-2.0, ungated) is the text tower of a Gemma 4 based embedder: 271M parameters loaded text-only, 768 dimensions with Matryoshka 512/256/128, mean pooling, and the same task prompts as EmbeddingGemma 1. sidekick serves buckets up to 2,048 tokens; the model takes 8,192.

path	grade	worst-case cosine	ms per input
Core ML Neural Engine (served)	A	0.999995	10.1
Core ML GPU	A	0.9999993	14.1
Core ML CPU	F (bucket invariance only)	0.99993	28.0
ONNX Runtime CPU (1.28.3)	A	1 − 2e-12	31

Graded on an Apple M1 Max with macOS 27.0, on the parity suite's 51 cases, against sentence-transformers 6.1 in fp32. Grades on x86-64 will follow.

• Converting it needs a newer environment. tools/convert_embeddinggemma2.py needs transformers ≥ 5.19 and sentence-transformers ≥ 6.1, plus Pillow and torchvision, which its processor imports even for text-only loading. Keep that in a separate environment; the other converters stay on transformers 4.57.6 (docs/CONVERTING.md).
• Its sliding window is 512 tokens each side, inclusive and not halved, unlike EmbeddingGemma 1's. Every layer sees the whole input up to 513 tokens. The 2,048-token test cases exercise the band.
• fp16. The model's card warns that fp16 gives NaN or degraded embeddings. In sidekick's measurement the residual stream and attention scores fit fp16 comfortably. What overflowed was RMSNorm squaring its input: norm inputs reach about 1,600. Power-of-two pre-scales on every RMSNorm, GELU computed from tanh, and a pooling scale per bucket fix it. Without them, Core ML's CPU path lost 5%.
• Core ML's CPU path fails only the bucket-invariance gate. Its softmax rounds differently once the key axis passes 512 keys, so an embedding moves by about 6e-5 between buckets, while its accuracy is 0.99993. The fix for the CPU broke the Neural Engine, so the model is served on the Neural Engine.
• The ONNX export passes the model prepared attention masks, because transformers 5.19's mask functions don't trace. On a 1,970-token input where the window is live, it matches the checkpoint's own forward pass at cosine 1.0000000.

Core ML on Intel Macs

v0.8.0 noted that Core ML's CPU path on Intel isn't bucket-invariant. This release finds the cause, grades five more models, and settles what to recommend. Everything below was measured on one Mac Pro (Xeon W-3235, AMD GPU, macOS 26.6.2).

• The cause. Core ML's CPU attention-score matmul (Q×Kᵀ) on Intel computes the last, partial tile of its output by a different kernel path. For inputs that nearly fill their bucket, a few scores then round one fp16 ulp differently from the same scores in a wider bucket. It's deterministic and costs no accuracy. Apple silicon is bit-exact. tools/repro_cpu_matmul_tail.py reproduces it.

• The parity suite's CPU bucket gate is now per architecture. On x86-64 macOS it is cosine 0.99999 for embedders and Δp 5e-4 for classifiers and rerankers; the classifier value is provisional, since two classifiers measure it. Apple silicon stays exact, and the GPU and ANE gates are unchanged.

• Grades on Intel:

  model	Core ML CPU	Core ML GPU
  bge-small-en-v1.5	A	A
  nlptown-sentiment	A under the new gate	A
  ms-marco-minilm-l6-v2	B	A
  lumma-fev-0.1b	not graded (over 4 hours)	A
  gliner2.5-decide	not graded	F (bucket invariance)
  laya-typed-decisions	not graded	F (bucket invariance)
  agent-jev	not graded	not graded

• Compute units stay per model. On this Mac the GPU is 6–13× faster than the CPU on the small models, and as accurate or better. It is no more bucket-invariant, though: Core ML logs that MPSGraph doesn't support padded tensors on Intel Macs. [models."<id>"] compute_units = "cpu_and_gpu" selects it.

• Unsupported on Intel Macs:

  • gliner2.5-decide and laya-typed-decisions, as measured: they fail on the GPU and are impractically slow on the CPU;
  • laya-en and julia-1, by model family.

  The decision is recorded as an amendment to D38.

A comparison page for every graded model

ironblock.github.io/sidekick lists every graded model by path, precision, chip and runtime: its grade, measured error, latency and memory. It is sortable and filterable, and a "best path" mode picks one path per model and chip by accuracy, speed or size. A "keep off the GPU" option excludes GPU paths, for running beside a GPU-served model.

The page and the data tables in docs/MODELS.md are both built from one results file, docs/data/models.json, by tools/models_table.py. CI fails if the two disagree. Regenerating the tables corrected some drift: EmbeddingGemma 1's ONNX footprint is 2.7–3.4 GB over three runs, not 3.0–3.5.

What's Changed

• feat(docs): a model comparison page on GitHub Pages, from one results file by @ironblock in #75
• docs: link the model comparison page from the README and MODELS.md by @ironblock in #76
• feat(docs): generate MODELS.md's data tables from the results file by @ironblock in #77
• refactor: ONNX Runtime in its own crate, and report the backends a build serves by @ironblock in #78
• feat: google/embeddinggemma-2 on Core ML (ANE A) and ONNX Runtime (A) by @ironblock in #79
• test: reproduce Core ML's Intel CPU bucket drift; grade four more models on Intel by @ironblock in #80
• chore: release v0.9.0 by @ironblock in #81

Full Changelog: v0.8.0...v0.9.0