sidekick v0.12.0

sidekick v0.12.0 serves chunked Core ML embedders, so Qwen3-Embedding-0.6B now runs on the Neural Engine, where it grades A. It refuses an all-zero or non-finite embedding instead of serving it, after Core ML's CPU path on an M4 Max was measured returning zeros. It also logs which config file it read, and computes every published grade from a single definition.

Upgrading

• HTTP API and C ABI: unchanged in shape. Two kinds of request now get an error where they used to get a result:
  • an embedding whose output is all zeros or non-finite now gets a 500 internal_error instead of a 200 (see below);
  • an input that tokenizes to nothing, which only a tokenizer that adds no special tokens can produce, now gets a 400 on every backend, where ONNX models answered 500.
• Config keys: unchanged. The default config file on macOS is ~/Library/Application Support/sidekick/config.toml. On Linux it's ~/.config/sidekick/config.toml. --help and the README used to give only the Linux path, so a config written there on a Mac was ignored. The sidekickd starting log line now names the file it read, or the path it looked for.
• Manifests: a Core ML embedder may now carry [chunking] and a {chunk} artifact placeholder, as classifiers already could (D37). Static and ONNX embedders still reject both.
• Rust API: additions only. sidekick_embed::degenerate reports whether an embedding is unservable. sidekick_server::Config::resolve_path says which config file load reads. Nothing was removed or changed.
• Qwen3-Embedding-0.6B's served path is now Core ML on the ANE. MODELS.md and the comparison page list its chunked Core ML build, where they listed onnx-cpu. onnx-cpu stays its route on Linux and on Intel Macs, which have no ANE.
• The release archives are the same four as 0.11.0, with the same bundled ONNX Runtime versions.

Chunked embedders: Qwen3-Embedding-0.6B on the Neural Engine (D37)

Qwen3-Embedding-0.6B carries 1.11 GiB of fp16 weights per bucket. That is over the 1 GiB a single program could carry onto the ANE on an M1 Max (D32), so the model had no Core ML build. The embedder converter now splits each bucket into two programs of about 570 MB, and sidekickd serves the chain. At load, it checks that the chunks connect: the inputs, hidden_out to hidden_in, and the last chunk's output.

• Splitting changes no arithmetic. In fp32, the chunks equal the unsplit model exactly, and the GPU chain is bit-identical to the unsplit GPU program in every bucket. F2LLM-v2-160M, which isn't chunked, reconverts byte-identically.
• Grades on an M1 Max under macOS 27.0: A on the ANE (worst cosine 0.999986), A on the GPU (0.999997), and B on the CPU (0.999684). The compute plan puts 2,129 of its 2,135 operations on the ANE.
• Grades on an M4 Max under macOS 26.5.2: the ANE and GPU grades are the same. The CPU's is not; see below.

All-zero embeddings are refused

On an M4 Max under macOS 26.5.2, Core ML's CPU engine returns all-zero embeddings from Qwen3-Embedding-0.6B for some inputs, chunked or not. Truncating the model shows the depth involved: at 23 of its 28 layers every output is non-zero, and from 24 layers some are zero. Such an output is finite, so an unchunked build served with cpu_only under 0.11.0 would have answered with a 200. The same program compiled for fp32 is healthy on that machine, and so is onnx-cpu (A, 29 ms per input). The M1 Max under macOS 27.0 is unaffected.

• The Core ML and ONNX embedders now return an error when an output has a non-finite value, or when every value is zero. The error names the model, the bucket and the compute units. When the CPU returns zeros, the error also suggests other compute units. Static embedders are unaffected.
• On an M4-class Mac, serve Qwen3-Embedding-0.6B on its default, the ANE, or on the GPU, not with a cpu_only override.
• The converter now reports "all-zero output" for such cases, where it used to report a NaN parity cosine.
• ane_check reports non-finite and all-zero output as such, instead of blaming the attention mask.

The ANE weight limit depends on the chip or the OS

The same M4 Max under macOS 26.5.2 runs Qwen3-Embedding-0.6B's unsplit 1.110 GiB program on the ANE. Its compute plan puts 2,124 of 2,129 ops there, and the program takes 15 ms per input against 43 ms on the CPU. The M1 Max under macOS 27.0 puts none of it there. The two machines differ in both chip and OS, so it is open which of them moved the limit. The 1 GiB constant (D32) stays, as the conservative choice. Chunking keeps a chunkable model's programs under it, losslessly and at under 2% on the GPU. Deciding at load from the machine's own compute plan is tracked as #102.

Grades from one definition

• One table: the grade thresholds are [grades] in fixtures/parity/expectations.toml. The parity suite and tools/models_table.py both read it.
• Computed letters: every published letter is computed from its numbers, and CI fails on any row whose stored letter disagrees.
• Shared text: docs/MODELS.md and the comparison page share one formatter and generated legends.
• No grade changed. The check uncovered one mismatch, which was fixed by re-grading: nlptown-sentiment on an Intel Mac's Core ML CPU path is A under the current Intel bucket gate, with the same artifacts and measurements.

Also

• The converter checks early that coremltools has its native libraries. coremltools 9.0 has no Python 3.14 wheel. The package pip installs there used to fail only after a full conversion. CONVERTING.md states the supported Python versions.
• The converter's errors: the "can't be chunked" error names the backbone class. The over-the-limit error offers --chunks auto only for a backbone that can chunk.
• New and corrected docs:
  • CONVERTING.md cites the community ANE reference (hollance/neural-engine). It says where Core ML's diagnostics are logged, and notes that GPU grades assume Core ML's default fp32 accumulation.
  • It also notes that the onnxruntime used for exports needn't match the daemon's bundled runtime.
• tools/bench_threads.py, the sweep behind MODELS.md's thread table, is committed.

What's Changed

• fix(convert): honest chunking errors and an early coremltools check by @ironblock in #93
• fix(server): document the real default config path, and log the one read by @ironblock in #97
• One definition of the grades: computed letters, shared scales, one formatter by @ironblock in #98
• docs(converting): the export's onnxruntime needn't match the daemon's by @ironblock in #99
• feat: chunked Core ML embedders, and Qwen3-Embedding-0.6B on the ANE (D37) by @ironblock in #100
• fix(embed): refuse an all-zero or non-finite embedding instead of serving it by @ironblock in #101
• docs: the M4 Max's ANE limit and its zero-output CPU path, and the thread sweep tool by @ironblock in #107
• docs(converting): the community ANE reference, Core ML's log, GPU accumulation by @ironblock in #105
• chore(release): 0.12.0 by @ironblock in #108

Full Changelog: v0.11.0...v0.12.0