Skip to content

sidekick v0.17.0

Latest

Choose a tag to compare

@github-actions github-actions released this 30 Oct 00:11
· 4 commits to main since this release
0e851b4

sidekick v0.17.0 serves open chat models on the Neural Engine beside Apple Foundation Models, through the same /v1/chat/completions (D53). The first is MiniCPM5-2B, as minicpm5-2b: a build with rotations folded into its weights and GPTQ onto 8-bit palettes, run as a stateful Core ML decoder that keeps the GPU free for a larger model. It grades B against the original under a new rubric for chat models (D54, D55) and decodes at about 18.5 tokens/s on an M1 Max. Its graded build is published on Hugging Face as sidekickd/minicpm5-2b-coreml, and sidekickd fetch minicpm5-2b installs it. The chat route also changes for Foundation Models: one JSON Schema layer checks response_format for every chat backend and refuses what no backend can honor, guided replies keep the schema's property order, seed and top_p are honored, and temperature: 0 is greedy. Embedding, classification and reranking requests behave as in 0.16.0.

Upgrading

  • /health's chat section is reshaped, a breaking change for anything that reads it. The single object (chat.model, chat.availability, chat.context_limit, chat.variant, chat.variant_id, chat.model_capabilities, chat.fm_sdk) moves to chat.models["apple-fm"] with the same fields, plus backend, request_timeout_secs and request_timeout_source. chat.model is gone: the key is the model's id. chat.lane is new, and null unless an open model is installed. An open model's entry adds, among others, its state, load_error, queue_depth and lane_queue_depth, request and load counts, its decoder's calls and stalls, its Neural Engine share (ane) and its grammar engine's state (guided). A new top-level request_timeout_overrides lists the per-model timeouts applied and those that name no model served. docs/API.md, "Listing models and health", describes each field.
  • response_format schemas are checked against one subset, and the rest is a 400 (#179). A keyword outside it is a 400 unsupported_schema whose message names the keyword and its JSON pointer, on apple-fm as on open models. Schemas the Foundation Models bridge accepted while dropping parts of them are now refused: among others minimum, maximum, pattern, format, minLength, $ref and $defs, oneOf and allOf, a union of types, enum of non-strings, an open object, and an array without items. The subset is in docs/API.md, "Structured output" (D8 and D53).
  • Guided replies on apple-fm follow the schema's property order (#180). Foundation Models now generates properties in the order the schema lists them (the bridge sorted them by name before), and the reply is re-serialized as compact JSON on one line with its keys in that order ({"name":"Ada","age":36}), so identical requests at temperature 0 give identical bytes. The generation order can change what the model writes: put the fields it should decide first first. A nullable property ("type": [T, "null"], or anyOf with {"type": "null"}) is now supported as an object property's value, and a required nullable property the model leaves out comes back as null.
  • Sampling on apple-fm. temperature: 0, or top_p: 0, now selects Foundation Models' greedy mode, so a request repeats byte for byte on the same Mac and macOS; it used to set a temperature of 0 on the default sampling. top_p and seed, which were ignored, are honored through Foundation Models' random sampling mode (seed alone samples the whole distribution, top_p 1). A request with none of the three samples as before (D22 amended; docs/API.md, "Sampling").
  • Range checks: temperature outside 0–2 and top_p outside 0–1 are a 400 invalid_request_error, on every chat model.
  • Guardrail refusals keep their status and code, a 400 content_filter, but the message now names the model and says whether output was guided: "apple-fm's guardrails refused this request, whose output was plain text: …", where it was "The on-device model's guardrails rejected this request: …". A refused guided request also carries OpenAI's param: "response_format", since guided output can be refused where the same prompt as plain text isn't (#178); no other error sets param. A refusal after text has streamed now sets the final chunk's constrained to whether output was guided. The daemon never retries a refusal or answers it with another model.
  • Installing an open model changes how apple-fm runs. With none installed, apple-fm runs as before. With one, apple-fm and the open models share one queue on the Neural Engine, the ANE chat lane, because a Foundation Models request overlapping a generation slows both about 25×: requests run one at a time in arrival order; past queue_depth waiting (default 4), a request is a 503 backend_busy with Retry-After: 5; time in the queue counts against the request timeout; and a request that timed out keeps its turn until Foundation Models has stopped generating, which for a non-streamed request can take tens of seconds (#197).
  • A streamed chat request whose client disconnects gives up its place in a queue or a load wait, on every chat model, apple-fm included.
  • HTTP API: other changes are additions. /v1/models lists each chat model's backend (foundation_models for apple-fm, coreml for an open model), context_length (prompt and completion together), honored_fields, guided_streaming and guardrails, and for an open model its compute_units, graded_os and its decoder's compile_cache. A request to an open model whose decoder hung is a 503 backend_unavailable with the reason. JSON objects the daemon assembles itself, such as /health, now keep their keys in the order they are written instead of sorted by name.
  • Config keys, all optional: ane_budget_mib (default 2,816), the Neural Engine footprint past which loading an open model logs a warning; [models."<id>"] request_timeout_secs, one model's request timeout over the global one, for any model (chat, embedder or classifier; at least 1); and, for open chat models, [models."<id>"] queue_depth (default 4) and schema_in_prompt (default true). model_idle_ttl_secs also unloads an idle open model. docs/CONFIG.md lists them.
  • Manifests: a third kind, chat.toml, for an open chat model (task = "text-generation", interface = "kv-state-1"): its decoder's chunk programs, the embedding and RoPE tables the host reads, the tokenizer, the checkpoint's chat template, sampling defaults and a default request timeout (docs/design/open-llm.md, "The manifest"). Unknown keys are refused, so a manifest that asks for something the daemon can't honor is skipped, with the reason in /health's skipped_models. Open models are skipped too in a build without Core ML (the Linux archives), on a Mac without a Neural Engine, and when the config's compute_units for them is anything but cpu_and_ne. The id apple-fm is reserved. manifest.toml and classifier.toml are unchanged.
  • Command line: no new flags. sidekickd fetch installs a chat build whole (--buckets doesn't apply), and sidekickd warm and the startup warm-up (D52) compile an open model's decoder as they compile any Core ML bucket.
  • C ABI: unchanged. SK_ABI_VERSION is the same; libsidekick.dylib serves no chat.
  • Rust API: additions, with source-level changes. sidekick_core::ChatRequest gains top_p, seed, deadline and hold, and its schema is a validated sidekick_core::JsonSchema instead of a serde_json::Value. sidekick_core::Error gains Busy, LoadTimeout and UnsupportedSchema, which an exhaustive match must handle. The stop-sequence shaper moves from sidekick_fm::shaping to sidekick_core::shaping. New: the sidekick-llm crate; sidekick_core::chat_manifest, sidekick_core::schema, ResolvedChat, WorkHold, FOUNDATION_MODELS_ID and ModelRegistry's chat, chat_ids, chats, skip_chat and skip_chats; sidekick_coreml::KvDecoder and its program contract; and sidekick_embed::warm's load_with, warm_with, refuse and refusal. The workspace enables serde_json's preserve_order feature, which Cargo's feature unification turns on for every crate in a build that includes sidekick's: a serde_json::Map there keeps insertion order instead of sorting its keys.
  • The release archives are the same four as 0.16.0, with the same bundled ONNX Runtime versions, and the macOS archives' sidekickd keeps its signing identifier, io.github.ironblock.sidekickd. sidekickd now statically links llguidance (MIT) and minijinja (Apache-2.0); no new library ships beside it. THIRD_PARTY_NOTICES.md adds the MiniCPM5-2B and SmolLM2-135M-Instruct chat templates and the Wikipedia excerpts that the test fixtures, the converter's calibration and the chat benchmark use; none of them is compiled into sidekickd.

Open chat models on the Neural Engine (D53)

sidekick is for the work beside a large model that owns the GPU. Its chat route served only Apple's Foundation Models, whose guided generation drops optional fields (#177) and whose guardrails refuse guided output on ordinary text (#178). The chat benchmark (docs/CHAT-BENCHMARK.md) found MiniCPM5-2B level with Foundation Models on titles, fixed-vocabulary tags and instructions, well ahead on extraction, and refusing nothing. This release runs it on the Neural Engine.

  • Choosing a model: a request goes to the chat model its model names, exactly: there is no default, no alias and no fallback from one model to another. /v1/models lists what each chat model honors, so a client chooses.
  • The build: four chunk programs of 14, 14, 10 and 4 layers, each one 16-token function that keeps its layers' KV cache in Core ML state, over a 2,048-position cache, so the context is 2,033 tokens, prompt and completion together. 25 files, 2.63 GiB to download.
  • The grade: B on the Neural Engine under D54, by each family's mean KL divergence from fp32, its 99th percentile and top-1 agreement: prose 1.28e-3 / 1.17e-2 / 99.57%, repository 2.89e-4 / 2.07e-3 / 98.93%. It passes the task check against fp32 on the chat benchmark. Graded on an Apple M1 Max with macOS 27.0, which /v1/models reports as graded_os (docs/MODELS.md, "Chat models").
  • Speed and memory, on an M1 Max: about 18.5 tokens/s unguided, a 54 ms call per token, and 25–83 tokens/s for guided JSON, where the grammar forces part of the reply and forced tokens ride in the same call. Its programs take 2,185 MiB of the Neural Engine's budget, which the whole machine shares and Foundation Models' own model (about 2.3 GiB) counts against; one open model is resident at a time (docs/design/open-llm.md, "Memory budget").
  • A request is rendered with the checkpoint's own chat template, as transformers renders it, with thinking off. The text of special tokens in message content is rewritten, so a message can't forge a turn. usage is exact, streaming sends whole UTF-8 characters as they are generated, and a prompt plus max_tokens past the context is a 400 context_length_exceeded. Sampling defaults are the checkpoint's (temperature 1.0, top_p 0.95), so greedy isn't the default. The request timeout is 120 s, from the manifest. There are no guardrails. docs/API.md, "Open models", has the contract.
  • Guided JSON (response_format json_schema) is enforced with llguidance: each token is sampled from those the schema's grammar allows, so a reply that ends with finish_reason: "stop" is valid JSON matching the schema, on one line with its keys in schema order and , and : between values, the spacing the model writes unprompted. It streams as it is generated. The schema, with its descriptions, is also written into the system message, since the grammar only forces what may come next; schema_in_prompt = false leaves the prompt as sent. Valid JSON isn't correct values: constraining a model fixes its malformed replies, not its wrong answers.
  • A hung Neural Engine call (a 60 s watchdog) marks the model failed: its requests are a 503 backend_unavailable with the reason, refused before they queue (#202), the lane is freed for everyone else, and a restart recovers it.
  • Installing: sidekickd fetch minicpm5-2b installs the graded build from sidekickd/minicpm5-2b-coreml (2.83 GB), published under the notice tier for its training data (docs/PUBLISHING.md), and verifies every file against the built-in checksum ledger as usual. To build it yourself, docs/CONVERTING.md, "Causal language models", has the recipe.

Known limits

  • Other work on the Neural Engine stalls a generation. Foundation Models used by other apps or by macOS, and other apps' Neural Engine work, slow an open model's calls many times over while they run. sidekickd counts the stalls in /health and the request timeout bounds the worst case, but it can't prevent them (docs/BACKENDS.md, "Open chat models on the Neural Engine").
  • An install costs about 2.14 GiB of disk that can't be reclaimed. The first load compiles the decoder for the Neural Engine, and macOS keeps the compiled program, about one copy of the weights, in a system cache no user can delete. It is paid once per install and again after each macOS update.
  • The context is 2,033 tokens.
  • Not supported yet: tool calling (#209) and logprobs (#208), both a 400; frequency_penalty and presence_penalty, which are ignored (#207); top_k, min_p and logit_bias (#214); prefix caching (#206).
  • Graded on macOS 27 only, and Apple silicon only: no GPU path, no Intel Macs.
  • Re-fetching a chat build replaces an edited chat.toml (#216), and editing a hard-linked installed file such as chat_template.jinja in place also changes the Hugging Face cache's copy (#217).

Grading chat models (D54, D55)

  • A chat build is graded by KL divergence, not cosine (D54). A chat model samples a distribution, and a logit vector's cosine weighs all 130,560 vocabulary entries alike; on MiniCPM5-2B the two barely correlated. The grade is the KL divergence of the build's next-token distribution from fp32's, its mean and 99th percentile, with top-1 agreement as a floor, per family of held-out texts, and the build earns its worst family's letter. Both calibration draws must land in the same band. A task check runs the build through sidekickd and the checkpoint in fp32 on a seeded sample of the chat benchmark, and each family's score may fall short of fp32's by at most a fixed tolerance. The bands are in docs/CONVERTING.md, "Grading"; the encoders' and classifiers' rubric (D26) is unchanged.
  • Two precision rewrites for the Neural Engine (D55). The ANE added up to 3.1 times the PyTorch screen's divergence on long chat-framed text, nearly all of it in the first chunk, from two instances of the linear's small-input floor (D17): the value matmul, whose probabilities shrink as the context grows, and the early output projections. The converter scales both by powers of two with an explicit descale, which leaves the GPTQ weights unchanged; the prose family went from C to B.

Measurements and tooling

  • The causal-LM converter (tools/convert_causal_lm.py, in tools/sidekick_convert) converts a Llama-architecture checkpoint into a chat.toml build: rotations, GPTQ onto palettes, calibration on a committed fixture disjoint from what is graded, a PyTorch screen before any Neural Engine compile, and a compute-plan check that every operation is on the ANE. tools/grade_causal_lm.py grades the installed build on the Neural Engine (docs/CONVERTING.md, "Causal language models").
  • The chat benchmark (docs/CHAT-BENCHMARK.md) compares Foundation Models with small open models on titles, tags, extraction and short instructions, guided and prompt-only, with its committed results.
  • The checksum ledger, model cards and MODELS.md cover chat builds: a chat build's D54 grade and task check attach to its files, ledger.py verify checks an install, tools/model_card.py writes its card with the measured compile cost, and MODELS.md's "Chat models" table is generated from the ledger.
  • The release gate covers chat models: the first 12 items of each chat-benchmark family, prompt-only and guided, at temperature 0, run twice; replies must be byte-identical between the runs, guided replies must satisfy their schema, usage must match the build's tokenizer and template, and each family's score is held to the baseline. This release passed it.
  • tools/ane_budget_probe.py times a decoder's calls while other Neural Engine work runs beside it, the measurement behind the ANE chat lane and ane_budget_mib.

What's Changed

  • feat(tools): chat benchmark of Foundation Models against small open models by @ironblock in #176
  • docs(design): open LLMs on the chat route, beside Foundation Models (D53) by @ironblock in #189
  • feat(server): route chat by model id, with per-model timeouts by @ironblock in #190
  • test(server): widen the per-model timeout test's timing margin by @ironblock in #193
  • feat(chat): one JSON Schema layer for every chat backend; seed and top_p on Foundation Models by @ironblock in #191
  • docs(design): schedule the ANE chat models from the measured budget (D53) by @ironblock in #192
  • feat(coreml): a stateful kv-state-1 decoder chain on the Neural Engine (#183) by @ironblock in #194
  • feat(llm): serve open chat models on /v1/chat/completions through the ANE decoder (#185) by @ironblock in #196
  • feat(llm): constrained decoding for open chat models with llguidance (#186) by @ironblock in #199
  • feat(convert): the causal-LM converter; MiniCPM5-2B grades C on chat-wrapped prose by @ironblock in #195
  • fix(llm): refuse a hung model's requests before they queue for the lane by @ironblock in #202
  • docs(notices): list the chat benchmark's Wikipedia excerpts by @ironblock in #200
  • feat(tools): the ANE budget probe, on the kv_decoder runner by @ironblock in #201
  • feat(convert): grade chat models by KL divergence and a task check (D54); MiniCPM5-2B grades C by @ironblock in #204
  • fix(convert): scale the causal LM's value matmul and output projection inputs for the ANE by @ironblock in #205
  • feat(publishing): prepare MiniCPM5-2B's graded build for publishing by @ironblock in #215
  • chore(release): 0.17.0 by @ironblock in #218

Full Changelog: v0.16.0...v0.17.0