sidekick v0.10.0

sidekick v0.10.0 serves NLI cross-encoders as zero-shot classifiers. /v1/classify gains an nli format (D41) that scores exactly as Hugging Face's zero-shot-classification pipeline does. Ten NLI models convert and grade: five classics, an ANE build of one of them, and four modern MIT-licensed models. Every Core ML path matches its fp32 checkpoint's MNLI accuracy.

Upgrading

• HTTP API: additions only. On nli models, /v1/classify accepts a new hypothesis_template extension and multi_label. /v1/models lists hypothesis_template among an nli model's extensions, along with its default template. Other formats behave as before.
• A new classifier manifest format, [classify] format = "nli", with a [classify.nli] table (entailment, contradiction, n_classes, hypothesis_template). A 0.9.0 daemon skips an nli manifest as unparseable and serves the rest. Existing manifests need no change.
• Rust API changes:
  • A new variant, ClassifyFormat::Nli, breaks exhaustive matches on that enum.
  • Struct literals must add new fields: ClassifyParams { multi_label, hypothesis_template }, Prepared { nli } and ClassifySection { nli }.
  • New items: manifest::NliSection, NliPairs, NliPair, nli_value and check_hypothesis_template.
  • Nothing was removed.
• The C ABI and config keys are unchanged. Classifiers stay Core ML only, so the nli format runs on macOS: on Apple silicon, or on an Intel Mac's CPU and GPU.
• The release archives are the same four as 0.9.0, with the same bundled ONNX Runtime versions.

The nli format (D41)

An NLI cross-encoder judges whether a premise entails a hypothesis. As a zero-shot classifier, each candidate label becomes a hypothesis, such as "This example is {label}.", paired with the input text.

• Scoring is Hugging Face's pipeline, exactly:
  • With several labels and multi_label: false, probs is a softmax across labels of the entailment logits.
  • With multi_label: true, or exactly one label, each label is scored on its own as entailment against contradiction. With one label you get P(entailment).
• hypothesis_template must contain exactly one {}. Omit it to use the model's default.
• Truncation cuts only the input text, never the hypothesis. truncation_side: "left" keeps the end, which is where a stack trace or compiler error carries its message. In our end-to-end tests, a trace ending in a PermissionError was judged a success when cut from the right and a failure when cut from the left.
• Each (text, hypothesis) pair is one prediction, in the smallest bucket that fits. usage.prompt_tokens counts every pair.
• A known limit of these models, not of the conversion: in fp32 too, they read "212 passed, 3 failed" as "the tests passed". Keep a pattern for test counts and exit codes beside an NLI detector.

NLI models

Graded on an Apple M1 Max with macOS 27.0, on a shared NLI corpus: 339 requests and 419 pairs, including agent-style texts and long inputs truncated on both sides. Grades give the worst |Δp| against the fp32 checkpoint. MNLI is 308 validation pairs, matched and mismatched; every Core ML path's MNLI accuracy equals fp32.

model	served on	grade	ms per request	MNLI (matched / mismatched)
cross-encoder/nli-MiniLM2-L6-H768	GPU	A	4.0	0.877 / 0.864
cross-encoder/nli-MiniLM2-L6-H768, ANE build (nli-minilm2-l6-h768-ane)	ANE	B	2.4	0.883 / 0.870
cross-encoder/nli-deberta-v3-xsmall	GPU	B	4.2	0.844 / 0.851
cross-encoder/nli-deberta-v3-small	GPU	A	4.0	0.857 / 0.857
cross-encoder/nli-deberta-v3-base	GPU	B	6.7	0.896 / 0.883
FacebookAI/roberta-large-mnli	GPU	A	15.6	0.916 / 0.877
MoritzLaurer/deberta-v3-base-zeroshot-v2.0-c	GPU	A	6.7	0.942 / 0.916 (two-way)
MoritzLaurer/deberta-v3-large-zeroshot-v2.0-c	GPU	A	17	0.948 / 0.935 (two-way)
MoritzLaurer/roberta-large-zeroshot-v2.0-c	GPU	B	15.6	0.942 / 0.942 (two-way)
MoritzLaurer/bge-m3-zeroshot-v2.0-c, to 2,048 tokens	GPU	A	15.5 (208 at 2,048 tokens)	0.942 / 0.922 (two-way)

• Which to use. deberta-v3-base-zeroshot-v2.0-c is the best general choice on the GPU. For a detector running beside a GPU-served model, nli-minilm2-l6-h768-ane stays on the Neural Engine. For long inputs or other languages, use bge-m3-zeroshot-v2.0-c. Its tokenizer config caps input at 512 tokens, but the model takes 8,192 and sidekick serves it to 2,048.
• The ANE build uses three rewrites that are exact in fp32: a matmul softmax, erf GELU, and a rescale of each layer's output-linear inputs. Together they make the ANE bucket-invariant and take it from F to B. roberta-large-mnli reaches only C with the same rewrites, so it has no ANE build. The DeBERTa models are slower and less accurate on the ANE than on the GPU.
• Licensing. The -c models are trained only on commercially licensed data. Some other popular NLI models are trained partly on ANLI, which is CC BY-NC 4.0. tools/convert_nli.py converts BERT, RoBERTa, XLM-RoBERTa and DeBERTa-v3 cross-encoders, so whether to use such a model is your call.
• On an Intel Mac (Xeon W-3235, GPU), nli-minilm2-l6-h768 and nli-deberta-v3-small grade F, on bucket invariance only. This is the Intel GPU behaviour described in 0.9.0. Within a bucket they're accurate, and MNLI decisions agree with fp32 on every pair. nli-minilm2-l6-h768 takes 11.5 ms per hypothesis there with compute_units = "cpu_and_gpu".

What's Changed

• feat(classify): an NLI zero-shot format (D41) by @ironblock in #82
• feat(tools): five classic NLI cross-encoders for the nli format by @ironblock in #83
• feat(tools): modern NLI models (zeroshot-v2.0-c), and NLI graded on an Intel Mac by @ironblock in #84
• chore: release v0.10.0 by @ironblock in #85

Full Changelog: v0.9.0...v0.10.0