Repository navigation
v0.11.1
lattice v0.11.1
Patch release. No breaking changes to the documented API: code that compiles against 0.11.0 compiles unchanged against it. Two items that are hidden from the documentation changed shape, and they are listed at the end of these notes.
The headline is Gemma 4 text serving on the CPU from both HTTP servers, request-level thinking control for chat requests, and a pass that moves model families behind serving providers. The GPU and CPU decode paths also move onto the shared decoder driver, and lattice-tune gains adapter loading and batch generation options.
New: Gemma 4 text serving
lattice serve and lattice_serve can now serve a Gemma 4 E2B text checkpoint on the CPU backend through the shared decoder driver, streaming and non-streaming.
lattice servetakes the family from the top-levelmodel_typeofconfig.json; only a positivegemma4selects Gemma, and a missing, unreadable or malformed config keeps the Qwen path the directory had before (#1856).- A Gemma 4 checkpoint in the Q4 format is refused at startup with the stable code
gemma_metal_unsupportedbefore any loader runs, whatever features the binary was built with (#1856). lattice_serveselects the family and backend through the same detection and route table, and refuses a Gemma 4 Q4 checkpoint the same way (#1860).- Requests Gemma 4 cannot serve, and startup options that configure them, return stable codes documented in
docs/serve-http-api.md(#1860). lattice serveprints[route] selected family=<f> backend=<b> format=<fmt>at startup, and prints[route] served family=<f> backend=cpu mode=<stream|nonstream> driver=<shared|bypassed> opened=<n> consumed=<n>for each CPU-served request (#1856).lattice_serveprints the same startup line, and CPU-served requests print the same served line withdriver=sharedderived from the driver's own counters (#1860).- In a single-arm HTTP measurement with a Gemma 4 E2B checkpoint on the CPU, a short prompt took 7.4 to 8.2 s for 32 tokens and a 2,512-token prompt took 477 to 493 s to the first token, on one CPU core (#1860).
Request-level thinking control
lattice serveand the standalone server accept an optionalchat_template_kwargsobject whose booleanenable_thinkingkey controls thinking; false renders the closed-thinking prefix from the Qwen3.5-2B chat template before tokenization and context checks, and generation runs with thinking off (#1887).- Unknown or duplicate keys, non-object values and non-boolean values return the existing invalid-request error envelope. Qwen keeps its current default when the key is omitted or true, and Qwen reasoning budgets are ignored while thinking is disabled (#1887).
- Gemma refuses true and accepts false as a no-op, and its existing reasoning-budget refusals remain (#1887).
ChatRequestgainschat_template_kwargsandRequestedChatOptionsgainsenable_thinking. Neither type can be built with a struct expression outsidelattice-inference(the first is#[non_exhaustive], the second has crate-private fields), so existing code compiles unchanged. Existing public function signatures are unchanged, and the resolved mode travels to generation in the provider'sGenerateConfig(#1887).
Design records
- ADR-090 D10 now admits a family by an inert provider-registration entry outside the serving surface, with no change to any existing family-selection, admission, preparation or construction algorithm, and measures the acceptance by an extension diff that adds only the family's subtree, assets, tests and registration entry. The amendment makes no code changes (#1865).
- ADR-098 (Proposed) extends the option-letter scorer to decision models that answer several typed questions about one state in a single prefill, adding a read-out primitive, a letter table beyond
AtoZ, an optional fp32 decision head, and a/v1/systemoneendpoint compatible with the public Decision Index wire format. The record is Proposed and makes no code changes (#1888).
Serving internals
- The serving runtime contract, which is
ServingRuntime,RuntimeFactoryand the compile-time check that a runtime is notSend, and the Gemma 4 CPU runtime now compile with theservefeature alone on every platform (#1866). - The two serving binaries now decide a checkpoint's family through one directory-aware selector; registered Qwen and Gemma providers make identity claims, and the selector binds exactly one of them or refuses (#1867).
lattice_serveasks the provider selected for its checkpoint which explicit options the family refuses and how the worker factory is built, instead of deciding both inline, with startup output and its order unchanged (#1869).- Request preparation is one crate-private
RequestPreparationtrait object held byserve::prepare::PreparationHandle, implemented beside each family's serving provider, with the public functions and signatures unchanged (#1871). lattice serveruns CPU completion and streaming through a shared CPU host whose two entries build each request's runtime on a blocking task (#1885).generate_f16,generate_q8andgenerate_q8_neonnow run through the shared decoder driver, with their public signatures and outputs unchanged (#1853).- Qwen Metal text requests on both servers print the per-request route line
[route] served family=qwen35 backend=metal mode=<stream|nonstream> driver=shared opened=<n> consumed=<n>, and Qwen Metal vision requests print the same line withdriver=bypassed, taken from the driver trace of each request (005aedb).
Speculative decoding
- Greedy MTP decoding with the sequential verifier, and self-speculative decoding, on Metal now run through the shared decoder driver, with the opt-in selectors
LATTICE_MTPandLATTICE_SELF_SPEC, the greedy-only restrictions, the draft depths and the public types and signatures unchanged (#1859). - With
LATTICE_MTP_VERBOSE, shared routes printroute=shared-mtporroute=shared-self-spec, and the batch-GEMM verifier is recorded as excluded withroute=legacy-batch-gemm-excluded(#1859). - One behaviour difference on the error path: when sequential verification itself returns an error, a draft that is a stop token now ends the request instead of being emitted (#1859).
Efficient CPU prefill and decode
- Batched CPU prefill now normalizes only the last prompt row with the final RMSNorm, since the earlier scratch rows were never read after that step (#1881).
- The twelve per-row LoRA loops in batched prefill now go through the shared
apply_lora_rowshelper, which checksis_activeonce per projection and skips the row loop when the projection is inactive (#1881). - In a paired A/B run of
qwen35_generate --emit-phase-eventson Apple silicon, 48 trials at prompt context 1024 and 512 with base and head built--release --locked --features f16, prefill was unchanged at both contexts (at 1024 tokens, 144.93 to 145.03 tok/s), and no speed claim is made (#1881). - Both PaddleOCR-VL greedy generators no longer run one more decode step after emitting the last budgeted token; the emitted tokens are unchanged (#1875).
lattice-tune
generate_loragains--format raw|chat,--no-thinkwith--format chat, and--grammar-file Fto constrain every generation with a GBNF grammar compiled against the loaded tokenizer's vocabulary (#1873).- Batch mode,
--prompts-file P --out O [--resume], loads the model and adapter once, generates every row of a JSON Lines file in order, and appends one flushed JSON line per row, continuing past a failed row and exiting 2 if any row failed (#1873). train_grad_fullgains--load <PATH>to start from a saved PEFT safetensors adapter, and with--steps 0it is an eval-only path that scores an existing adapter against--data-dir(#1876).- Held-out scoring prints teacher-forced agreement beside the NLL: at each scored completion position the argmax of the logits is compared with the gold token, and a sequence agrees only when every scored position agrees (#1876).
Performance
- A new per-token allocation instrument reports allocation calls, reallocation calls and requested bytes over the warm decode region of the Qwen CPU route through the shared decoder driver. On a Qwen3.5-0.8B checkpoint, the greedy profile over 13 region tokens made 13 allocation calls and 17 reallocation calls for 984,824 bytes, from three agreeing repeats per arm (#1854).
Testing and CI
- The Gemma 4 admission test gains a zero
hidden_size_per_layer_inputarm that must be refused through the loader before any weight access (#1858). - Characterization tests pin both servers' startup errors, request refusals and CPU request isolation before family selection and model loading move behind a common provider path (#1864).
- The startup goldens grow from 32 to 61 cases, pinning competing startup failures and the unknown-format startup path (#1868).
- The CPU route of
lattice serveis characterized from the current binary, for both families, before both CPU arms move behind a shared serving host (#1870). - The prefix-cache disposition tests now use a patterned model, so a corrupted saved recurrent snapshot or KV row can change a logit (#1874).
- A stale Metal decode-loop sampling count is corrected, and the test now runs in CI by exact name (#1861).
- The
cargo-denyjob downloads a pinned release binary with a checked SHA-256 instead of a Docker action, so the required check no longer fails on a registry rate limit (#1872). - The benchmark supervisor installs the Spotlight exclusion marker on the default
target/and.cache/trees before setup or measurement, and refuses the run when the marker cannot be placed (#1877).
Changes to hidden items
These items are marked #[doc(hidden)] and are not part of the documented API. They are listed for anyone who uses them anyway.
lattice_tune::train_support::full_driver::FullDriverConfighas a new public field,load_path: Option<String>, behind thetrain-backwardfeature. A struct literal of this type needs the new field (#1876).lattice_inference::serve::prepare::PreparationHandleno longer implementsDebug(#1871).
Upgrading
No source changes are needed. Two behaviour changes reach existing requests:
- On the Metal serving worker, strict-grammar requests that already disable thinking now also render the closed-thinking prefix in their prompt (#1887).
- In Metal speculative decoding, when sequential verification returns an error, a draft that is a stop token now ends the request instead of being emitted (#1859).