Skip to content

v0.11.1

Choose a tag to compare

@ohdearquant ohdearquant released this 11 Oct 16:02
· 863 commits to main since this release
3c42eb6

lattice v0.11.1

Patch release. No breaking changes to the documented API: code that compiles against 0.11.0 compiles unchanged against it. Two items that are hidden from the documentation changed shape, and they are listed at the end of these notes.

The headline is Gemma 4 text serving on the CPU from both HTTP servers, request-level thinking control for chat requests, and a pass that moves model families behind serving providers. The GPU and CPU decode paths also move onto the shared decoder driver, and lattice-tune gains adapter loading and batch generation options.

New: Gemma 4 text serving

lattice serve and lattice_serve can now serve a Gemma 4 E2B text checkpoint on the CPU backend through the shared decoder driver, streaming and non-streaming.

  • lattice serve takes the family from the top-level model_type of config.json; only a positive gemma4 selects Gemma, and a missing, unreadable or malformed config keeps the Qwen path the directory had before (#1856).
  • A Gemma 4 checkpoint in the Q4 format is refused at startup with the stable code gemma_metal_unsupported before any loader runs, whatever features the binary was built with (#1856).
  • lattice_serve selects the family and backend through the same detection and route table, and refuses a Gemma 4 Q4 checkpoint the same way (#1860).
  • Requests Gemma 4 cannot serve, and startup options that configure them, return stable codes documented in docs/serve-http-api.md (#1860).
  • lattice serve prints [route] selected family=<f> backend=<b> format=<fmt> at startup, and prints [route] served family=<f> backend=cpu mode=<stream|nonstream> driver=<shared|bypassed> opened=<n> consumed=<n> for each CPU-served request (#1856).
  • lattice_serve prints the same startup line, and CPU-served requests print the same served line with driver=shared derived from the driver's own counters (#1860).
  • In a single-arm HTTP measurement with a Gemma 4 E2B checkpoint on the CPU, a short prompt took 7.4 to 8.2 s for 32 tokens and a 2,512-token prompt took 477 to 493 s to the first token, on one CPU core (#1860).

Request-level thinking control

  • lattice serve and the standalone server accept an optional chat_template_kwargs object whose boolean enable_thinking key controls thinking; false renders the closed-thinking prefix from the Qwen3.5-2B chat template before tokenization and context checks, and generation runs with thinking off (#1887).
  • Unknown or duplicate keys, non-object values and non-boolean values return the existing invalid-request error envelope. Qwen keeps its current default when the key is omitted or true, and Qwen reasoning budgets are ignored while thinking is disabled (#1887).
  • Gemma refuses true and accepts false as a no-op, and its existing reasoning-budget refusals remain (#1887).
  • ChatRequest gains chat_template_kwargs and RequestedChatOptions gains enable_thinking. Neither type can be built with a struct expression outside lattice-inference (the first is #[non_exhaustive], the second has crate-private fields), so existing code compiles unchanged. Existing public function signatures are unchanged, and the resolved mode travels to generation in the provider's GenerateConfig (#1887).

Design records

  • ADR-090 D10 now admits a family by an inert provider-registration entry outside the serving surface, with no change to any existing family-selection, admission, preparation or construction algorithm, and measures the acceptance by an extension diff that adds only the family's subtree, assets, tests and registration entry. The amendment makes no code changes (#1865).
  • ADR-098 (Proposed) extends the option-letter scorer to decision models that answer several typed questions about one state in a single prefill, adding a read-out primitive, a letter table beyond A to Z, an optional fp32 decision head, and a /v1/systemone endpoint compatible with the public Decision Index wire format. The record is Proposed and makes no code changes (#1888).

Serving internals

  • The serving runtime contract, which is ServingRuntime, RuntimeFactory and the compile-time check that a runtime is not Send, and the Gemma 4 CPU runtime now compile with the serve feature alone on every platform (#1866).
  • The two serving binaries now decide a checkpoint's family through one directory-aware selector; registered Qwen and Gemma providers make identity claims, and the selector binds exactly one of them or refuses (#1867).
  • lattice_serve asks the provider selected for its checkpoint which explicit options the family refuses and how the worker factory is built, instead of deciding both inline, with startup output and its order unchanged (#1869).
  • Request preparation is one crate-private RequestPreparation trait object held by serve::prepare::PreparationHandle, implemented beside each family's serving provider, with the public functions and signatures unchanged (#1871).
  • lattice serve runs CPU completion and streaming through a shared CPU host whose two entries build each request's runtime on a blocking task (#1885).
  • generate_f16, generate_q8 and generate_q8_neon now run through the shared decoder driver, with their public signatures and outputs unchanged (#1853).
  • Qwen Metal text requests on both servers print the per-request route line [route] served family=qwen35 backend=metal mode=<stream|nonstream> driver=shared opened=<n> consumed=<n>, and Qwen Metal vision requests print the same line with driver=bypassed, taken from the driver trace of each request (005aedb).

Speculative decoding

  • Greedy MTP decoding with the sequential verifier, and self-speculative decoding, on Metal now run through the shared decoder driver, with the opt-in selectors LATTICE_MTP and LATTICE_SELF_SPEC, the greedy-only restrictions, the draft depths and the public types and signatures unchanged (#1859).
  • With LATTICE_MTP_VERBOSE, shared routes print route=shared-mtp or route=shared-self-spec, and the batch-GEMM verifier is recorded as excluded with route=legacy-batch-gemm-excluded (#1859).
  • One behaviour difference on the error path: when sequential verification itself returns an error, a draft that is a stop token now ends the request instead of being emitted (#1859).

Efficient CPU prefill and decode

  • Batched CPU prefill now normalizes only the last prompt row with the final RMSNorm, since the earlier scratch rows were never read after that step (#1881).
  • The twelve per-row LoRA loops in batched prefill now go through the shared apply_lora_rows helper, which checks is_active once per projection and skips the row loop when the projection is inactive (#1881).
  • In a paired A/B run of qwen35_generate --emit-phase-events on Apple silicon, 48 trials at prompt context 1024 and 512 with base and head built --release --locked --features f16, prefill was unchanged at both contexts (at 1024 tokens, 144.93 to 145.03 tok/s), and no speed claim is made (#1881).
  • Both PaddleOCR-VL greedy generators no longer run one more decode step after emitting the last budgeted token; the emitted tokens are unchanged (#1875).

lattice-tune

  • generate_lora gains --format raw|chat, --no-think with --format chat, and --grammar-file F to constrain every generation with a GBNF grammar compiled against the loaded tokenizer's vocabulary (#1873).
  • Batch mode, --prompts-file P --out O [--resume], loads the model and adapter once, generates every row of a JSON Lines file in order, and appends one flushed JSON line per row, continuing past a failed row and exiting 2 if any row failed (#1873).
  • train_grad_full gains --load <PATH> to start from a saved PEFT safetensors adapter, and with --steps 0 it is an eval-only path that scores an existing adapter against --data-dir (#1876).
  • Held-out scoring prints teacher-forced agreement beside the NLL: at each scored completion position the argmax of the logits is compared with the gold token, and a sequence agrees only when every scored position agrees (#1876).

Performance

  • A new per-token allocation instrument reports allocation calls, reallocation calls and requested bytes over the warm decode region of the Qwen CPU route through the shared decoder driver. On a Qwen3.5-0.8B checkpoint, the greedy profile over 13 region tokens made 13 allocation calls and 17 reallocation calls for 984,824 bytes, from three agreeing repeats per arm (#1854).

Testing and CI

  • The Gemma 4 admission test gains a zero hidden_size_per_layer_input arm that must be refused through the loader before any weight access (#1858).
  • Characterization tests pin both servers' startup errors, request refusals and CPU request isolation before family selection and model loading move behind a common provider path (#1864).
  • The startup goldens grow from 32 to 61 cases, pinning competing startup failures and the unknown-format startup path (#1868).
  • The CPU route of lattice serve is characterized from the current binary, for both families, before both CPU arms move behind a shared serving host (#1870).
  • The prefix-cache disposition tests now use a patterned model, so a corrupted saved recurrent snapshot or KV row can change a logit (#1874).
  • A stale Metal decode-loop sampling count is corrected, and the test now runs in CI by exact name (#1861).
  • The cargo-deny job downloads a pinned release binary with a checked SHA-256 instead of a Docker action, so the required check no longer fails on a registry rate limit (#1872).
  • The benchmark supervisor installs the Spotlight exclusion marker on the default target/ and .cache/ trees before setup or measurement, and refuses the run when the marker cannot be placed (#1877).

Changes to hidden items

These items are marked #[doc(hidden)] and are not part of the documented API. They are listed for anyone who uses them anyway.

  • lattice_tune::train_support::full_driver::FullDriverConfig has a new public field, load_path: Option<String>, behind the train-backward feature. A struct literal of this type needs the new field (#1876).
  • lattice_inference::serve::prepare::PreparationHandle no longer implements Debug (#1871).

Upgrading

No source changes are needed. Two behaviour changes reach existing requests:

  • On the Metal serving worker, strict-grammar requests that already disable thinking now also render the closed-thinking prefix in their prompt (#1887).
  • In Metal speculative decoding, when sequential verification returns an error, a draft that is a stop token now ends the request instead of being emitted (#1859).