Skip to content

v0.7.3 — trellis vLLM fix for gemma-4 KV-shared layers

Latest

Choose a tag to compare

@cnygaard cnygaard released this 26 Jul 00:03
67814e2

Patch release. One fix since v0.7.2:

Trellis checkpoints of gemma-4 E-series models now serve on vLLM. vLLM constructs the full q/k/v fused linear for every layer, but KV-shared layers (num_kv_shared_layers — e.g. 20 of 35 on E2B) carry no k/v weights in any checkpoint; those shards stayed at their sentinel init and the trellis apply aborted at engine init with trellis_packed must be 2-D. Never-loaded shards now emit exact-zero columns — the shell/e8p paths' de-facto semantics (attention discards those columns) — in both the shard-batched (S4b) and per-shard fallback branches.

  • New GPU unit test reproduces the pre-fix crash and pins both branches (loaded shard bit-identical, unloaded columns exact zeros)
  • Validated end-to-end: gemma-4-E2B-it-trellis-3inst-4bpw ran a full AIME-2026 avg@8 eval (240 thinking generations) on vLLM 0.25.1 under FULL cudagraph
  • The HF eager path was never affected; no kernel or checkpoint-format changes