v0.5.2

Fixed

• Top-k sampling returned wrong tokens on the Triton engine. When
  sampling with top_k over a large vocabulary, the operation that
  selects the k highest-scoring tokens could return values that were not
  the true top k, together with token identifiers that did not match the
  values they were returned with. Generated text was still fluent, which
  is what made this hard to notice: the sampler was choosing from a
  corrupted candidate set rather than failing outright. Anyone who ran
  Triton-engine generation with top-k sampling in 0.5.1 should regard
  that output as unreliable and re-run it.

  Two things had to be true at once for the fault to appear, which is
  why it escaped earlier testing: the vocabulary had to be large enough
  to be processed in several passes, and the number of candidates those
  passes produced had to not be an exact power of two. Combinations that
  happened to land on a power of two were correct, so a spot check could
  easily pick a clean case and conclude the operation was sound.

  Greedy decoding was never affected — it does not use this
  operation at all. Mixture-of-experts routing was never affected for
  any published model: routing takes a single-pass path that skips the
  faulty step entirely, for any model with 256 experts or fewer, and the
  largest published model has 128. The PyTorch engine was never affected
  in any mode; it uses PyTorch's own implementation.

  The operation is now checked on every release against an independent
  reference, across vocabulary sizes, candidate counts and seeds, with
  the returned values, the returned identifiers, and the agreement
  between the two verified separately.

• Minutes-long stall on the first token when combining a large
  vocabulary with a large top_k on the Triton engine. The final
  selection step is generated as one fully unrolled routine whose
  compilation cost grows steeply with the number of candidates it has
  to sort — on the tested hardware, 15 seconds, 48 seconds, and then
  over two and a half minutes as that number doubled. top_k of 64 on
  a 152k-token vocabulary landed in the worst part of that curve and
  looked like a freeze. The candidate set is now kept small enough for
  the step to stay in the fast region, by scanning the vocabulary in
  fewer, wider passes. The selected tokens are unchanged; only tokens
  with exactly equal scores may now be broken in a different order.
  Beyond the range where this remains practical, the engine now reports
  a clear error instead of appearing to hang.

Added

• Unreachable-code elimination at load time: operations no output
  depends on are skipped instead of executed. This complements the
  build-side pruning shipped earlier — it also covers models built
  before that landed, and models installed from the registry that
  cannot be rebuilt locally. Operations that write in place, and random
  draws, are always kept: their effect is not visible in the graph's
  data flow.
• Exact identity elimination: operations that return their input
  unchanged — a slice covering a whole axis, a reshape to the shape a
  tensor already has — are removed from the executed sequence, and
  every consumer reads the original tensor. Validated byte-identical
  across the whole model zoo. Arithmetic that merely looks like an
  identity (multiplying by one, adding zero) is deliberately left in
  place: it is not exact in floating point, and belongs to a separate
  pass with a different acceptance rule.
• Horizontal matmul fusion now rewrites the graph rather than only
  planning: several projections reading one activation with different
  constant weights become one wider matrix multiply plus column reads,
  built from ordinary operations both engines already run.
• Two opt-in graph optimisation passes (both off by default; with
  their flag unset the engines execute exactly what they executed
  before). Common-subexpression elimination recognises an operation
  whose result was already computed with the same inputs and reads the
  first result instead of recomputing it. Horizontal matmul fusion
  recognises several projections that read the same activation with
  different constant weights — the classic query/key/value shape — so
  the engines can run them as one wider matrix multiply instead of
  several narrow ones. The first is validated byte-identical across the
  whole model zoo; the second ships its planner and its measurement
  contract while its own byte gate runs.

Fixed

• Warm serving of CPU-streamed models survives multiple requests:
  large models served with per-request weight reloading (CPU-streamed
  placement on a single GPU) crashed on the second request with a
  device mismatch at the first layer-norm — the weight-streaming hooks
  installed on the first load were not re-armed after the
  between-request unload. The unload path now uninstalls the streaming
  state so every reload re-installs it (both engines).
• Text-only prompts can drive speech generation on omni models:
  a text-only request with --mode audio on a build declaring the
  generative-speech contract now runs end-to-end (both engines);
  previously it was refused for lacking an image/audio input even
  though the requested OUTPUT was speech.

Removed

• Dead vendored kernel-utility package (kernels/utils) that could
  not be imported (missing internal modules) and had no consumers.

Added

• Per-instance serving daemon channel: NBX_SOCKET_PATH gives a
  daemon (and every client that targets it) an isolated socket, pid
  and log path, so independent daemons can coexist on one machine.
  Default behavior without the variable is unchanged.

• Frozen-plan replay on autoregressive decode (Triton mode): the
  per-token decode graph of KV-cached language models can now be
  recorded once and replayed as direct kernel launches, cutting the
  per-step launch overhead. The KV append writes at a device-resident
  position counter, MoE token routing tables are built on the GPU
  (removing one host sync per expert layer per step), and every
  replayed bucket is verified byte-equal against the normal path
  before being adopted — outputs are byte-identical by construction.
  Opt-in via NBX_TRITON_REPLAY=1 + NBX_REPLAY_KV_DECODE=1; the
  Qwen3-Coder-30B int4 reference row moves from 2.08 to 4.86 tok/s
  (×2.34) on one V100-32G with identical answers.

• CUDA-graph launch of recorded decode plans (Triton mode): a
  recorded decode graph can additionally be instantiated as a single
  CUDA graph per length bucket and launched as one call per token,
  removing the per-kernel submission cost on top of frozen-plan
  replay. Byte-identical outputs (per-bucket verification unchanged);
  any capture error falls back loudly to direct replay. Opt-in via
  NBX_REPLAY_GRAPH=1; measured on the Qwen3-Coder-30B int4 reference
  row: 4.86 → 5.78 tok/s (+18.8%; cumulative ×2.78 over the
  pre-replay baseline) on one V100-32G with identical answers.

• Warm serving keeps KV cache buffers across requests (Triton
  mode): the KV cache is now owned by the executor and reset in
  place between requests instead of being reallocated per request —
  removing per-request allocation churn and keeping recorded replay
  plans valid across a serving session.

• Device-resident scalar arithmetic on Triton mode: binary ops
  (add/mul/div/sub) with a 0-d GPU scalar operand now compute the
  scalar in-register instead of syncing it to the host — outputs stay
  byte-identical (host-binding arithmetic reproduced exactly), and
  per-step host syncs disappear from diffusion timestep chains
  (Ming-class image generation: 1,196 syncs per request removed),
  making those graphs eligible for frozen-plan replay.

• Step cache extended to autoregressive image-generation legs: the
  opt-in diffusion step cache now also covers the image-generation
  denoise loop of omni/multimodal models (same configuration channels,
  same skip-rate reporting, both engines).

• Failure diagnostics for Triton op errors: a failed Triton op now
  reports each tensor argument's device, pointer, and shape in the
  error message, so placement bugs adjudicate themselves from the
  traceback.

• Persistable autotune selections for replay runs: the kernel
  configurations selected during a recorded Triton-mode run are saved
  per hardware architecture and re-applied on later runs, so replayed
  graphs never re-tune — recorded-run start on the Sana row drops
  56 to 46 s with byte-identical output. Stored configurations are
  validated against the kernel's current search space before use;
  anything stale simply re-tunes.

• Quantized-execution kernel family (first increment): pure-Triton
  int4 groupwise dequantize-GEMV kernels for LLM decode, gated
  byte-for-byte against their own dequantize-then-multiply reference
  (same math, same accumulation order — proven across the target
  decode shapes). Weight traffic drops 4x vs fp16 by construction;
  kernels are not yet wired into the engine (the quantized-artifact
  format decision precedes integration).

• Weight-storage encoding compatibility gate: builds that declare
  a packed weight-storage encoding are checked against the engine's
  supported set at load time — an unsupported scheme is refused with
  a clear message (naming the scheme and the fix) before any weight
  I/O, and encoded variants always ship next to their full-precision
  build.

• Encoded-weight execution on Triton mode (dense LLMs): builds
  using the packed weight-storage encoding now run end-to-end — the
  loader assembles per-tensor triplets under the graph keys, the
  transpose marker is zero-cost (the packed layout is already the
  transposed orientation), decode routes through the byte-gated fused
  dequantize-GEMV and prefill through a transient dense dequantize.
  First validated model produces the same greedy answer as its
  full-precision build at a quarter of the weight bytes.

• Encoded-weight execution for MoE models on Triton mode: the
  fused grouped-GEMM over experts learns in-register decompression —
  per-expert packed triplets ride three pointer tables and the
  B-tile is rebuilt each K-iteration with the tier's canonical fp32
  dequantization, byte-identical to a dequantize-then-grouped-GEMM
  reference kernel (proven per expert shape, decode and prefill,
  both routing arms). First validation: the 30B-A3B coding model's
  encoded variant generates coherent text on a SINGLE V100-32G
  (17 GB of weights vs 57 GB full precision).

• Compiled-mode refusal for encoded-weight builds: running an
  encoded variant without --triton refuses at weight-load with a
  message naming the supported path and the full-precision build
  published next to the variant — never a silent fallback.

• Launch-tuple census diagnostic: NBX_REPLAY_TUPLE_CENSUS=1
  (with replay enabled) records two steady decode steps' kernel-launch
  tuples and prints the varying surface per sequence — the
  measurement instrument behind extending replay to token-decode
  graphs. Observation only; never replays.

• Teacher-forced scoring mode: --set global.score_mode=1 turns a
  Triton-mode text request into one prefill with logits at every
  position and returns a per-window NLL record — the perplexity
  instrument for build-variant quality clauses.

• Decode address stabilization (replay groundwork, opt-in): under
  the decode-replay development flag, token-decode steps carve every
  transient allocation from a per-sequence bump slab recycled each
  step, and attention reads bucket-padded KV views with a persistent
  additive length mask (padding proven byte-neutral: identical greedy
  output with and without). Measured: the per-step varying launch
  surface collapses from 3,428 tuples to the 144 KV-append write
  offsets plus one sampling seed — the address-stability prerequisite
  for replaying token decode.

Fixed

• Step-cache override precedence: an explicit --set global.step_cache_* value of 0 now correctly overrides a
  process-level environment pin instead of silently falling through.

• Diffusion step cache (opt-in, per-model): models configured with
  a step_cache threshold reuse the previous denoising prediction on
  low-change steps (relative-L1 signal, bounded consecutive skips,
  first/last steps always computed) — Sana-1024 on Triton mode drops
  from 24.6 to 17.5 s/image at 29.5 dB PSNR with 8/20 steps skipped.
  Outputs are approximate by design; the pass is inactive unless
  explicitly enabled and reports its skip rate on every run.

• Frozen-plan replay for Triton mode (opt-in): NBX_TRITON_REPLAY=1
  records a graph's kernel-launch sequence once and replays it as
  direct C-launcher calls with byte-identical outputs — diffusion
  steps on Triton mode run up to 2.2x faster (Sana-1024: 52.6 to
  24.4 s/image, outputs byte-identical). Ineligible graph classes
  (KV-cache decode, streaming flows) automatically keep the normal
  path.

• Heterogeneous multi-GPU placement hardening: placement estimates
  no longer bind any symbolic dimension below its traced extent, block
  packing accounts per-device conversion costs and reserves per-GPU
  activation headroom — large omni models (Qwen3-Omni class) now run
  multi-GPU on mixed 16G/32G rigs where the vendor stack cannot load.

• Vertical fusion pass (opt-in): NBX_OPTIM_FUSION_VERTICAL=1
  fuses matmul + activation (SiLU/GELU) pairs into single Triton
  kernel launches with byte-identical outputs, reducing per-step
  launch count on Triton mode (measured +1.2% on diffusion steps
  where the pattern dominates).

• Warm-serving text answers for multimodal builds: text-mode
  requests against multimodal-family models (Qwen3-VL class) now
  return the answer text in the serving response instead of a raw
  passthrough; the request mode is honored and, absent a mode, a
  produced text surface is detected automatically.

• Per-token streaming for vision-language models: the serving
  daemon's streamed generate now emits its per-token events for VLM
  builds on both execution engines — time-to-first-token is measurable
  over the RPC for image+text requests, same event semantics as plain
  LLM decoding.

• Image input over the serving daemon: generate accepts
  image_path; the preprocessing is the exact brick the CLI cold path
  uses (one source of truth), so warm and cold requests see identical
  model inputs.

Fixed

• Speech-mode requests can no longer be silently degraded: a
  mode=audio request on a speech-capable build is guaranteed to run
  the flow that carries the speech leg — previously a text-only prompt
  could be routed to the plain text path and return tokens with no
  audio and no error. Missing prerequisites now fail loudly.

• Warm daemon output resolution for multimodal builds: the request
  mode now reaches the output writer, so audio/image answers save with
  the right extension instead of erroring; the speaker selection is
  forwarded to the speech leg (it was accepted by the CLI and dropped
  by the daemon).

• Constant folding at load (opt-in): with NBX_OPTIM_CONST_FOLD=1,
  subgraphs computable from model parameters alone are executed once at
  weight-bind time — through the same kernels the hot path would have
  used, so outputs stay byte-identical — and their results become
  load-time constants, leaving the per-forward hot loop. Applies to
  both execution engines; conservatively skips symbolic shapes,
  in-place-aliased tensors, fused-op inputs, and streaming (zero3/lazy)
  placements, which refuse with a clear message. Off by default while
  the zoo-wide byte gate and the fold cache land.

• Token streaming over the serving RPC: generate accepts
  stream: true and the daemon pushes one event per decoded token on
  the same connection before the final response, on every execution
  mode. DaemonClient.generate_stream(...) yields the events —
  time-to-first-token is now measurable (and measured, in the
  benchmark harness) instead of estimated. Non-streaming requests are
  byte-identical to before.

• Optimization analysis engine (report-only): a new analysis module
  maps optimization opportunities across every installed package graph —
  constant-foldable subgraphs, exact algebraic identities, duplicated
  computations, dead code, layout candidates and kernel-fusion sites —
  each labeled with the gate class a future transformation must pass
  (byte-preserving vs drift-gated). Nothing is transformed: the sweep
  (python -m neurobrix.core.optim.sweep) only writes a dated findings
  report per model plus a synthesis table. Findings are honest to
  symbolic dimensions: bounds frozen at trace values are surfaced as
  package-coverage suspects instead of being reported as removable.