basal 2.3.2

basal 2.3.2

basal 2.3.2 gets the engine ready for basal models built on Qwen3.5 (the new 0.8B pico), whose Gated DeltaNet
layers carry a recurrent state. Modes that cannot serve such a model now refuse it, the default picks one that can,
and --mode gguf gives correct answers for it on the CPU. --mode fp8 now stops early on GPUs that cannot run it,
and two fixes to the facts block come from the community.

• Qwen3.5 models are refused where they would be wrong. 2.3.1 served such a model in --mode fast, the CUDA
  default, and its answers were wrong without any error. The fast engine shares a request's state between questions
  as attention keys and values only. Its Gated DeltaNet layers keep a recurrent state instead, which that
  sharing loses. Now fast, fast-nocompile, fast-exit, fp8, nvfp4 and mps refuse such a model before its
  weights are downloaded or loaded (6-7 s on an H100). The message names --mode eager and --mode vllm. Without
  --mode, such a model runs in eager on CUDA and on Apple Silicon. Check on one H100 with the 0.8B pico, served
  without --mode: the server picked eager and answered a five-question request. On 1,700 held-out decisions its top answer
  agreed with the trainer's own readout on 99.65% (bf16, as served) and on 99.94% in fp32 (max |dp| 0.005).
• --mode gguf on hybrid models. On the CPU, llama.cpp returned non-finite probabilities whenever one decode
  held several prompts of a Qwen3.5 model: 300 of 300 prompts at 16 per call, and every request with several
  questions. Hybrid and recurrent architectures (Qwen3.5's qwen35, Mamba, RWKV and the others llama.cpp flags) now
  decode one prompt at a time on every device. With the pico's Q8_0, the top answer agreed with eager on
  99.0% of 300 held-out decisions on 16 CPU threads, on 99.0% of 1,700 on an H100 and on 99.0% of 300 on an Apple M4
  Pro (Metal). No probability was non-finite, and a five-question request was answered on all three (0.25 s on the
  H100, 1.5 s on 16 CPU threads). The cost: a request's state is now computed once per question and option order,
  not once per request.
• --mode fp8 below compute capability 8.9 (Ampere, such as the RTX 3090 or A100) stops before it loads the model,
  with a message that suggests --mode fast (bf16). Before, torchao failed only after the load. Reported in #33.
• Other models are unchanged. Without --mode, a Qwen3 model and basal-omni-1.0-mini still run fast on an
  H100. On 200 held-out decisions their answers match 2.3.1 (the same probabilities on every item, max |dp| 0).

Install:

uv pip install "basal[media] @ https://github.com/rkinas/basal/archive/refs/tags/v2.3.2.tar.gz"

From the community

• Grzegorz Brzezinka (@agentGreg): the facts block reads amounts with a dot thousands separator ("10.000,00 zł",
  #31), and the "roku" after a date is no longer read as a one-year duration (#32).
• @krzysztofkaminski: the first Ampere numbers (RTX 3090, #33). They led to the --mode fp8 check above and to the
  RTX 3090 row in HARDWARE.md and SERVING.md.

Not in 2.3.2

• The fast engine for Gated DeltaNet models, with the recurrent state kept per layer in CUDA graphs, comes in 2.4.
  Until then the pico runs in eager (exact), in vllm (faster on CUDA) or in gguf.
• Compact rows sized to the free GPU memory (#33: basal-2.0-4.5B with --batches compact runs out of memory at
  start-up on 24 GB). This needs a start-up measurement on a 24 GB card. The 8-row tree buffer is only 1.4 GB on the
  4.5B, so the memory goes to the graphs, and a rule we cannot check would be a guess. Until then, use the default
  batches on 24 GB, or GraphBackend.TREE_ROWS = [1, 2, 4].
• Shared prefixes for hybrid models in --mode gguf. On Metal, 2.3.1's shared decode gave the same answers as one
  prompt at a time (max |dp| 0.0002 on a five-question request), but not on the CPU. Reusing the state would need
  llama.cpp's per-sequence state copies, which are not checked yet.