basal 2.3

basal 2.3

• basal-omni on the fast engine. The CUDA graph engine now runs basal-omni (Gemma 4): state once for every
  question, compiled layers, FlashAttention 3 / FlexAttention. On one H100, 3.0-6.1x the decisions per second of 2.2's
  --mode eager on single questions and 13.7-17x on several questions about a long document, with the same answers
  within bf16 rounding.
• A faster --mode eager for basal-omni (images and audio): the state, and the image and audio encoders, run once
  per request instead of once per question.
• vLLM presets for basal-omni: mini's shared layers skipped at prefill (1.43x / 1.60x, on by default), a deterministic preset with
  bit-identical answers run to run, and images and audio for mini.
• Fixes: basal-omni without --mode no longer stops at start-up on a GPU; the eager backend avoids a PyTorch
  attention bug for every model; and the prompt language is detected once per request (#17, Milosz, @Butterski).

Install:

uv pip install "basal[media] @ https://github.com/rkinas/basal/archive/refs/tags/v2.3.0.tar.gz"

basal-omni on the fast engine

basal-serve --model Remek/basal-omni-1.1-max --port 8000                # without --mode: fast on Hopper with FlashAttention 3, eager elsewhere
basal-serve --model Remek/basal-omni-1.1-max --mode fast --port 8000    # ask for it explicitly

What the engine does for Gemma 4:

• State once, ask many. A request's state is computed once; every question and option order runs over its keys and
  values, also in Gemma 4's sliding-window layers (a question sees exactly the window it would see as a separate
  prompt).
• Answer rows only. basal-omni-mini's last 18 of 42 layers reuse earlier layers' keys and values, so they run only
  at the positions where answers are read (about 40% less compute per decision); on max and ultra the last layer does.
• Kernels. FlashAttention 3 in the sliding-window layers (Hopper), FlexAttention in the 512-wide global layers,
  merged projections, compiled layers in CUDA graphs, batches without padding. --gemma-attn math keeps the reference
  kernel.
• Memory. Long states keep only the last window of keys and values in the sliding layers (a 10,000-token state on
  ultra: 1.6 GB instead of 9 GB), 8 MB cuBLAS workspaces per CUDA graph, no image / audio towers in this text mode.
• Calibration. Fast mode serves each model's shipped CALIBRATION.json.

Decisions per second, 16 requests in flight, against 2.2's --mode eager on the same GPU ("one question": one question
about a ~1.5k-token state; "five questions": questions per second, five questions about a 3k-10k-token state):

model	GPU	one question: 2.2 -> 2.3	gain	five questions: 2.2 -> 2.3	gain
basal-omni-1.0-mini	H100	10.0 -> 61.2	6.1x	1.6 -> 21.4	13.7x
basal-omni-1.0-mini-FP8	H100	10.0 -> 80.6	8.0x	1.6 -> 19.6	12.5x
basal-omni-1.1-max	H100	4.8 -> 16.2	3.4x	0.72 -> 10.6	14.7x
basal-omni-1.0-ultra	H100	2.0 -> 6.1	3.0x	0.28 -> 4.8	17x
basal-omni-1.0-mini	RTX PRO 6000 Blackwell (Max-Q)	5.6 -> 23.2	4.1x	0.85 -> 18.4	21.6x
basal-omni-1.0-max	RTX PRO 6000 Blackwell (Max-Q)	2.3 -> 5.0	2.2x	0.37 -> 5.3	14.2x

(2.2 ran ultra at 4 and 2 requests in flight; more ran out of memory. Blackwell rows: --mode fast --gemma-attn flex,
opt-in until its calibration check passes there.)

Measured against 2.2 on 10,606 development decisions, one option order: the same top answer on 99.85-99.99% of the
decisions where the model is sure (margin >= 0.2) and on 98.9-99.4% overall, accuracy within 0.06 points, and with the
shipped calibration files the share automated at a 1% error target within 0.1 points (the observed error stays below
the target at 1% and 5%). In fp32 every path gives the answers of separate prompts.

A faster --mode eager for basal-omni

The eager backend (images, audio, --deterministic, evidence) runs Gemma 4 through the engine's own forward: the
state and the media once per request, every question and option order over them, and mini's last 18 layers at the
answer rows only. One H100, 16 in flight: mini 10.0 -> 17.2 decisions per second (one question) and 1.6 -> 15.3
questions per second (five questions); max 0.72 -> 4.5. Images and audio: max's latency at the median 204 -> 131 ms
with three questions about one image. --eager-forward hf keeps 2.2's path. Opt-in: --eager-attn auto (FlashAttention 3 in the sliding layers and
the memory-efficient kernel with copied keys and values in the global ones, Hopper; mini 24.4 decisions/s, max 8.7)
and --ple-offload on (mini's 5.6 GB per-layer embedding table in host memory: 5.2 GB less GPU memory, about 4%
slower, the same answers).

vLLM presets

• Shared-layer prefill skip, on by default for basal-omni-1.0-mini (--vllm-fast-prefill off restores 2.2):
  mini's last 18 layers reuse earlier keys and values, so at prefill they run at the answer position only. One H100,
  16 in flight, against 2.2's vLLM path: 60.8 vs 41.5 decisions/s (one question), 39.0 vs 24.2 questions/s (five
  questions about a long document); the same top answer as --mode eager on 99.94% of the decisions where the model
  is sure.
• --vllm-preset deterministic (opt-in): batch-invariant kernels, no prefix cache, no chunked prefill. Two runs of
  basal-omni-1.1-max at 32 requests in flight gave bit-identical probabilities on all 10,606 decisions. It costs
  throughput (1.1-max: 0.64x on single questions, 0.22x on five questions).
• Images and audio through vLLM for basal-omni-1.0-mini (--mode vllm --media on): the same prompts and media
  tokens as --mode eager --media on; 96.5% (images) and 98.3% (audio) the same top answer, all of them where the model
  is sure; 6.0x / 7.9x the requests per second at 8 in flight. basal-omni-1.1-max is refused for now (its images did
  not match the eager path closely enough).
• Not in 2.3: an FP8 key / value cache preset (it lowered automation on mini and was slower on max and ultra) and FP8
  weights made at load time (both failed their checks).

Fixes

• basal-omni without --mode stopped at start-up on a CUDA GPU in 2.2 (the fast engine could not load Gemma 4).
  2.3 runs fast on Hopper GPUs with FlashAttention 3 (the configuration that passed its checks) and
  eager on other GPUs.
• A PyTorch attention bug (pytorch/pytorch#191937, fixed in torch 2.14.1): before 2.14.1, SDPA's memory-efficient
  kernel returns wrong attention for keys and values expanded from a single key / value head at some query lengths,
  which is what transformers passes with an attention mask. basal-omni-1.0-max was off by 6.2 in an answer's
  log-probability on a 1,857-token prompt. The eager backend now runs every model without that kernel on older torch
  (2.2 did so for Gemma 4 only); the fast engine never passes such keys and values. Served basal models were not
  affected in 2.2.
• Faster preparation of large requests (#17, Milosz, @Butterski): the prompt language is detected once per request,
  not once per question (a 40k-character compress request with 326 questions: 509 ms, now 2 ms).

basal-2.0 (Bielik) on 2.3

basal-2.0 models take the same --mode fast path as in 2.2. In --mode eager, transformers' forward now runs without
PyTorch's memory-efficient kernel on torch older than 2.14.1 (see Fixes); basal-2.0's own layers have two or more
key / value heads, which our checks found unaffected by the bug. A side-by-side measurement of 2.3 and 2.2 on
basal-2.0 follows with 2.3.1.

Thanks

Milosz (@Butterski) for #17.