basal 2.5.0

basal 2.5 (2 November 2026). Default answers to decision requests are byte-identical to 2.4.1 (checked on an H100:
basal-omni-1.0-mini, basal-omni-1.0-max and basal-omni-pico with --mode fast, mini and pico with --mode eager); what
is new is opt-in unless a line says otherwise. Changes on the default path: a question named <task>::<part> next to
task questions is refused (#59); error bodies gain code / detail (#62); GET /v1/models lists the certified
thresholds and the context (#64); every response carries x-request-id (#68).

Scale out and operate

• basal-router: several replicas behind one port. Requests over the same state go to the same replica (state-hash
  routing with bounded load), so its state cache keeps working. --gpus 0,1 starts one replica per GPU and stops them
  with the router, also when the router is killed. On two H100s the router served 1.58x the requests of one GPU
  (twice the clients); scaling is not linear yet.
• Cascade (--escalate, --escalate-below): low-confidence questions are asked again of a bigger model, and every
  answer says which model answered (answered_by). --route picks a replica by the script of the state.
• Multi-level cascade in basal-router: repeat --escalate URL ... --escalate-below P for ordered levels (e.g.
  basal-omni-pico -> mini -> ultra), each with its own threshold; with two or more levels every answer names its level
  and model, /metrics counts answers per level, and a level that is down leaves the previous answer. One level answers
  as before. Accuracy and the share each level answers: docs/RESULTS.md "Cascade: pico, mini, ultra".
  Measured on a held-out half: mini -> ultra at 0.95 serves about 1.5x the requests of ultra alone on one H100 at the
  same accuracy (within 0.5 pt); at 16 clients its p95 latency is higher than ultra's (docs/RESULTS.md).
• basal-router --blend (opt-in): the weighted mean of several models' probabilities; 0.6 ultra + 0.2 mini + 0.2
  pico scored +0.45 pt above ultra alone on a held-out half (95% interval +0.09 to +0.81). A quality option, not a
  speed option: every question goes to every model. Blended answers are not calibrated; refit thresholds with
  basal-calibrate before automating them.
• GET /metrics (Prometheus), HTTP 503 while the model warms up, and a failed start-up exits instead of hanging.
• LLM fallback in basal-router (opt-in: --fallback URL --fallback-model NAME --fallback-below T): after the last
  level, a choice question whose top probability is still under T goes to a chat model you serve behind an
  OpenAI-compatible endpoint, and its answer ("answered_by": "fallback") replaces the basal one. With
  basal-omni-1.0-ultra and, as an example, Gemma 4 31B-it with thinking at T = 0.8, on a held-out half: +1.40 pt over ultra alone (95% interval
  +0.90 to +1.89; 8.4% of the questions sent), when no call is cut by the timeout. One call took 54 s at the median and
  308 s at p95 (one H100, one request at a time); with the default --fallback-timeout 60, 46 of 100 calls kept the
  basal answer, about three quarters of the gain on that sample (--fallback-timeout 600 for offline or review use).
  Fallback answers are not calibrated, and they repeat only with the endpoint settings in docs/SERVING.md
  (batch-invariant vLLM, no prefix cache, no chunked prefill).

Faster start-up

• basal-prepare and --prepared (auto): the kernels tuned for this GPU and version are saved once and loaded at
  start-up. basal-omni-pico starts in 72 s instead of 287 s and gives the same answers in every start; basal-omni-1.0-mini
  starts in 186 s from empty caches instead of 315 s (H100).

Answers you can rely on (opt-in)

• Answer sets and abstain: "coverage": 0.9 returns the set of options that holds the right one with that
  probability; with max_error, plain answers get "abstain": true below the certified threshold.
• Second read (--reread T): answers below T are read again (the state twice) and the two reads combined;
  --reread-image reads an image again at more soft tokens.
• Option order: basal-bench --order-check measures how often the top answer changes when the options are reversed
  or rotated; --order-average K serves the mean over K option orders, all reading the state once. It helps
  basal-omni-1.0-mini only (+0.6 pt top-1 on our development set with K = 2, at about twice the latency per question);
  on basal-omni-1.0-max and basal-omni-pico it does not help, so it stays off by default.
• Two-way models (trained to read the state in both directions) in the fast engine, and isolated options
  (basal.json "option_isolation") for models trained to read the options as a set.

More hardware and models

• basal-omni-1.0-ultra in block FP8 (--mode fp8, one 80 GB GPU: 1.35x the decisions/s of bf16, 53 instead of
  74 GB at 16 clients) or NVFP4 (--mode nvfp4, Blackwell: 2.3x the scoring speed, 2.4% of top answers change).
  Ultra only: other basal-omni models are refused in both modes. In block FP8 the state cache is off in 2.5.0:
  --state-cache-gb is not used (a warning at start-up; requests are served as without the cache) until 2.5.1.
• FP8 checkpoints on one 80 GB GPU: the -FP8 repositories load without a bf16 copy on the GPU, so
  basal-omni-1.0-ultra-FP8 serves on one 80 GB H100 in --mode fast and --mode fp8 (before: out of CUDA memory at
  load); checkpoints that fitted before load the same tensors bit for bit.
• basal-omni-1.0-ultra in int8 (--mode int8, Ampere and newer, for GPUs without FP8): 99.4% of top answers as
  bf16, with its own calibration. Ultra only.
• --state-cache-bits 8: cached states as 8-bit integers, about 1.7x as many states per GB; 99.45% of top answers
  as the 16-bit cache on basal-omni-1.0-max.
• basal-omni-pico in the fast engine with the state cache and several questions over one state.
• Apple Silicon: --mlx-forward basal runs the last layers at the answer rows only: basal-omni-1.0-mini-MLX
  answers one prompt in 131 ms instead of 225 ms (M4 Pro). mlx-lm's forward stays the default.
• basal-attach: serve a chat model that was never trained on basal (Gemma 4, Qwen, ...) as a decision model; it
  starts uncalibrated and basal-calibrate fits it on your labelled requests.
• basal-agree: any backend against the native engine: top-answer agreement with a 95% interval, |Δp|, paired
  accuracy, and whether each question gets the same answer alone as in a request.
• basal-omni-pico on --mode mlx keeps float32 matmuls on an Apple M5 (TF32 off unless MLX_ENABLE_TF32 is set; not
  yet verified on M5 hardware; on an M4 Pro the answers are byte-identical to before).

Clients and integrations

• TypeScript client (clients/typescript, no runtime dependencies): typed questions and answers, answer sets,
  abstain, ask_if / depends_on, images and errors; build it from the repository (not on npm yet).
• Presets (triage, moderation, compliance) in basal-run --preset and Basal.preset; thin LangChain, LlamaIndex and
  CrewAI tools over the HTTP API (basal.adapters; SERVING.md "Agent frameworks").
• The Python client talks to other /v1/systemone servers (bearer key, list criteria); the server takes ask_if /
  depends_on, an images alias and --error-format openai.
• The calibration report shows the temperatures as served.

Start-up and operations

• A live web page for basal-serve (--ui on, off by default): status, traffic since start, the state cache, GPU
  and host telemetry, the configuration and the last 200 requests' metadata (never their text; --ui full also shows
  each request and its answer). Open one page at a time: on one H100 (basal-omni-1.0-mini) throughput within 1% of
  --ui off, p50 at 8 clients +0.4-2.2% with 1-2 pages and +5.2% with --ui full; at 1 client answers byte-identical.
• One page for basal-router and all its replicas (basal-router --ui on, off by default): routing, readiness,
  requests in flight, failures and the router's last 200 requests, next to one card per replica with its own status,
  traffic, state cache and GPU telemetry (replicas started with --ui on). On 2 H100 (basal-omni-1.0-mini, one page
  open): answers at 1 client byte-identical to --ui off, p50 at 8 clients +1.3%, within --ui off's own spread.
• Prepared start-up shipped with the model: the model repositories carry prepared/<key>/ for the H100
  (basal-2.0-mini, basal-omni-pico, basal-omni-1.0-mini, basal-omni-1.0-max); --prepared auto fetches only the
  directory for your GPU and versions, and the answers are byte-identical to a cold start. First start on an H100 with
  empty caches: basal-omni-1.0-mini 290 -> 172 s, basal-omni-1.0-max 259 -> 164 s, basal-omni-pico 105 -> 81 s.
• basal-omni-pico starts the same every time on the H100: fla's kernel configs are pinned, so empty, warm and long-used
  caches give bit-identical answers. 2.4.1's pico fast answers could differ between starts (kernel autotuning); 2.5's do not.
• torch.compile's cache survives a reboot: ~/.cache/basal/torchinductor instead of /tmp (TORCHINDUCTOR_CACHE_DIR
  still wins); RTX 4060 Ti, first start after a reboot: basal-2.0-mini 417 -> 238 s (#47).
• basal-omni-1.0-ultra on a 48 GB card: --quant-load cpu quantises one layer at a time on the GPU, so ultra in
  --mode fp8 or int8 fits a 48 GB card, and --graph-memory-gb N sizes the CUDA graphs for N GiB, so every GPU
  given the same N answers the same: on an H100 with 32 GB taken, 1,700 of 1,700 answers bit-identical to the unloaded
  card (QUANTIZATION.md). Also fixed: with little free GPU memory, a prompt just past the largest compact graph no longer fails (HTTP 500).
• /metrics: the engine version, state-cache evictions and prefix hits, a read that never waits for the engine (#48).
• AMD Strix Halo (--mode strix, community-maintained, not yet verified on our hardware): start-up compiles every
  Triton kernel copy, so no request waits for one (#51); it needs a C compiler at start-up (build-essential). Short
  requests no longer wait for a clocked-down GPU: one short question 390 -> 161 ms on a Radeon 8060S, answers unchanged.

Speed and models

• Less host work in the CUDA cascade (basal-2.0, many questions over one state): pass 2 for 40 questions 62.4 -> 48.6
  ms on an A100, every probability bit-identical (#44; H100: a 40-question request 54.0 -> 51.4 ms, bit-identical).
• basal-omni-pico: --prefix-cache on and --batches compact (with --readout-layer on) in the fast engine, with
  answers as close to fp32 as --mode eager's (1,700 held-out decisions, H100); compact rows serve 1.11-1.16x the
  short single questions per second at 16 clients (no speed-up on long states).
• basal-omni-pico on a Mac: --mode mlx serves it from its own Hugging Face repository (bf16 weights, computed in
  float32); on 1,700 held-out prompts its answers match CUDA eager float32 (top-1 100%). --mode mps stays refused for it.
• basal-omni-1.1-max images and audio through --mode vllm --media on: the same top answer as the eager media path on
  97.4% of images and 95.8% of clips (every one where eager's margin is at least 0.2), 2.7x / 2.9x the requests per
  second at 8 in flight; "detail": "high" stays on --mode eager.
• --mode vllm reads in vLLM's own kernels, so its answers are close to --mode eager's, not equal: top answer
  the same on 98.3% of 361 readouts (basal-omni-1.0-mini), 98.3% (basal-omni-pico), 98.6% (basal-omni-1.0-max) and
  97.6% (basal-omni-1.1-max), H100, 27 Oct. vllm answers may differ from eager on near-ties; vLLM's default preset varies
  from run to run; use --vllm-preset deterministic for reproducible answers. Deterministic preset: bit-identical to
  2.4.1 on 4 models; not available for basal-omni-pico (vLLM has no batch-invariant GDN kernel; basal-serve refuses it
  before vLLM starts).
• Experimental --deterministic --lanes N (still behind BASAL_EXPERIMENTAL_LANES): now exact, not yet faster.

Tools and integrations

• rank and basal-navigate (experimental): rank thousands of candidates in one request (yes / no screening, then
  one choice over the top 20), and retrieval by navigating a document tree level by level, no index. With
  basal-omni-1.0-max on one H100 against a dense 8B embedder: MultiDoc2Dial R@1 +9.6 (44.3 vs 34.7), ConditionalQA
  Hit@1 +19.2 (63.5 vs 44.3). It is slow: one query at a time on that H100 a search takes p50 3.0 s / p95 3.5 s per
  MultiDoc2Dial turn and p50 9.2 s / p95 18.4 s per ConditionalQA question; at 8 concurrent clients the median is 24 s
  and 64 s. Details: docs/NAVIGATE.md.
• basal-navigate search --split-at N (experimental, off by default): a level with more than N candidates goes
  out as several requests of at most N, so trees beyond --rank-max-candidates (10,000) are searched without refused
  requests. Not faster: Natural Questions (2.68M paragraphs, one H100) 51.6 s per question vs 51.4 s with one request
  per level; scores differ by at most 0.045, the first passage and the top 10 the same on every measured question.
• Tokenize once: the server tokenizes the part a request's prompts share (template and state) once and each
  question's own tail after it, for the tokenizers of the basal models; the engine reads the same token ids as
  before. A rank request of 300 candidates over a 12.5k-token state: 0.14 s of CPU instead of 5.8 s (one thread).
• Guard and route presets: guard_input, guard_output and route_intent (Python and TypeScript), strict or
  permissive. Their default thresholds are hand-picked, not certified; with max_error they use the served model's
  own calibration per question type. SERVING.md "Presets".
• basal as the local backend of your decision code (docs/BACKEND.md, a one-function example) and a basal skill for
  coding agents (skills/basal).
• basal-finetune: LoRA on your own labelled decisions (docs/FINETUNE.md; tested on Apple Silicon with
  basal-omni-pico, other GPUs not yet). basal-selfcheck: compares your setup with shipped reference answers;
  docs/OLD_GPUS.md runs basal-omni-pico on older GPUs (GTX 10xx, RX 480 / 580, Intel Iris Xe) through llama.cpp's Vulkan
  backend.
• basal-serve answers the OpenAI Decisions API (/v1/decisions, client.decisions.create), translating to the same
  readout as /v1/systemone.
• LM Studio (opt-in): basal-export-gguf --systemone turns a basal -GGUF file (prompt format 2) into an LM Studio
  decision model: LM Studio 0.4.26+ (beta channel) with llama.cpp engine 2.65.1+; choice / score / yes-no, up to 26 options;
  on LM Studio's /v1/systemone and /v1/decisions, calibrated temperatures, no basal-serve needed. LM Studio stops a
  request after 30 s: 5 of 306 long requests of the basal-2.0-4.5B check hit it, all while macOS background media
  analysis was busy (none on an otherwise idle M4 Pro; INSTALL.md: the workaround); the 607 answered questions agree
  with basal-serve on every top answer (probabilities within 0.0041; basal-omni-pico 620 / 620, within 0.0013).
• For contributors: make help and read-only pre-commit checks (#46).
• Continuous integration: every pull request runs ruff and the CPU tests on Linux and macOS (#53).
• State cache on Apple Silicon (--mode mlx): --state-cache-gb and --prefix-cache on, both --mlx-forward
  paths, with answers bit-identical to the cache off; a cached 2.1k-token policy 13.7 -> 0.9 s per request on a
  MacBook Air M4 (#50).
• Accuracy by document length: docs/RESULTS.md and each model card show accuracy per document-length bucket, with
  the number of items in each.
• The state cache keeps branches: cached states that share a history are all kept, and a request continues from
  the longest one it can extend (#56). On branching agent histories (H100) it reused 1.6-1.7x the prefix tokens of
  2.5's cache, with answers as close to the uncached ones.

Fixes and contributions (community batch, 28-29 Oct)

• Tasks: in a request with task questions, a question named <task>::<part> is refused (HTTP 422) instead of being
  read as the task's own expanded question (#59).
• Errors: an invalid max_error or set bound is refused with HTTP 422 before the model runs, and every error body
  carries a code (INVALID_REQUEST, OUT_OF_MEMORY, INTERNAL_ERROR; a 422 also detail); error keeps its text
  (#62).
• GET /v1/models lists the certified thresholds per max_error target and question type (when the calibration
  matches the model) and the context: limit, overflow policy and truncation head share (#64).
• MCP: a server with several models offers list_models and a model argument on decide / decide_many;
  basal-mcp --model NAME sends that model with every request; one-model servers list the same tools as before (#66).
• basal-serve --api-keys PATH (opt-in): SHA-256 digests of the keys in a JSON file, every route but GET /health
  needs a key, admins see key use at GET /v1/keys; --cors-origins for browsers; every response carries x-request-id
  (#68); basal-router passes a client's key on to the replicas it reaches (never to --fallback).
• basal-interpolate: linear weight interpolation of two checkpoints of one base model, streamed tensor by tensor;
  quantized or mismatched pairs are refused (#60).
• LoRA adapters at load (--adapter DIR on basal-serve, basal-calibrate and basal-bench; basal-fuse-adapter
  writes the merged checkpoint): mlx-lm or PEFT LoRA (rsLoRA too) merged with PEFT's arithmetic; answers equal the
  fused checkpoint's and PEFT's merge's bit for bit (E4B and 31B on an H100 in fast / eager / fp8 / vllm, E4B in mlx,
  a basal-finetune pico adapter). Refused with a message: GGUF, FP8 checkpoints, fast-exit, nvfp4, int8, vllm for
  Qwen3.5, DoRA (#52).
• basal-calibrate records a Hugging Face cache snapshot as its repository id and commit, so the file is applied when
  that snapshot is served (#71). basal-viz explains the state with the facts basal-serve adds for "facts": "auto"
  (#73) and reads basal-omni's Gemma 4 text decisions (#75), and on a Mac the MLX engine's layers (--mode mlx /
  mlx-q8, Llama-architecture models) (#78). The float32 MLX tests pass on M5 GPUs (#76).
• Python client: a method for every question and task type, decide with question builders, AsyncBasal for
  asyncio, and errors by status (InvalidRequest 422, OutOfMemory 503 with retries=, ServerError), all still a
  RuntimeError (#58).
• A request with an empty "questions" object is refused with HTTP 422 (INVALID_REQUEST); 2.4.1 answered it with
  HTTP 500.
• Replicas that basal-router starts with --gpus listen on 127.0.0.1 only (before: 0.0.0.0, reachable past the
  router); a --host after -- still applies.
• Edited chat states with the state cache (--mode fast --state-cache-gb) could return HTTP 500 with NaN answers (GPU
  memory never zeroed); basal-omni-1.0-ultra, one H100: 27 of 1,248 edited-chat requests before the fix, 0 after.
  2.4.1 had the same bug.

Faster, with a measured trade-off (opt-in, off by default; each changes some answers, see WHATS_NEW for the numbers)

• --batches compact for basal-omni on Blackwell workstation GPUs (flex kernels, RTX PRO 6000): mini 1.17-1.20x,
  max 1.21-1.27x the questions per second at 16 clients; the same top answer on 98.0-99.75% of readouts.
• --global-attn triton (Hopper): basal's own kernel for basal-omni's 512-wide global layers; on 24k-32k-token states
  mini 1.10x, 1.1-max 1.16x, ultra 1.03x; the same top answer on 98.3-99.7%.
• --vllm-overlap on (--mode vllm): up to four engine calls in flight; 1.01-1.07x (1.21x for mini, one question
  per request), the same top answer on 98.2-99.7% (two runs with it off differ about as much). With --media on it
  batches media requests too: basal-omni-1.1-max on an H100, new images 1.18-1.49x, audio about the same (0.89-1.23x);
  the same top answer as off on 99.65-100%.
• --compile-scope layer (basal-2.0, Llama): start-up 47 s instead of 228 s on basal-2.0-mini (H100, cold caches); the
  same top answer as a whole-model cold start on 98.7% (two whole-model cold starts: 98.3-98.5%) (#49).
• --long-lane-gdn on: --long-lane for basal-omni-pico; five questions per state 0.12 pt fewer top answers equal to
  float32 than eager bf16 (99.39% against 99.51%).
• --mode mlx --mlx-compute bf16 for basal-omni-pico on a Mac: 0.136 s per prompt against the GGUF Q8_0 path's
  0.171 s (1.25x, M4 Pro); the same top answer as CUDA float32 on 99.41% (the default float32 compute: 100%).

Known limits

• Prepared start-up from a model repository does not load its kernels in 2.5.0 (found after the release): --prepared auto takes the repository's
  prepared/<key>/ without its inductor.bin, so the server compiles as on a cold start (same answers, no time saved). Workaround: download
  the directory once before the first start, e.g. hf download Remek/basal-omni-1.0-mini --include "prepared/*". A directory written by
  basal-prepare on your machine is not affected. Fixed in 2.5.1.
• Repeated requests over long states: with --mode fast on one H100, basal-omni-1.0-ultra gives bit-identical
  probabilities for the same 24k-32k-token request sent again (one at a time) in 95 of 100 cases, as 2.4.1 did;
  basal-omni-1.0-mini in 100 of 100. Ultra's answers on long states are not guaranteed to be bit-stable.
• With --mode fast and --state-cache-gb, a rare race can fail one request with HTTP 500 (an error about FX symbolic
  tracing): about 1 request in 8,000 in our tests; the server recovered and kept serving (retry the request). Present
  since 2.3.1. --state-cache-lock on (opt-in) removes it at a measured cost: one H100, basal-omni-1.0-max, 8 clients,
  8,740 requests: 0 failed instead of 1, 8.19 instead of 8.88 requests/s (-8%), p50 630 instead of 577 ms.

From the community: Miłosz Kucharski (@Butterski: #44, #47, #49), Paweł Kiszczak (@pawelkiszczak: #46, #52, #62, #64, #66, #68, #71, #73, #75), Marcin
Brzeżański (@marcinbrzezanski: #48, #56), Dariusz Koryto (@dkoryto: #53, #58, #59, #60), Grzegorz Stark (@gstark0: #50),
Michał Kardyś (@kardysm: #51), Grzegorz Brzezinka (@agentGreg: #76, #78). Thank you.

Coming in 2.5.1: certified thresholds for the ready question sets (today they are hand-picked), int8 on Apple M5,
the state cache in block FP8 (--mode fp8), and a fix for the rare HTTP 500 with --mode fast and the state cache
without --state-cache-lock's throughput cost.