Splash 1.3.2

Splash 1.3.2 adds NVFP4 and FP8 checkpoints, more GGUF layouts, optional BF16 recurrent states and thinking budgets. The SSD prefix cache is now on by default, with faster prefill and lower TTFT.

Models

These changes apply to Qwen3.8-27B and Qwen3.6-35B-A3B checkpoints and compatible fine-tunes:

• Load MLX nvfp4, NVIDIA Model Optimizer and compressed-tensors NVFP4/FP8 checkpoints, and llama.cpp NVFP4 GGUFs. For example: nvidia/Qwen3.8-27B-NVFP4 and mlx-community/Qwen3.6-35B-A3B-nvfp4 (#356).
• Load more GGUF variants, including UD-Q8_K_XL, Q5_0/Q5_1 weights, IQ1_M and IQ2_S token tables, other previously refused embedding types, fused MoE gate/up experts and mixed-type GDN alpha/beta tensors. Quantized checkpoints can now contain BF16 and F16 weights; whole BF16, F16 and F32 targets remain unsupported (#379, #403; fixes #375 and #394).

Memory and caching

• --state-format bf16 halves the GDN recurrent-state storage. A cached state shrinks from about 153 to 81 MiB on the 27B and from 64 to 34 MiB on the 35B. Computation stays FP32, with rounding when BF16 states are stored; FP32 remains the default (#385).
• The SSD prefix cache defaults to auto, with a quota up to 32 GiB that follows available space on its local volume. --max-cache-disk 0 turns it off. The cache adjusts as the volume fills and can keep serving after a full-disk write; unused persistent caches are cleaned up after 14 days (#408).
• --persistent-cache keeps the SSD cache across restarts and remains off by default. It now works without an explicit disk quota. --max-cache-disk-writes SIZE reduces optional writes to a daily budget; conversation resume states, their KV pages and the clean-stop flush can exceed it. Writes have no budget unless one is set (#408).
• The RAM prefix cache grows only into spare memory and gives memory back when macOS warns. Idle reclaim avoids repeatedly discarding an extra recovery margin when no request needs it (#383, #418; fixes #325).
• POST /v1/memory/release frees idle model weights on demand while keeping the prefix cache. It works with --idle-release off; the next request restores the weights. A busy engine or a restore in progress returns 409 engine_busy, and 200 confirms the weights are released (#374, #416).
• cache_salt isolates KV and cached states between different salts, including on disk. Chat/text completion metrics and the Done log now report KV reused beyond a request's initial cached prefix as deduplicated_tokens, including reuse after suspension (#382, #415; fixes #368).

Thinking and agent compatibility

• Limit reasoning with thinking_token_budget in Chat Completions and Responses, or set a server default with --thinking-token-budget N. The default is uncapped; -1 disables the cap for a request. Messages honors enabled thinking's required budget_tokens (at least 1024) and accepts budgets at or above max_tokens (#397, #411; fixes #373).
• --thinking-end-text TEXT can turn the model toward its answer when the budget runs out. The text must end with </think> and fit within 64 tokens. With structured output or constrained tools, the cutoff waits for a complete character before closing thinking, avoiding grammar failures on split Unicode characters; reasoning can therefore run a few tokens past the budget (#405, #421).
• Messages applies Claude Code's mid-session tool_addition and tool_removal blocks in conversation order, including additions by reference. Later definitions replace tools of the same name (#391; fixes #389).
• Tool names can contain dots, spaces, punctuation and Unicode, up to 256 characters within the call format's rules. Non-strict tools with unusable JSON Schema descriptions no longer reject the whole request; strict tools still require valid schemas (#400; fixes #392).
• Output limits larger than the remaining context now generate up to the window's end instead of returning 400. Optional null fields, seed: -1, stop strings alongside tools or structured output, and malformed historical tool arguments are handled without rejecting otherwise usable requests. Reasoning history and template options are accepted in more client formats; context-management fields do not make Splash edit or compact the supplied history (#401, #402).

Latency and decoding

Individual-change benchmarks on a 40-core M5 Max measured:

• Sending an unconstrained request's first token as soon as prefill selects it reduced short-prompt median TTFT by about 37 ms on the 27B and 13 ms on the 35B (#358).
• Capturing cached states inside prefill commands reduced prompt-processing time for follow-ups adding 100–300 tokens to a cached 8K prefix by 11–14% on the 27B and 20–23% on the 35B (#406).
• Vision encoding of a synthetic 16,384-patch image fell from 1.419 to 1.141 seconds, about 20%, with the 27B vision tower (#384).

MoE decode skips a redundant gather for one-lane staged plans: measured 35B cycle time fell 1.4–2.5% on M5 Max, M5 Pro and M6. GDN decode folds retained rows into the next step instead of running a separate commit, improving most tested cases by up to about 5.5%; its M3 Max one-lane 35B cycle was 0.4–0.8% longer (#365, #366).

Status and reliability

• /status.active_requests lists each live request's phase, priority, prompt progress, generated tokens and age. Started logs show effective reasoning settings; per-request speculative decoding counts appear in Chat/text completion timings and metrics.speculation, with tokens per cycle on the Done line (#255, #357, #395, #399).
• Cancelled, expired or failed prefill commands keep their previous checkpoint until a successor is published, so retries retain their recovery point (#420).
• Non-finite prefill results are kept out of the cache, damaged restored states are invalidated, and NaN/negative-infinity logits cannot become selectable through penalties. Failure messages report the invalid logit counts (#381, #396).
• Engine relaunches preserve the first engine's context when the memory plan can hold it, preventing a changed ANE calibration from leaving the server permanently unavailable (#424).
• Truncated emoji and other lone surrogate characters tokenize as U+FFFD, including in scoring prompts. Responses keeps a completed tool call's status intact when a stop string cuts a later call; Messages correctly rejects unsupported assistant prefills even after dropped system messages (#354, #398, #412, #413).
• Request-start logging no longer blocks token delivery on the runtime reader thread. GPU tape-offset overflow is rejected before submission, and startup reports obsolete prepared-weight files from Splash 1.1 that can be removed (#369, #380, #414).

Thanks to @janhilgard, @linson007 and @daijiong1977 for their contributions.

Upgrade

brew update
brew upgrade incoai/tap/splash

Restart the server after upgrading: the native/server protocol changes from 8 to 10.

The default SSD cache reserves a state-staging buffer, so memory-limited Macs may advertise a smaller automatic context. Use --max-cache-disk 0 to turn it off. Existing FP32 persistent caches remain reusable when model and cache layouts match; BF16 states use a separate namespace.

Full changelog: 1.3.1...1.3.2