Repository navigation
BaseRT 0.5.3
BaseRT uses the NPU in Ryzen AI machines to read prompts faster. On a
Linux machine with an AMD Ryzen AI chip, a prompt of 1,024 tokens or more is
now processed by the Radeon GPU and the chip's XDNA2 NPU together, by
default. On a Ryzen AI 9 HX PRO 475 (Radeon 890M), Qwen3.5-9B prefills a
2,048-token prompt at 865 tokens a second, against 613 on the GPU alone, and
on a Ryzen AI Max+ 395 at 2,662 against 2,217. On Macs, gemma-4 E2B and E4B
answer sampled requests faster, and a prompt with several images is served
again and prefills in one pass.
New
- The NPU shares prompt processing on Ryzen AI machines. During prefill
the NPU computes part of each layer's feed-forward block, and part of the
linear-attention layers in Qwen3.5 and Qwen3.6, while the GPU computes the
rest. The engine bundle carries what the NPU needs for gemma-4 E2B and E4B,
Qwen3.5-9B, Qwen3.6-35B-A3B and Qwen3.8-27B. Dense models gain the most:
on a Ryzen AI Max+ 395 a 2,048-token prompt prefills 20% faster with
Qwen3.5-9B and 14% faster with gemma-4 E4B (7,580 tokens a second against
6,630), and 2% faster with the mixture-of-experts Qwen3.6-35B-A3B. On the
smaller Radeon 890M the NPU carries a larger share: 41% faster with
Qwen3.5-9B and 28% with gemma-4 E4B. Decoding is unchanged. - What the NPU needs. A kernel with the
amdxdnadriver and the NPU's
firmware (Ubuntu 26.04 has both), AMD's XRT runtime (sudo apt install libxrt2 libxrt-npu2on Ubuntu 26.04), and an unlimited locked-memory
limit for the process (ulimit -l unlimited, orLimitMEMLOCK=infinity
for a systemd service). Where one of these is missing BaseRT says which in
one[npu_ffn]line when a model loads and runs on the GPU alone, as
before; the same bundle works with and without XRT installed. - One switch turns the split off, on AMD and on Macs. The NPU computes
its part in 8-bit integers, so answers can differ slightly from the
GPU's own. Over 200 passages of 2,048 tokens from WikiText-2, Qwen3.5-9B
chose the same next token every time with the split as without it.
--no-npuonbasert bench,chatandcomplete(the same flag as
--no-aneon a Mac),baseRT_set_neural_engine(0)in the C API,
baseRT.set_neural_engine(False)in Python,--basert-neural-engine off
onbasert serve, orBASERT_NPU_FFN=0in the environment give the
GPU's results.
Changed
- gemma-4 E2B and E4B decode sampled requests faster on Macs. A request
that samples (a temperature, penalties, a grammar, or no temperature
named) made BaseRT stop and wait for the GPU 19 to 22 times per token
with these two models. Those waits are gone. On an M5 Pro, gemma-4 E2B
answers a sampled request with a short prompt at 110 to 120 tokens a
second, against 70 to 84 before, and with seeded sampling the text is the
same. Greedy requests were already fast and are unchanged. - Several images in one prompt prefill in one pass. A prompt's images
were prefilled one at a time, each paying a fixed cost. Images that fit in
one prefill chunk now go through together: on an M1 Max, Qwen3.6-35B-A3B
takes four 448-pixel images and produces its first token in 976 ms,
against 1,801 ms. The capability record'sserving.media_runs_per_chunk
tells a scheduler that the engine does this, andBASERTD_MEDIA_ONE_RUN=1
restores one image per pass. basert mergeno longer needs twice the model's size on disk. Merging
an overlay that keeps every tensor's format now patches a copy-on-write
clone of the model instead of rewriting it: on a Mac, merging a 5 GB
overlay into a 360 GB model takes 0.3 seconds and no extra disk. Other
merges still rewrite the model, now with one copy on disk instead of two.
Fixed
- Requests with two images. A request whose prompt held two images
close together failed withengine lane fault (code 6)(HTTP 500 from
basert serve), so a conversation that sent screenshots failed on every
turn once it held two. With a short prompt the failure also took down the
other requests being served at the same moment. Both are fixed, on every
platform. - Image positions in Qwen3.5 and Qwen3.6 prompts with several images.
From the second image on, these models saw every later token at the wrong
position, so their answers differed from the reference implementation's.
Positions now match it. A prompt with one image was always right. - Media placeholders with nothing attached are an error everywhere. A
caller of the C API that fed an image or audio placeholder without
attaching the media got an answer that had not seen it, with no error, in
several cases. Every such request now fails with a media fault.
basert servealways attaches what it renders, so its requests do not
change. - Quantized gate projections in Qwen3.5 models. The converter kept two
small projections of the linear-attention layers in 16-bit floats whatever
the profile asked for, and a model that did store them quantized decoded
wrongly. The converter now follows the profile and the engine reads both
forms.
Known issues
- The NPU split on Qwen3.5 and Qwen3.6. These models are more sensitive
to small numerical differences than gemma-4. On Qwen3.6-35B-A3B the next
token differed on 7 of 200 WikiText-2 passages with the split, against 1
of 200 when only the GPU's chunk size changed. Turn the split off where
the GPU's exact results matter. - Qwen3.8-27B needs more NPU memory than the default. Set
BASERT_NPU_FFN_RESIDENT_GIB=24; with the default 4 GiB only part of the
model uses the NPU. - A failing NPU fails the request it was serving. If the NPU returns an
error while a prompt is being prefilled, that request fails. Under
basert serve, superfluid 0.2.0 replaces its engine worker and serves
the next request; superfluid 0.1.15 fails every later request until the
server is restarted.
The engine ABI is unchanged since 0.4.0, so superfluid 0.1.15 serves this
engine. superfluid 0.2.0 is the paired release.
See installation for every
way to install BaseRT.