Skip to content

BaseRT engine 0.5.3

Latest

Choose a tag to compare

@prabod prabod released this 19 Nov 19:51
7d5ddaf

BaseRT 0.5.3

BaseRT uses the NPU in Ryzen AI machines to read prompts faster. On a
Linux machine with an AMD Ryzen AI chip, a prompt of 1,024 tokens or more is
now processed by the Radeon GPU and the chip's XDNA2 NPU together, by
default. On a Ryzen AI 9 HX PRO 475 (Radeon 890M), Qwen3.5-9B prefills a
2,048-token prompt at 865 tokens a second, against 613 on the GPU alone, and
on a Ryzen AI Max+ 395 at 2,662 against 2,217. On Macs, gemma-4 E2B and E4B
answer sampled requests faster, and a prompt with several images is served
again and prefills in one pass.

New

  • The NPU shares prompt processing on Ryzen AI machines. During prefill
    the NPU computes part of each layer's feed-forward block, and part of the
    linear-attention layers in Qwen3.5 and Qwen3.6, while the GPU computes the
    rest. The engine bundle carries what the NPU needs for gemma-4 E2B and E4B,
    Qwen3.5-9B, Qwen3.6-35B-A3B and Qwen3.8-27B. Dense models gain the most:
    on a Ryzen AI Max+ 395 a 2,048-token prompt prefills 20% faster with
    Qwen3.5-9B and 14% faster with gemma-4 E4B (7,580 tokens a second against
    6,630), and 2% faster with the mixture-of-experts Qwen3.6-35B-A3B. On the
    smaller Radeon 890M the NPU carries a larger share: 41% faster with
    Qwen3.5-9B and 28% with gemma-4 E4B. Decoding is unchanged.
  • What the NPU needs. A kernel with the amdxdna driver and the NPU's
    firmware (Ubuntu 26.04 has both), AMD's XRT runtime (sudo apt install libxrt2 libxrt-npu2 on Ubuntu 26.04), and an unlimited locked-memory
    limit for the process (ulimit -l unlimited, or LimitMEMLOCK=infinity
    for a systemd service). Where one of these is missing BaseRT says which in
    one [npu_ffn] line when a model loads and runs on the GPU alone, as
    before; the same bundle works with and without XRT installed.
  • One switch turns the split off, on AMD and on Macs. The NPU computes
    its part in 8-bit integers, so answers can differ slightly from the
    GPU's own. Over 200 passages of 2,048 tokens from WikiText-2, Qwen3.5-9B
    chose the same next token every time with the split as without it.
    --no-npu on basert bench, chat and complete (the same flag as
    --no-ane on a Mac), baseRT_set_neural_engine(0) in the C API,
    baseRT.set_neural_engine(False) in Python, --basert-neural-engine off
    on basert serve, or BASERT_NPU_FFN=0 in the environment give the
    GPU's results.

Changed

  • gemma-4 E2B and E4B decode sampled requests faster on Macs. A request
    that samples (a temperature, penalties, a grammar, or no temperature
    named) made BaseRT stop and wait for the GPU 19 to 22 times per token
    with these two models. Those waits are gone. On an M5 Pro, gemma-4 E2B
    answers a sampled request with a short prompt at 110 to 120 tokens a
    second, against 70 to 84 before, and with seeded sampling the text is the
    same. Greedy requests were already fast and are unchanged.
  • Several images in one prompt prefill in one pass. A prompt's images
    were prefilled one at a time, each paying a fixed cost. Images that fit in
    one prefill chunk now go through together: on an M1 Max, Qwen3.6-35B-A3B
    takes four 448-pixel images and produces its first token in 976 ms,
    against 1,801 ms. The capability record's serving.media_runs_per_chunk
    tells a scheduler that the engine does this, and BASERTD_MEDIA_ONE_RUN=1
    restores one image per pass.
  • basert merge no longer needs twice the model's size on disk. Merging
    an overlay that keeps every tensor's format now patches a copy-on-write
    clone of the model instead of rewriting it: on a Mac, merging a 5 GB
    overlay into a 360 GB model takes 0.3 seconds and no extra disk. Other
    merges still rewrite the model, now with one copy on disk instead of two.

Fixed

  • Requests with two images. A request whose prompt held two images
    close together failed with engine lane fault (code 6) (HTTP 500 from
    basert serve), so a conversation that sent screenshots failed on every
    turn once it held two. With a short prompt the failure also took down the
    other requests being served at the same moment. Both are fixed, on every
    platform.
  • Image positions in Qwen3.5 and Qwen3.6 prompts with several images.
    From the second image on, these models saw every later token at the wrong
    position, so their answers differed from the reference implementation's.
    Positions now match it. A prompt with one image was always right.
  • Media placeholders with nothing attached are an error everywhere. A
    caller of the C API that fed an image or audio placeholder without
    attaching the media got an answer that had not seen it, with no error, in
    several cases. Every such request now fails with a media fault.
    basert serve always attaches what it renders, so its requests do not
    change.
  • Quantized gate projections in Qwen3.5 models. The converter kept two
    small projections of the linear-attention layers in 16-bit floats whatever
    the profile asked for, and a model that did store them quantized decoded
    wrongly. The converter now follows the profile and the engine reads both
    forms.

Known issues

  • The NPU split on Qwen3.5 and Qwen3.6. These models are more sensitive
    to small numerical differences than gemma-4. On Qwen3.6-35B-A3B the next
    token differed on 7 of 200 WikiText-2 passages with the split, against 1
    of 200 when only the GPU's chunk size changed. Turn the split off where
    the GPU's exact results matter.
  • Qwen3.8-27B needs more NPU memory than the default. Set
    BASERT_NPU_FFN_RESIDENT_GIB=24; with the default 4 GiB only part of the
    model uses the NPU.
  • A failing NPU fails the request it was serving. If the NPU returns an
    error while a prompt is being prefilled, that request fails. Under
    basert serve, superfluid 0.2.0 replaces its engine worker and serves
    the next request; superfluid 0.1.15 fails every later request until the
    server is restarted.

The engine ABI is unchanged since 0.4.0, so superfluid 0.1.15 serves this
engine. superfluid 0.2.0 is the paired release.

See installation for every
way to install BaseRT.