strixite v0.5.0

strixite v0.5.0

Conversations at once - and the memory to hold them.

Conversations and memory

• Multiple conversations at once (total-slots, 2 by default, up to 8): concurrent conversations are served
  together, each at the full context, and their decode steps run as one batched forward through the model. A step
  with no draft joins the batch as a single row instead of running alone. Two 512k conversations is the measured
  sweet spot; shorter contexts fit more (six 256k slots take less GPU memory than v0.4.0's single slot did).
  Requests past the slots wait in arrival order. If the machine cannot hold the slots you asked for, startup says
  so and names the count that fits. How it works
• A slot's memory grows with its conversation: K/V pages are mapped as a conversation fills them instead of
  every slot committing its full capacity up front. Two 512k slots now cost less than one slot did in v0.4.0 -
  GPU buffers 84.7 GiB (was 91.9 for one slot), and depth a conversation never reaches is memory the prompt cache
  can use instead. Startup holds back one slot's K/V at full depth (8.2 GiB); the rest arrives as it is used.
  Where the memory goes, slot recipes

Speed (this build, end to end through the server, one request at a time, two runs per depth; v0.4.0 numbers
as published for it - its 250k-481k points were single runs)

context	prefill t/s	decode t/s
32k	1,900 (v0.4: 1,693)	60.3 (57.7)
120k	1,817 (1,630)	58.7 (58.7)
250k	1,716 (1,570)	63.8 (56.4)
350k	1,685 (1,533)	56.3 (55.7)
481k	1,633 (1,484)	54.3 (52.8)

Prefill is 10-12% faster everywhere (the grouped-expert kernels, the dot-product Q4 path, staged attention splits);
decode is flat to ~4% up (attention on the matrix units even for a single row, a faster 8-bit K read). And the other
headline - sharing conversations in one forward - changes what the machine does with a second client, not the speed
of one: a slot that waits costs nothing, and two conversations that generate together read the weights once between
them instead of twice.

Fixes and stability

• A conversation could be refused memory while free RAM existed: the prompt cache measured the machine twice,
  a few requests apart, and decided on the second reading. One reading now, and growth asks before deciding.
• Multi-slot correctness under load: the structured-output grammar automaton is safe across concurrent slots, a
  failed request cannot take other slots' workers down, and per-slot statistics count every slot.
• Loader and sampler bound checks (a corrupt weights file is refused with the failing tensor named, leading-NaN
  logits fail cleanly instead of poisoning the answer).
• A new log-level setting (error / warn / info / debug) - debug adds the per-step detail that's useful in a
  bug report.
• Paged K/V never releases a page while the server runs: reusing an address the kernels may still hold proved
  unsafe on this hardware, so a slot keeps what its deepest conversation used until restart.

Container

podman pull ghcr.io/shawnshekari/strixite:v0.5.0

Upgrading from v0.4

• Container: pull :v0.5.0 (or :latest) and restart.
• From source: git pull, then build as before. The weights are unchanged. The Quick start and the release
  image now pin TheRock's first GA tarball (10.1.0), the same ROCm the nightlies passed.
• Your own copy of strix-server.conf: new in this release: total-slots = 2, chunk-shared = 2048
  (prefill chunks while another slot decodes) and log-level = info. A copy keeps its own values;
  git diff v0.4.0 v0.5.0 -- deploy/strix-server.conf shows every change.
• Prompt-cache entries from v0.4 stay valid - the saved-state format is unchanged.

This week @Cheblan independently sent two pull requests implementing batched multi-slot decode and a paged KV
cache - the same two features this release ships, built from the same read of the same machine. That converging
is a good sign for all of us.

Contributors: @shawnshekari

v0.4.0

strixite v0.4.0

An 8-bit KV cache by default, the fixes from your reports, and a little more speed.

Memory

• 8-bit KV cache (kv-cache = q8, the new default): at 512k one full conversation state is 8.3 GiB instead of
  13.5, GPU buffers 91.9 GiB instead of 95.3, and saved conversations in the prompt cache are ~40% smaller. The startup
  check now has ~10 GiB of slack on a 128 GB machine (was 1.6), and a 256k context fits next to a desktop with ~17 GiB
  to spare. kv-cache = bf16 brings back the old format. Where the memory goes
• Image decodes are limited to 2 at once (vision-preprocess-limit); one past the limit gets a retryable 503 instead
  of pushing the machine into a low-memory stall.

Fixes

• The prompt cache no longer re-reads a long conversation every turn when RAM is full (#11): loading an entry
  from disk now waits for the evictions it started instead of refusing.
• Repeated images are no longer decoded on every request (#9, item 1): an agent step with 20 known screenshots
  and 1 new one spends 0.07 s preparing images instead of 1.76 s.
• No thinking cap by default: v0.3 capped thinking at 2,048 tokens and the model seemed less smart for it.
  thinking-cap = N still sets one; a request's own budget always wins.
• Output bytes that aren't valid UTF-8 become U+FFFD instead of failing the request.
• After a failed request, later requests no longer re-report its stale kernel error.
• A request's own 0x01 bytes are text, never an image (v0.3.1's fix, carried forward).

Speed (measured end to end through the server, one request at a time, two runs per depth; v0.3 numbers measured on v0.3.0 - v0.3.1 only added a chat-template fix)

context	prefill t/s	decode t/s
32k	1,693 (v0.3: 1,638)	57.7 (59.0)
120k	1,630 (1,583)	58.7 (56.7)
250k	1,570	56.4
350k	1,533	55.7

Prefill is ~3% faster (a faster linear-attention prefill path); decode is unchanged within noise. MTP drafting stops
at a confidence margin of 1.5 (was 2.0), slightly faster on agent work.

Container

podman pull ghcr.io/shawnshekari/strixite:v0.4.0

Upgrading from v0.3

• Container: pull :v0.4.0 (or :latest) and restart. The image carries the new settings; flags you pass are
  kept.
• From source: git pull, then build as before. The weights are unchanged.
• Your own copy of strix-server.conf: v0.3's file set two values this release changes, thinking-cap = 2048
  (now 0, no cap) and mtp-margin = 2 (now 1.5). A copy keeps the old ones until you change them;
  git diff v0.3.1 v0.4.0 -- deploy/strix-server.conf shows every change. The 8-bit KV cache and the faster prefill
  come from built-in defaults, so they apply either way.
• Saved prompt-cache entries from v0.3 hold a 16-bit KV cache: v0.4.0 deletes them at startup (one
  prompt cache: dropping ... line each) instead of misreading them, so the first turn of an old conversation reads
  its prompt again.

Contributors: @shawnshekari, and @maxm86545 (reported in #9)

v0.3.1

Fixed

• A tool result holding binary bytes no longer breaks a conversation with images. If a conversation had an image in it and a tool result printed a binary file (head on an executable, say), every request from then on failed with expand_image_pads: the tokenized prompt carries N image_token_id rows, the request sent M image(s) - the binary's 0x01 bytes were mistaken for image markers, and the output stayed in the history. Those bytes are now ordinary text (#10). A session already stuck on v0.3.0 works again after updating, or start a new one.

Container

podman pull ghcr.io/shawnshekari/strixite:v0.3.1 (also :latest) - running it.

Contributors: @shawnshekari

v0.3.0

New

• Images. Chat requests can carry images - PNG, JPEG, WebP, GIF (first frame), BMP and TGA, sent as data: URLs in image_url parts of user messages. They're resized the way the model's own image processor does it (bicubic, antialiased; up to ~2.1 megapixels - a 1080p screenshot is read whole, about 2,000 tokens), and a follow-up turn reuses an image from the prompt cache instead of encoding it again. The vision encoder takes about 0.2 s for a small image and 0.9 s for a full-size screenshot on the iGPU. On by default; it costs nothing when no image is sent. Video, HEIC / AVIF, TIFF, SVG and remote URLs are a 400 that says what it got. Details.
• A cap on thinking, on by default. A request that doesn't set its own thinking budget thinks at most 2,048 tokens; past the cap the server waits for the end of a sentence (up to 256 more tokens) and then tells the model to answer. A request's own budget always wins; thinking-cap = 0 turns the cap off. In a day of my own agent use with a 1,024-token cap, thinking fell from ~70% of all serving time to ~44%; 2,048 is the default after an overnight comparison on the longest-thinking coding task I test with: every cap passed, and at 2,048 the agent needed 35 steps against 70 at 1,024 (one run each).
• Precision switches for the prefill's delta rule and attention (gdn-precision, attention-precision: exact, the default, or fast).

Fixed

• The model no longer copies the "answer now" sentence. Clients that send past reasoning back (opencode does) kept every budget stop in the prompt, and after a few turns the model started opening its own thinking with a paraphrase of it and thinking almost nothing. The server now drops those sentences - and their paraphrases - from earlier turns.

Upgrading

Image input needs two files the weights repo has had since today: vision.strixw (~0.9 GB) and Qwen3.8-Flash-Next/preprocessor_config.json. Run your hf download command again - it fetches only the new files - or set vision = off to stay text only. With vision on and either file missing, the server refuses to start and names it.

Container

podman pull ghcr.io/shawnshekari/strixite:v0.3.0 (also :latest) - running it.

Contributors: @shawnshekari

v0.2.4

New

• Prefill progress in the stream. A streamed request with "return_progress": true gets prompt_progress chunks (total, cache, processed, time_ms) when the prefill begins and after each prefill chunk - the same field and keys as llama.cpp's server, so clients that already show it for llama.cpp (llama.cpp's own web UI, for one) show it for strixite too. Works on /v1/completions as well; off unless asked. Thanks @TheBeaninator (#4). Details.

Faster

• 7 draft tokens per step (was 5). Decode is about 1-6% faster on coding-agent work - most on short contexts, about 2% at 128k.

Fixed

• Tool calls begun inside the think block now run. The model sometimes started a <tool_call> before closing its thinking; the call was read as reasoning text, nothing ran, and the turn could go on until the client gave up. The server now closes the think block ahead of such a call (think-close-on-call, on by default).
• Prompt-cache writes give their memory back as they go. A queued write held all of a saved conversation's RAM until it finished; now each part is released as it is written, so memory dips under pressure are shorter.

Container

podman pull ghcr.io/shawnshekari/strixite:v0.2.4 (also :latest) - running it.

Contributors: @shawnshekari, @TheBeaninator (#4)

strixite v0.2.3

v0.2.2 was tagged but never released; this release carries all of it, plus what came after.

New

• Streamed tool calls. A tool call's arguments now stream as they're generated, instead of arriving in one piece once the call is complete - clients show them as they come. Thanks @TheBeaninator (#5).
• The think-block / tool-call guard. The model sometimes reached for end-of-turn while naming a tag in inline code ("until </tool_call>"), ending a turn mid-thought with an empty reply, or mid tool call. The server now takes end-of-turn off the table inside an open think block or tool call, so the turn goes on to its answer.
• Literal tags. Request text is always plain text: a <tool_call> or <think> written in a message, or in a tool call's arguments, is never read as a control token.

Memory safety (#6)

• The prompt cache never grows into memory that isn't there: it makes room before a save or load allocates, and checks its margin every second in the background, so another process growing between requests is answered within seconds.
• A startup memory check: after loading, the server checks it has room for its margin plus one full-capacity saved conversation. Short of it, it exits with status 4 and names the settings that fit (low-memory = fail, the default); low-memory = adapt starts with its caches turned down instead.
• A lost GPU context exits the process (status 3), so systemd or your supervisor restarts it cleanly instead of every request failing.
• New: Memory - where the 128 GB goes, what the server checks, and recipes for sharing the machine with a desktop or another model.

Changed defaults

• The built-in defaults are the values in deploy/strix-server.conf - 512k context, YaRN, 16k-token prefill chunks: the server runs the same with or without --config.
• The prompt cache's RAM margin is 4 GiB (was 24). The defaults assume a Strix Halo dedicated to strixite - that's where it's fastest. If yours is also a desktop or runs another model, see Memory.

Container

podman pull ghcr.io/shawnshekari/strixite:v0.2.3 (also :latest) - running it.

Contributors: @shawnshekari, @TheBeaninator (#5), @Rose22 (suggested in #3), @enncoded (#6)

v0.2.1

A build fix release: strixite now builds outside the one setup I test on. No change to the server's behaviour, speed
or weights; if v0.2.0 runs for you, there's nothing new to get here.

Fixed

• TheRock can live anywhere. The strix preset pinned the ROCm path to ~/tools/therock-tarball/install, and a
  preset's values win over everything else, so neither ROCM_PATH nor a STRIX_ROCM_ROOT environment variable
  could change it. Now: cmake --preset strix -DSTRIX_ROCM_ROOT=/path/to/install (or export STRIX_ROCM_ROOT=...),
  and the compiler follows from it. The old path is still the default.
• GCC 16 hosts build. On distributions with GCC 16 (Arch, Fedora 44, ...) the build stopped with
  error: expected identifier in <format>: outside a HIP compile, HIP's headers define __noinline__ to nothing,
  and GCC 16's <format> uses [[__gnu__::__noinline__]]. Fixed in the source. Tested with GCC 16.2.1.
• A clear error instead of a broken build. A configure-time check now catches a C++ standard library the ROCm
  clang can't compile and says what to do. -DSTRIX_GCC_INSTALL_DIR=/usr/lib/gcc/x86_64-pc-linux-gnu/15 picks the
  GCC to build against.

Docs

The Quick start now covers where the build looks for TheRock, picking a GCC, STRIX_MODELS_DIR for weights kept
somewhere other than ~/models/strix-infer (every default path follows it), and why --config matters: without
it the server runs on its built-in defaults, a 256k context instead of 512k.

The container image is unchanged: ghcr.io/shawnshekari/strixite:v0.2.0 is still current.

Thanks to @Rose22 for the detailed report (#3).

Contributors: @shawnshekari

strixite v0.2.0

The second release of strixite. Faster prefill at long context, structured output, the first two outside
contributions, and pages that explain the weights and the server's behaviour.

	v0.1.0	v0.2.0
Prefill, 32k context	1,547 tokens/s	1,602 tokens/s	+3.6%
Prefill, 120k context	1,494 tokens/s	1,555 tokens/s	+4.1%
Decode, 32k context	54.8 tokens/s	57.6 tokens/s	+5.1%
Decode, 120k context	53.7 tokens/s	57.0 tokens/s	+6.1%

Both releases built from their tags and measured the same afternoon on the same machine (one Strix Halo, 128 GB),
two runs each at each depth, the median shown. Decode in these runs includes the model's thinking and its MTP
drafting, so it moves with what the model writes; the prefill numbers are the steadier comparison.

What's new

• Structured output - response_format json_object and json_schema, enforced token by token by a grammar
  compiled from the schema. Thinking is untouched; the constraint starts after the think block. What is enforced,
  what is ignored, and what is refused with a 400 naming the keyword (never accepted and left unenforced):
  docs/server.md.
• Faster prefill at long context - 16,384-token prefill chunks (+3.0% / +2.8% at 65k / 131k), the experts'
  combine step reading half the bytes (BF16 partials, the same long-context accuracy as FP32), and fewer barriers in
  the attention loop.
• MTP: no wasted forward after a rejected draft - the token sampled at a rejected position starts the next step
  instead of being forwarded alone. Contributed by @TheBeaninator (#1).
• MTP: drafts read a 4-bit copy of the LM head rows they score - half the bytes per draft call, made once at
  load; the verify still decides every token, so output is unchanged. 2-4% faster MTP decode. Contributed by
  @TheBeaninator (#2).
• Thinking controls, documented - every spelling clients use for thinking on / off and budgets, the thinking
  server default, and the thinking nudge (now triggered by repetition only; the length trigger was removed after it
  measured as ignored or harmful): docs/server.md.
• What quant is this? - every part of the weights, its format, why, and what it costs in accuracy:
  docs/weights.md.

Container image (new)

Don't want to build? The server and its ROCm runtime come ready to run as a ~400 MB image, on any Linux
distribution with podman or docker - you still download the weights and set the host's kernel parameters:

podman pull ghcr.io/shawnshekari/strixite:v0.2.0

Every step, each podman run flag explained, start at boot and troubleshooting:
Run strixite in a container.

Weights

Unchanged: strixw format version 4, the files at
wemoh/Qwen3.8-Flash-Next-strixw. The MTP draft head's
4-bit copy is made at load from the same file.

Upgrading

Rebuild and restart. deploy/strix-server.conf gained think-nudge-rate, think-nudge-min-tokens and thinking
(all at their previous behaviour) and chunk = 16384. If you keep your own config, copy the chunk line over -
the built-in default is 512, much slower for long prompts.

Contributors: @shawnshekari (everything not credited otherwise above - structured output, the prefill work, the
thinking nudge and the docs) and @TheBeaninator (#1, #2). Thanks to @TheBeaninator for the first outside pull
requests - both measured carefully and easy to review.

Linux and Strix Halo only. AGPL-3.0; the weights are under the Qwen Community License 1.0.

strixite v0.1.0

The first public release of strixite: the LLM inference engine I wrote from scratch in HIP for AMD Strix Halo
(gfx1151), running Qwen3.8-Flash-Next.

	on one AMD Strix Halo, 128 GB
Decode, real agent use	~47-50 tokens/s, flat from 0 to 492k context
Context	512k tokens
Prefill	~1,370 tokens/s over a real 490k-token conversation
Agent tasks	19 / 19 on terminal-bench-mini's core suite

What's in it

• strix_server - the engine behind an OpenAI-compatible API: chat completions with streaming, tool calls and
  reasoning, multi-token prediction (up to 5 drafts per step), a prompt cache in RAM and on disk, 512k context with
  YaRN
• inspect_strixw - inspect and verify a weights file (--verify re-hashes every tensor)
• convert_qwen4exp, convert_ngram_table, transcode_ngram_table - rebuild the weights from the original
  checkpoint
• deploy/strix-server.conf (every setting with the reason for its value) and a systemd user unit

Weights

Reads strixw format version 4 - the files at
wemoh/Qwen3.8-Flash-Next-strixw (~115 GiB, checksums
included).

Tested setup

• AMD Ryzen AI MAX+ 395 / Radeon 8060S (gfx1151), 128 GB, Fedora 43 (kernel 7.1)
• Kernel parameters amdgpu.gttsize=126976 ttm.pages_limit=32505856 amd_iommu=off, BIOS VRAM at its 512 MB minimum
• TheRock ROCm therock-dist-linux-gfx1151-10.1.0a20260822 (HIP 7.16)

Getting started

Source only - build it with the Quick start in the README.
The server has no authentication and listens on all interfaces by default; run it on a trusted network or set
host = 127.0.0.1.

Linux and Strix Halo only. AGPL-3.0; the weights are under the Qwen Community License 1.0.