Kyojin 1.5.6

Kyojin 1.5.6

Faster first token on both models, above all for new conversations that reuse a long system prompt. Decode speed is unchanged. Everything below is on by default in tools/glm/serve.py and tools/qwen/serve.py.

All figures were measured served, on a Strix Halo box (128 GB), with one load per arm.

New conversations on a long cached system prompt (both models)

A prompt that shares cached pages with an earlier conversation, but not its recurrent state, now writes that state where the shared pages end (EXL3_BRANCH_CKPT). The next conversation resumes there instead of replaying the whole system prompt.

Case	Before	1.5.6
Qwen, 11.6K shared system prompt + ~700-token first turn	2.97 s	0.96 s
GLM, 9K shared prefix	2.41 s	1.42 s
GLM, 12.1K shared prefix	3.84 s	1.51 s
GLM, 14.7K system prompt + 4.5K-token turns, first new conversation	28.4 s	8.0 s

On GLM, cold prompts also checkpoint every 2048 rows before the tail window (EXL3_CKPT_INTERVAL_PP). An off-grid resume keeps its short remainder in one chunk (EXL3_TAIL_OFFGRID).

GLM-5.3-Flash

• Mid-size prompts. Large dense layers decode their weights inside the GEMM for forwards of up to 1023 rows (EXL3_DENSE_MPW2). First token on 200 to 900-token prompts is 5.7 to 12.2 % faster, cold and follow-up. Top-1 agreement with an fp32 reference never falls below the previous path over 96 rows.
• Garbage collector freeze. Long-lived objects stay out of collection, as in the Qwen server (EXL3_GC_FREEZE). First token is 2.1 % faster at 8K and 7.9 % faster at 2K.
• Texts and speed. Short greedy texts are equal to 1.5.5 (6 of 6). Decode is x1.008.

Qwen3.8-Flash-Next

• Cold prompts. GDN prefill now runs as one kernel per layer, in 16-row chunks, instead of two kernels (EXL3_GDN_HF). The kernel is adapted from hipfire; its NOTICE and Apache-2.0 licence ship next to it. Cold 4K first token is x0.83 (mean of 4 samples).
• GDN-HF quality. The kernel is not bitwise. Perplexity against the true next tokens is equal: 4.91972 before, 4.91967 after, over 49K paired positions. Some greedy texts therefore differ from 1.5.5 after a few tokens.
• Short prompts. The tensor-core MoE GEMM starts at 9 rows instead of 32. First token on short chat prompts takes 212 ms instead of 360 ms.
• Follow-up turns. The tail checkpoint is kept across gaps of up to 8192 tokens (EXL3_TAIL_MAXGAP). On 12K-token agent pages, the median first token is x0.847.
• Bit-identical changes. The attention and GDN blocks keep o_proj's fp16 output (EXL3_NC2_ATTN_F16, EXL3_NC2_ABLIT_F16).
• Decode is x0.993, within noise.

Other engine changes

• The router uses a multi-row Triton top-k instead of a stable sort over all experts.
• MoE prefill adds the gated shared expert in one pass.
• The QSA indexer projects the row-padded input as a view.
• GLM gets an opt-in 16-row KDA prefill kernel (EXL3_KDA_HF, off by default).

How to upgrade

1. Pull the 1.5.6 tag.
2. Rebuild the extension with the build step from doc/install.md. Native files changed.
3. Restart your servers.

No weights change. Existing packs and slot files work as they are.

Kyojin 1.5.5

Kyojin 1.5.5

GLM-5.3-Flash: faster prefill at long context. Prefill now uses big chunks at every depth instead of stopping at 32K.

Context	1.5.4	1.5.5	Gain
128K	595 tok/s	670 tok/s	+12.6 %
256K	546 tok/s	612 tok/s	+12.1 %
500K	504 tok/s	557 tok/s	+10.5 %

Quality was checked against the true next tokens on 3 independent sets of long-context windows: perplexity is unchanged or slightly better, next-token accuracy is unchanged, and the 500K needle test still passes. Peak memory still fits a 500K context on a 128 GB Strix Halo.

Follow-up prefills on an already long context (the next turn in a chat) gain 8 to 40 % as well.

Measured on a Strix Halo box (128 GB), served, same machine, one load per arm.

Kyojin 1.5.4

Kyojin 1.5.4

Faster GLM-5.3-Flash prefill. An 8K prompt now runs as one 8192-row chunk instead of two 4096-row chunks. Qwen is unchanged.

What got faster

Served on a Strix Halo box (128 GB), same machine, against the 1.5.3 ladder, tokens per second for prefill:

Prompt	1.5.3	1.5.4	Change
8K	664	697	+5.0 %
32K	672	681	+1.3 %

Quality: logits, 16 greedy ids and restored-prompt hashes are equal to the previous build at 8K and 32K. On the box gate, logits are bitwise equal on 24 of 24 prompts. The slot-tail reload proof passed.

What changed

• EXL3_PREFILL_BIG_CHUNK=8192 and EXL3_NC2_CKPT4 are now in the GLM server defaults. The second flag adds 4-slot KDA checkpoints so a single 8192-row chunk can still be restored.
• The mHC one-pass ticket capacity is 65536 rows, so the big chunks keep the fused mHC paths.

How to upgrade

1. Pull the 1.5.4 tag.
2. Rebuild the extension with the build step from doc/install.md. Native files changed.
3. Restart your GLM server.

No weights change.

Kyojin 1.5.3

Kyojin 1.5.3

Faster GLM-5.3-Flash prefill and faster follow-up turns in agent chats. Long-context decode no longer loses speed to bad copy matches.

What got faster

• Follow-up turns (GLM). A turn that adds a few hundred to a few thousand tokens now runs as one forward pass instead of two. Measured on a 5-turn tool chat: a 2040-token turn took 3.35 s instead of 4.34 s (-23 %), a 900-token turn 2.14 s instead of 2.58 s (-17 %). Turns that already took one pass are unchanged. This is EXL3_TAIL_MAXGAP=8192, now in the GLM server defaults.
• Slot files keep the prompt-end tail. A saved session slot now stores the tail checkpoint (format 2), so a reloaded session resumes at the end of its prompt. Reloaded logits are bitwise equal and the top-1 token is the same. Older slot files still load.
• GLM prefill. Prefill kernels now write GEMM-padded rows, keep attention outputs in fp16 where the next step reads fp16, and add the shared expert in the reduce step. All of it is on by default in the GLM server. The cold-prompt logits are bitwise equal to the previous build on 24 of 24 prompts from 300 to 8192 tokens, and the greedy text is equal on all 24.
• GLM lookup at long context. Lookup rounds now run only while they beat the drafter, with a probe every 32 rounds. At 128K, decode is 29.1 tok/s against 27.6 tok/s before, measured on a Strix Halo box (128 GB). QWSPEC_NGRAM_GATE=0 restores the old behaviour.

Full prefill and decode figures from 8K to 500K follow in the README.

How to upgrade

1. Pull the 1.5.3 tag.
2. Rebuild the extension with the build step from doc/install.md. Native files changed.
3. Restart your servers. Everything above is active at once. Each prefill flag can be turned off with its EXL3_NC2_* variable if you need the old path.

No weights change. Existing packs and existing slot files work as they are.

Kyojin 1.5.2

Kyojin 1.5.2

Faster decoding on GLM-5.3-Flash and Qwen3.8-Flash-Next. The output text did not change in our checks.

What got faster

Served on one Ryzen AI Max+ 395 with 128 GB, greedy requests, same prompts before and after, tokens per second for decode.

GLM-5.3-Flash

Request kind	Change
Copy and edit	+20.6 %
Structured	+1.9 %
Prose	+2.0 %
Mean of the three	+8.6 %

Qwen3.8-Flash-Next

Request kind	Change
Copy and edit	+13.0 %
Structured	+0.3 %
Prose	+1.4 %
Mean of the three	+5.2 %

These figures are measured against servers that already ran the two-stage draft head (below). The head alone adds +3.6 % on GLM and +5.4 % on Qwen.

Public figures, new-user install of this release (README install, published packs, default environment; full tables in doc/benchmarks.md):

Model	Prefill 8K	Speculative decode, prose to code	1.5
Qwen3.8-Flash-Next	1494	47.5 to 73.2	1481, 45.7 to 66.8
GLM-5.3-Flash	652	27.4 to 36.8	634, 26.2 to 33.4

Other gains:

• GLM decode with the new MLA VALU path: +2.0 to +2.4 % on top of the draft head.
• GLM at one session needs about 6 GiB less GPU memory (one recurrent slot per session instead of 16). With four sessions it needs 4.9 GiB less and the total decode speed goes from 38.8 to 40.4 tok/s.
• Qwen: the first token no longer waits for a 0.5 s collector pass after a cache_salt change. Prefill of an 8K prompt: 1306 to 1444 tok/s.

Quality: greedy output text was equal to the previous build on every prompt we compared. Speculative decoding still returns the same tokens as plain decode.

What changed

Two-stage draft head (on by default when the pack is there). The MTP drafters of GLM and Qwen pick draft tokens with a short list first and the exact head second. The pack ships with the model as draft2s.st (GLM-5.3-Flash-EXL3-Yamz, its Uncensored twin and Qwen3.8-Flash-Next-EXL3-Yamz on Hugging Face): download it with the model and the server uses it when its header matches the model. EXL3_DRAFT2S=<path> picks another pack; EXL3_DRAFT2S=off turns it off. Build your own with tools/qwen/draft2s_pack.py. Sampled requests (temperature above 0) keep the full head, because the exact accept rule needs the drafter's full distribution. No draft2s.st in the model folder = the old path.

GLM MLA VALU decode path (on by default). It runs the MLA absorb and unfold steps for 1 to 8 decode rows on a new kernel. Results are bitwise equal in our checks. EXL3_MLA_DEC_VALU=0 turns it off. This is a native change: rebuild the extension. An older build without the kernel falls back to the Triton path with one log line.

GLM copy lookup, on by default with one session. When the last 16 tokens appeared before, the server verifies the 7 tokens that followed. Default GLM_NGRAM=2,7,16,3. With more than one session it stays off. GLM_NGRAM=off turns it off. On a long answer that meets a near-tie, greedy text can differ from a run without lookup.

Qwen lookup. It takes a 16-token match, drafts 7 tokens and copies only what the two latest occurrences agree on. QWSPEC_NGRAM=2,3,5 restores the old setting. The collector freeze (EXL3_GC_FREEZE=0 to disable) removes the first-token pause. It is on for Qwen and opt-in on GLM.

GLM recurrent slots. One slot per session. EXL3_GLM_SLOTS=16 restores the old layout. Verify rounds of 5 to 8 rows use the same kernels as the Qwen server.

Sampled requests. With temperature above 0, GLM now draws the drafter's tokens from its own distribution and keeps them with an exact accept-or-resample rule. The output has the same distribution as plain sampling. EXL3_SPEC_SAMPLED=0 goes back to the old rule. Greedy requests are untouched. Qwen keeps the old rule.

Fix: Qwen with 5 or more sessions. With the MTP drafter on, a Qwen server started with --sessions 5 to 8 failed every decode step (AssertionError in moe_fused.py). The drafter layer now splits a batch of several jobs over its rows. One session runs exactly the old path.

Docs. doc/env_vars.md lists GLM_NGRAM, QWSPEC_NGRAM, QWSPEC_NGRAM_IDX, EXL3_SPEC_SAMPLED, EXL3_GLM_SLOTS, EXL3_GC_FREEZE, EXL3_DRAFT2S and EXL3_MLA_DEC_VALU. tools/glm/SERVE.md notes that tools/qwen must ship next to tools/glm.

How to upgrade

1. Pull the 1.5.2 tag.
2. Rebuild the extension with the build step from doc/install.md. This enables the MLA VALU path; an old build still runs on the Triton path and logs one line.
3. Restart your servers. Lookup, slots, sampled drafts and the MLA VALU path are active at once. For the draft head, download draft2s.st into your model folder (hf download yamz-labs/<model> draft2s.st --local-dir <model folder>); it is then used without a flag.

No weights change. Existing packs work as they are; draft2s.st is one new optional file (about 310 to 330 MB).

Kyojin v1.5.1

The Qwen and GLM servers gain the vLLM-style routes that standard benchmark clients expect.

What is new

• POST /tokenize on the Qwen and GLM servers.
• Token-id prompts on /v1/completions: prompt may be a list of token ids, sent to the model as is. The GLM server gains /v1/completions (stream and non-stream).
• cache_salt: requests with different salts never share the prompt cache. Requests without the field behave as before.
• Usage reports prompt_tokens_details.cached_tokens on these routes; ignore_eos is accepted; the GLM server reads chat_template_kwargs.

With these, standard benchmark clients that use the vLLM-style routes (RigMark for one) run unchanged against Kyojin.

How to update

git pull

No new pack to download. The container images stay at 1.5 for now; these routes are in the source install.

Kyojin v1.5

GLM-5.3-Flash starts in a minute and a half instead of thirteen, reads a new image in seconds, and serves several users at once. The prompt cache now survives images and edits on all servers.

What is new

• GLM: fast first start. A first start on a fresh install takes 95 s (it was about 13 minutes). The dense GEMM screening is off by default; prefill and decode speed are at the 1.4 level and answers are byte-identical. EXL3_DENSE_GEMM_TUNE=1 brings the tuner back.
• GLM: images without the wait. The first image of a new size costs seconds (a first 1024x1024 went from 433 s to 4 s). Thanks to @MangetsuC for this work (#64).
• GLM: several users at once. --sessions N decodes N requests together: 30 tok/s for one request, 37.5 in total for two at once, 40.6 for four.
• Prompt cache survives an image on the three servers: a conversation that holds an image no longer re-reads everything after it at every turn (#65).
• Follow-up prompts resume at the exact token. A repeated or edited prompt reuses all it can, and a repeat after an edit no longer returns the earlier continuation.
• --exact-prefix-cache (GLM, optional): a request resumed from the cache gives the same greedy text as a cold one, for about 5-12 % of cold prefill speed.
• MiMo vision tower 5x faster.
• Large requests. The GLM and MiMo servers accept bodies up to 64 MiB, as the Qwen server does.
• Container. A server that does not stop within 20 s of docker stop is killed, so the GPU memory is always freed (KYOJIN_STOP_TIMEOUT). A GLM pack without mtp_eh_proj.st is reported at start.

Figures

One Ryzen AI Max+ 395 with 128 GB; new-user install, published packs, default environment, server-reported prefill, speculative decode. Tokens per second.

Model	Prefill 8K	32K	64K	128K	256K	Decode prose / chat / code
Qwen3.8-Flash-Next	1486	1478	1454	1397	1349	45.8 / 48.3 / 67.0
GLM-5.3-Flash	623	629	602	-	-	26.3 / 30.8 / 33.0

How to update

git pull

Container users: docker pull ghcr.io/yamz-labs/kyojin-qwen:1.5 or ghcr.io/yamz-labs/kyojin-glm:1.5 (first container start: about 10 minutes for Qwen, 95 s for GLM; then 96 s and 64 s). No new pack to download.

This is still an early version; more improvements are coming. Reports and ideas are welcome in the issues.

Kyojin v1.4

The servers now answer well with no client settings: each model's recommended sampling, a thinking effort that fits the task, and no fixed reply limit. Qwen decodes faster on code and at 256K of context.

What is new

• Recommended sampling by default. The GLM and MiMo servers take temperature and top_p from the model card (1.0 and 0.95), as the Qwen server already did. A client that sends no sampling no longer decodes greedily.
• Thinking effort medium by default on the three servers. Medium is the model's own chat template plus one instruction: plan in the reasoning, write code and long text once in the answer, and keep a point once it is settled. On open tasks the models reach the answer with far less reasoning. --default-reasoning-effort selects another level, and --chat-template <pack>/chat_template.jinja serves the model's unchanged template.
• No fixed reply limit. A request that sets no max_tokens can use all the context left after the prompt (it was 4096 tokens on GLM and MiMo, 32768 on Qwen). --default-max-tokens sets a limit.
• Faster speculative decode on Qwen3.8-Flash-Next: code prompts go from 61.0 to 66.8 tok/s, and at 256K of context decode goes from 41.8 to 44.3 tok/s and prefill from 1292 to 1347 tok/s.
• Qwen server, first requests. After the weight load, part of the server's memory could sit in swap, and the first request on new text paid for reading it back. The server now reads those pages back before it reports ready.
• Full curves for the three models. GLM-5.3-Flash is now measured to 256K of context and MiMo-V2.6-Flash to 128K, with the same recipe as Qwen: see the charts and doc/benchmarks.md.
• GLM: the draft pairing fix reported in #35.
• Install notes for GCC 16 and more distributions (#37).

Figures

One Ryzen AI Max+ 395 with 128 GB; new-user install, published packs, default environment, server-reported prefill, speculative decode. Tokens per second; method in doc/benchmarks.md.

Qwen3.8-Flash-Next, 1.3 and 1.4:

	1.3	1.4
Decode, code prompts	61.0	66.8
Decode, chat prompts	48.6	48.3
Decode, prose prompts	45.7	45.7
Decode at 256K	41.8	44.3
Prefill at 8K / 32K / 64K	1471 / 1463 / 1439	1481 / 1477 / 1436
Prefill at 128K / 256K	1390 / 1292	1396 / 1347

The three models in 1.4:

Model	Prefill 8K	32K	64K	128K	256K	Decode prose / chat / code
Qwen3.8-Flash-Next	1481	1477	1436	1396	1347	45.7 / 48.3 / 66.8
GLM-5.3-Flash	634	637	613	589	567	26.2 / 31.4 / 33.4
MiMo-V2.6-Flash	760	702	599	458	-	28.9 / 31.2 / 37.0

Decode by context and the comparison with other engines are in doc/benchmarks.md.

How to update

git pull

No new pack to download. To keep the 1.3 behaviour on a server: --default-reasoning-effort with the model's own level, --default-temperature 0, --default-max-tokens 32768.

This is still an early version; more improvements are coming. Reports and ideas are welcome in the issues.

Kyojin v1.3

Kyojin 1.3

Long prompts keep their speed, speculative decoding costs less per step, the 16-bit context cache is the default again, and the servers report their start-up and their load. Thank you to everyone who sent reports and pull requests.

What is new

• Long-context decode. The draft model now reads the prompt together with the main model instead of afterwards, so a long prompt no longer slows the start of the reply. Qwen3.8-Flash-Next decodes at 41.8 tok/s with speculation after a 256K-token prompt, against 20.4 in 1.2.
• Cheaper speculative step. The verify step does fewer host round trips. Reported by @felladrin in #25. Qwen speculative decode on the card prompts: chat 48.6, prose 45.7, code 61 tok/s.
• 16-bit cache by default. The Qwen server uses the 16-bit K/V cache again. The 8-bit cache stays available with --cache-bits 8; it saves about 2.6 GiB at 256K.
• --sessions N (Qwen server): decode up to N requests together. Default is 1. Contributed by @felladrin (#24).
• /health with load progress on the Qwen, GLM and MiMo servers. The port answers from the first second: 503 with the stage, a percentage and a plain message while the model loads, then 200 {"status": "ok", "source": "kyojin"}. Works with llama-swap's checkEndpoint. Suggested by @mostlygeek (#20), start-up fixes by @felladrin.
• /metrics (Prometheus, llama.cpp metric names) on the GLM and MiMo servers, contributed by @nobert.
• GLM, two jobs together. Two requests decoded together now give the same tokens as each one alone. Reported by @ChrisC381 (#28) with a precise isolation: the router, the attention split and a tie-break in the top-k were the causes. EXL3_ROW_INV=0 restores the old path.
• Streaming ends with finish_reason, reported by @dturini12 (#10).
• Start-up. /health stays answerable during the first dense GEMM tuning of GLM, and a client that hangs up on /health no longer logs an error.

Before and after

One Ryzen AI Max+ 395 with 128 GB, same method for both columns (new-user install, published packs, greedy, engine-reported prefill, speculative decode). Tokens per second.

Qwen3.8-Flash-Next	1.2	1.3
Prefill 8K	1,494	1,471
Prefill 32K	1,478	1,463
Prefill 64K	1,457	1,439
Prefill 128K	1,395	1,390
Prefill 256K	1,277	1,292
Decode 8K	43.4	47.1
Decode 32K	40.2	49.6
Decode 64K	36.7	48.8
Decode 128K	31.9	46.9
Decode 256K	20.4	41.8
Chat	49.8	48.6
Prose	44.5	45.7
Code	59.1	61

GLM-5.3-Flash and MiMo-V2.6-Flash: speeds as published for 1.2, not measured again for this release.

How to update

cd kyojin
git pull
source tools/strix_halo/env.sh
./build.sh

Packs do not change: no new download. Restart the server. The first requests after a rebuild tune a few kernels and can be slower.

Comparison with Strata and Gufo

Prefill, speculative decode and fidelity against the original FP8 model, one machine, one method for every engine we ran (rival versions and packs named), plus an independent Strata prefill test: doc/benchmarks.md.

Thanks

@felladrin (the speculative-step report, --sessions, start-up fixes, the bench tool tests and --json, the troubleshooting page, triage), @nobert (/metrics), @ChrisC381 (the GLM paired-jobs report), @dturini12 (the streaming report), @mostlygeek (the llama-swap and container notes behind /health), @morrisfamily (benchmark report).

These are first versions. Speed, context length and model support keep improving.

Kyojin v1.1: Qwen3.8-Flash-Next

Kyojin v1.1: Qwen3.8-Flash-Next

Kyojin is the Yamz Labs inference engine for AMD Strix Halo, built on ExLlamaV3.

New

• Qwen3.8-Flash-Next support, end to end: server (tools/qwen/serve.py), speculative decoding on by default, vision tower optional (--no-vision saves memory), context tested to 128K tokens.
• Faster prefill on the recurrent layers, and one fused launch for two small projections in decode. Both bit-identical to the previous path.
• A batch of server fixes: an abandoned or failed request no longer leaks into the next one, dead clients stop generating, prompt-cache restore is exact.
• First-use kernel tuning now runs at load, so the first request is not slow.
• Optional runtime preset hook for Qwen3.8 (off by default): see the model card.

Numbers (same machine, defaults)

1412 tok/s prefill at 4K, 32.7 tok/s plain decode, 44.3 with speculation on chat. Versus Halogen 0.16.2 with its v2 checkpoint: it is faster on plain decode, most tasks with speculation and prefill; we are closer to the original model on KL. Full table on the model card.

Update

git pull && ./build.sh && source tools/strix_halo/env.sh
python tools/qwen/serve.py --model /path/to/pack -c 131072

GLM and MiMo packs keep working with their own servers.

Defaults

Context 65536 unless -c is given. Draft policy mix. Thinking follows the chat template. Sampling from the model's generation_config.json.

Tell us what you measure: https://github.com/Yamz-Labs/kyojin/issues