Add GLM 5.2 inference (SSD streaming and distributed) on ROCm for GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf

[kyuz0]

This PR adds inference support for GLM 5.2 on AMD Strix Halo systems using the ROCm backend for for GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf. This is both for SSD streaming and for distributed inference.

The model can now be split by layer across two nodes and kept resident in memory. Each node maps and loads only its assigned layers, so distributed inference does not require SSD streaming.

It uses the ROCm fixes and optimized attention, projection, MoE, and decode paths needed to make GLM 5.2 inference correct and practical on gfx1151.

The FP16 hipBLAS attention approach was informed by mgiustiniani's GLM 5.2 work.

Running

The following example splits the 78 transformer layers across two Strix Halo systems:

• coordinator: layers 0:37, address 192.168.100.2
• worker: layers 38:output, address 192.168.100.1

Start the coordinator:

DS4_GLM_MEMORY_GUARD_RESERVE_GB=8 \
ds4-server \
  -m /path/to/GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf \
  --ctx 30000 \
  --host 0.0.0.0 \
  --port 8000 \
  --role coordinator \
  --layers 0:37 \
  --listen 192.168.100.2 8081 \
  --dist-prefill-chunk 256 \
  --dist-prefill-window 2

Start the worker:

DS4_GLM_MEMORY_GUARD_RESERVE_GB=8 \
ds4-server \
  -m /path/to/GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf \
  --ctx 30000 \
  --role worker \
  --layers 38:output \
  --coordinator 192.168.100.2 8081

--dist-prefill-chunk 256 --dist-prefill-window 2 is the tested prefill configuration.

Testing

This was tested on two Strix Halo systems with 128 GiB of unified memory each, using ROCm 7.2.4 and:

GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf

Validation included:

• intermediate tensor and complete output-logit comparisons against the reference paths;
• coherent API and raw-continuation output;
• prefill and generation at 2K, 4K, and 8K context;
• clean distributed runs without crashes, invalid values, or KV-cache mismatches.

Performance

Results for the two-node 0:37 / 38:output split:

Context	Prefill	Decode
2K	50.79 t/s	2.46 t/s
4K	42.32 t/s	2.42 t/s
8K	23.79 t/s	2.33 t/s

The 8K prefill result measures the additional 4K suffix after an existing 4K context.

Caveats

• Only GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf tested
• This was mostly done iterating with GPT 5.6 Sol

[kyuz0]

I tested the same GLM-5.2-UD-IQ2_XXS_RoutedIQ2XXS_blk78Q2K.gguf model through both the single-node SSD-streaming path and the two-node distributed path.

Both runs used ds4's tracked glm52-openrouter-100 fixture: 100 continuations collected from OpenRouter z-ai/glm-5.2 with top_logprobs=20. The scorer measures how closely each local path reproduces the stored continuation probabilities and token choices.

Metric	SSD streaming	Distributed
Cases	100	100
Scored tokens	2,299	2,299
Average NLL	0.46919	0.46505
First-token matches	81	81
Average greedy LCP	6.53	6.61
API top-1 agreement	86.010%	86.190%
API pairwise agreement	77.587%	77.579%
API top-logprob coverage	99.768%	99.768%

The distributed run completed all 100 cases without ROCm errors, non-finite values, KV-cache mismatches, route failures, or worker crashes. Its quality metrics are effectively the same as the established SSD-streaming path.

[VBart]

Thanks a lot!

Tested it on my Strix Halo 128Gb with SSD streaming from 2xSamsung 990 PRO 1Tb in RAID0 on the GLM-5.2-UD-Q2_K_RoutedQ2K.gguf model.

This patch series fixes significant performance degradation after 2048 token, which I observed before on the current main branch.

Original: gen speed drops from ~2 t/s to 0.14 t/s after context reaching 2048 tokens.
With this patch series generation speed stays around 2.4 t/s.

[kyuz0]

Thanks @VBart. I am still running a few more SWE bench tasks, as I want to make sure that inference is at a certain quality bar before we ask @antirez to review and merge. I might bundle in some fixes for DeepSeek V4 Flash inference, that's currently broken in main and in this branch.

[antirez]

Thank you @kyuz0 ! Merged