Skip to content

glm5.2 branch: DeepSeek Flash decode regression (~2.8x) introduced by "Add GLM 5.2 runtime support" #532

Description

@andreaborio

While benchmarking our GLM fork line against main we found that the whole
glm5.2 branch decodes DeepSeek Flash about 2.8x slower than main, and bisected
it to the first commit of the line.

Machine: Apple M5 Pro 64 GB, macOS 26 (Darwin 25.5.0), Metal.
Model: DeepSeek-V4-Flash IQ2XXS (w2Q2K, AProj/SExp/Out Q8, chat-v2 imatrix),
80.8 GiB file, so model >> RAM and --ssd-streaming. Probe:

./ds4-bench -m MODEL.gguf --ssd-streaming \
    --prompt-file speed-bench/promessi_sposi.txt \
    --ctx-start 2048 --ctx-max 2048 --step-incr 2048 --gen-tokens 32

Single-commit points are ctx 2048 only, rebuilt via make after each
checkout, and their ranges combine two measurement sessions on separate
days; the "main" and "our line" rows are interleaved arms from one session
(A/B/C x2 back to back) swept at ctx 2048-4096.

commit gen t/s
main (5b95fa1) 6.8-8.3 (4 interleaved runs, ctx 2048-4096)
main (80ebbc3, current HEAD) 6.49 (1 run)
173a9a3 "Add GLM 5.2 runtime support" 1.94-2.21
34b3736 (pre pro-correctness merge) 2.76-2.88
bd89932 (glm5.2 HEAD) 2.73 (1 run)
our GLM branch (bd89932 + our fixes, default env) 1.9-2.8 (8 interleaved runs, ctx 2048-4096)

A clean detail for bisecting: 80ebbc3 (current main, fast) is the direct
parent of 173a9a3 (slow), so the whole regression sits inside that one
commit. It persists through the branch HEAD, is untouched by streaming env
configuration (disabling the expert readahead does not move it), and main
is not affected. First token on the glm5.2 line takes ~5.2-7.2 s
(gen_first_ms 5166-7180 across all runs); main's bench build predates the
first-token/steady columns, so for main only whole-generation t/s is
directly comparable, and on the glm5.2 line even the steady phase after
the first token (gen_steady_tps 2.6-6.5) stays below main's whole-run
6.8-8.3.

Prefill shows no clear regression once the page cache is warm (fork line
107-120 vs main 116-118 t/s at these sizes in the warm interleaved reps;
first-rep deltas are cache-warming noise).

We have not root-caused inside 173a9a3; happy to bisect deeper into it or
test patches on this box. Repro scripts available.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions