glm-flash-lite

Run GLM-5.3-Flash (125 GB, EXL3 3.05 bpw) on one 24 GB RTX 3090 with an OpenAI-compatible API, using RAM and NVMe for the experts that do not fit on the GPU.

Measured on 1x RTX 3090 (PCIe 4.0 x16), AMD EPYC 7443P (24 cores), 8-channel DDR4, and 4x Samsung 9100 PRO in RAID0. Decode runs each answer to its natural end. Full tables, dates and raw files: docs/results.md.

Image: ghcr.io/sybil-solutions/glm53-flash-offload (name kept for existing installs)

Omarchy Local AI. Run the line below. In Local AI, pick GLM-5.3-Flash, EXL3 3.05 bpw (55 GB RAM, experts on NVMe) and press Start. The plugin downloads the weights, builds the store and starts the server.

omarchy plugin add https://github.com/sybil-solutions/omarchy-local-ai --enable

Docker. Needs the NVIDIA container toolkit, the hf CLI, 125 GB for the weights, and 117 GB for the expert store on a fast local NVMe filesystem (xfs or ext4).

IMG=ghcr.io/sybil-solutions/glm53-flash-offload@sha256:aa74200f16c86588b101be2315aed2f643d148559179eef52a02599016b56e69   # v4.6-nvme
hf download turboderp/GLM-5.3-Flash-exl3 --revision 332ab457b709b7ba30dd9a448be5de03b80a7ac9 --local-dir /data/glm53
docker run --rm -v /data/glm53:/models:ro -v /mnt/nvme/glm53:/nvx "$IMG" pack-store     # once: 117 GB expert store
docker run -d --name glm53 --gpus '"device=0"' --memory 55g --memory-swap 55g --shm-size 1g --ulimit memlock=-1 \
  -e GLM53_MODE=nvme -e GLM53_EC_MAX_SLOTS=1376 -p 127.0.0.1:30000:30000 -v /data/glm53:/models:ro -v /mnt/nvme/glm53:/nvx:ro "$IMG"

The server is ready in about 80 s at http://127.0.0.1:30000/v1, model glm-5.3-flash. For 16 GB of RAM use --memory 16g --memory-swap 16g. For bit-exact output use GLM53_MODE=nvme-exact. The all-RAM fast mode uses an older image; see docs/results.md.

Every local-ai-registry recipe whose launch image is ghcr.io/sybil-solutions/glm53-flash-offload or ghcr.io/0xsero/glm53-flash-offload, single-GPU only. Only one hardware card has such recipes: the RTX 3090 24 GB. The GLM-5.3-Flash recipes on other cards (DGX Spark GB10, RTX PRO 6000 Blackwell, CMP 170HX, Apple M5 Ultra) use other engines and images (vLLM, SGLang, oMLX). No recipe runs this image on an Arc Pro B70 alone.

Status: accepted = the registry lab passed all six gates (load, chat, reasoning, tools, context, speed) on that digest; reported = the lab ran it and a gate failed, or only the owner's campaign measured it; candidate = not in the plugin catalog. Lab rows: prefill is the context gate (one 93,437-token prompt, tokens / request time), decode is the speed gate (one stream, first 30 s, natural completion, gate 15 tok/s). Sweep rows: bench/sweep.py, see docs/results.md. "agg" is the aggregate over all streams.

• RAM / storage: the NVMe recipes run in a container capped at 55 or 16 GiB (the registry asks for 59.1 / 17.2 GB free) and need 242.6 GB on a local NVMe filesystem with O_DIRECT (xfs, ext4): 125.3 GB of weights plus the 117.3 GB expert store from pack-store. All three need an x86-64 CPU with AVX2+FMA+F16C; the NVMe recipes ask for 40 or more logical CPUs. The all-RAM recipe needs ~238 GB of free RAM and the 125.3 GB of weights on local NVMe.
• The fast sweep rows ran on the previous image bb633b0b. The full sweep was not re-run on c5a58a40, the digest the recipe pins; only the lab run was.
• Image digests: v4.6-nvme sha256:aa74200f16c86588b101be2315aed2f643d148559179eef52a02599016b56e69 (55 GB recipe accepted in local-ai-registry PR #199); v4.4-nvme sha256:c5a58a40bd93e57227d3986b81e9648d422c465a3ff00e14f17dad65dadf2310; previous all-RAM image ghcr.io/0xsero/glm53-flash-offload@sha256:bb633b0bcb85573ad40b6c408af5e1036ae062c593e42479f4e4d2ab521e551d.

Hardware cost (prices dated 2026-10-08, from the earlier campaign): the cheapest build is the 16 GB RTX 3090 build at ~$5.0k, about $340 per tok/s at its 14.7 tok/s lab decode (reported, below the 15 tok/s gate). The 55 GB build is ~$5.4k, about $270 per tok/s at its accepted 20.0 tok/s.

Experimental, not supported (multi-GPU or companion): glm-5.3-flash.exllamav3-nvme-b70, the 55 GB NVMe recipe on an RTX 3090 plus an Intel Arc Pro B70 running the glm53-b70-expert-server companion (sha256:8e95f5f9f5e2373d5a80c556cdbc24ae5be8e63dd9f1520bdeb0bc0d02bc5fc5). It is a candidate with a reported proof from the owner's campaign N137 (26.5 tok/s C1 decode, 968 tok/s 32k prefill), not lab-run, and the plugin cannot start companions yet. See docs/b70-tier.md.

The model has 12,096 routed experts (42 MoE layers x 288), 9.4 MB each, and each token uses 8 per layer. Each expert is kept in one of three tiers. Two lanes compute them at the same time:

   GPU lane                                       CPU lane
  +-------------------------------+              +---------------------------+
  | VRAM expert cache, ~1,300     |              | 22 AVX2 threads compute   |
  | hottest experts (12.4 GB)     |              | RAM experts in place      |
  +---------------^---------------+              +-------------^-------------+
                  | PCIe copy when admitted                    | reads
  +---------------+--------------------------------------------+-------------+
  | RAM tier, pinned, sized from --memory: ~4,800 experts at 55 GiB          |
  | holds only experts that are NOT in VRAM                                  |
  +---------------^----------------------------------------------------------+
                  | O_DIRECT reads, deep queue, layer-ahead prefetch
  +---------------+----------------------------------------------------------+
  | NVMe store: every expert, 117 GB, one 4K-aligned record each             |
  +--------------------------------------------------------------------------+

• Expert cache in VRAM. The VRAM left after load holds the hottest experts. exllamav3's own kernels read them through repointed tables. On the all-RAM path, cache plus direct reads took decode from 7.5-8.4 tok/s (stock exllamav3) to 12.6.
• RAM tier with no duplicates. An expert evicted from VRAM moves into RAM, so VRAM and RAM never hold the same expert. 55 GiB of RAM then covers ~6,100 different experts instead of ~4,800. Per token at C1: ~120 picks hit VRAM, ~175 hit RAM and ~23 come from NVMe.
• NVMe record store. Each expert is one 4K-aligned record, read with O_DIRECT (no page cache) by 16 threads across a 4-drive RAID0. Capping reads at 8 GB/s cost 8 % of C1 decode, 33 % of C4 and 27 % of prefill.
• CPU tier. Cold experts already in RAM are computed by the CPU where they sit, instead of being copied over PCIe. This runs in parallel with the GPU, which adds the CPU's partial result back. Adding it, together with prefetch, took NVMe decode from 8.3 to 15.3 tok/s, at a decode KL of 0.005.
• Host engine without stalls. One controller thread plans each layer, issues the reads and answers the GPU through mapped memory. The GPU waits on per-expert "landed" flags, with no host sync per layer. Together with the no-duplicates RAM tier, this took exact decode from 7.3 to 8.3 tok/s.
• Layer-ahead prefetch. The next layer's router runs on the current layer's input, and its NVMe experts are read one layer early. NVMe misses fall from 40 to 22 per token, and it is 13 % faster in the same session. An earlier "prefetch off is faster" result was drift between sessions.
• Victim ring. Evicted experts are parked in a 24-slot VRAM ring. A copy engine drains them to RAM off the critical path (~7 GB/s of write-backs at C1).

Where the time goes now (v4.6, 55 GiB, C1, 55.8 ms per token): copying admitted experts to VRAM 31 %, non-MoE GPU work 20 % (KDA/DSA attention GEMVs at 50-57 % of VRAM bandwidth), the MoE kernel 18 %, waiting for the CPU lane 18 %, NVMe waits 10 %, host 2 % (the decode lookahead removed the step-boundary gaps). No resource is saturated: the GPU does real work 70 % of the time, the CPU lane 60 %, PCIe and NVMe run at under 40 % of their ceilings, because every layer waits on its slowest lane. Better scheduling alone would reach about 20 tok/s; every lane at its hardware floor about 41. Profile: moetier docs/latency-glm53-v46.md. Details: docs/how-it-works.md.

Point your coding agent (Claude Code, Codex, others) at skills/glm53-offload-setup/SKILL.md. It checks your GPU, RAM, CPU and NVMe, picks the mode, builds the store, measures, and tunes one setting at a time.

• docs/results.md: every measured table, per mode, with dates and raw files
• docs/how-it-works.md: the mechanisms, how each tier is sized, and every setting
• docs/b70-tier.md: optional Intel Arc Pro B70 as a second expert tier (GLM53_B70=1)
• docs/troubleshooting.md: error messages and fixes
• docs/reference.md: the full campaign record (image smokes, quality method, reproduction pins)

Credits: exllamav3 and the EXL3 checkpoint by turboderp; FreeToken (FlashML) for the host-memory expert tier idea; SGLang as the base image; GLM-5.3-Flash by Z.ai. MIT license (this repository); the weights carry their own license.