122B MoE served on five discarded PS5 APU boards (AMD BC-250) via RPC — with first gfx1013 shmem_staging numbers

[MelonUsk1337]

We run a small fintech research pipeline in Sweden, and instead of renting GPUs we built our inference tier out of e-waste: five AMD BC-250 boards (the "PS5 APU on a blade" crypto-mining cards, gfx1013, 16 GB unified memory, no matrix cores) on a 2.5GbE switch, orchestrated with llama.cpp's Vulkan backend + the new RPC pipeline parallelism from #18626 (b10727).

On price: we paid ~$100/board — less than 16 GB of RAM costs on its own. Add used NVMe drives and 2.5GbE USB3 adapters (each peripheral roughly what a whole board used to cost) and the full cluster still lands well under a single consumer GPU. Fair warning: the boards have been climbing — ~$200 on AliExpress now, and eBay prices have surged past reasonable. If you see one under $200, snatch it.

As far as we can tell from the published record, no BC-250 cluster has served anything bigger than ~35B before. Yesterday we measured, and today this serves our production workload:

Qwen3.5-122B-A10B (IQ3_S, 43 GiB) across 4 nodes + dirigent

	FA off (our old doctrine)	FA on + forced shmem_staging	delta
prefill (3.6k-token prompt)	96 t/s	162 t/s	+68%
generation	19.0 t/s	29.5 t/s	+55%

Context 28k, LLAMA_ARG_LOAD_MODE=dio, -np 1, prompt cache warm across runs. In production terms: the daily synthesis job (7 stock picks from a ~16k-token evidence prompt) that took 803 s on our previous build now finishes in ~440 s — a 45% cut, measured this morning. The shmem_staging part is the NVIDIA-gated FA path from #19625 forced on — single-node A/B numbers (gemma-4-26B MoE IQ3_S: +12% pp at d0, +34% at d4096, +50% at d8192) are in issue #25207. It has now served production overnight and all day with zero worker crashes.

Single-node, the patched scalar path does pp2048 537 t/s (d0) / 505 (d4096) / 473 (d8192) on a 26B MoE — the depth penalty is essentially gone.

Why this is fun, price-wise (community numbers, sources in the thread):

Machine	Price	Qwen3.5-122B tg
5x BC-250 (this)	~$500 in boards ($100/ea paid; ~$200 now)	29.5 t/s
Ryzen AI Max 395 / Strix Halo 128GB	~$2,000	28.5 t/s (FP4)
DGX Spark	$3,999	48–55 t/s (int4+MTP)
Mac Studio M3 Ultra 256GB	~$5,600	57 t/s (mxfp4)
Mac Mini M4 Pro 64GB	~$2,000	~2 t/s (does not really fit)

Smaller models fly
Smaller models fly on this thing too: the same gemma-4-26B MoE over RPC across two boards does pp 508 t/s / tg 73.5 t/s, and a Qwen3.6-35B-A3B MoE across three boards does pp 559 t/s / tg 68 t/s (IQ4_XS — which, fun fact, is now speed-parity with Q4_K_S on this build; the old −23% i-quant decode penalty we measured on b10405 is gone).

Honest caveats: our IQ3_S is a smaller/lower-bit quant than the FP4/int4 builds above; single-box machines pay no RPC network tax; and W/token still favors the Macs and the Spark — our win is capex and the e-waste angle, not efficiency. Numbers degrade with depth on every platform; ours are quoted at the depths.

Credit where due: the BC-250 community stack made this possible - akandr's board docs, TechMakesArt's gfx1013 fork, and the cyan-skillfish-governor project (which, funny enough, we wrote, nice to see a fork of it credited as the biggest single win in someone else's benchmark). Also worth a look: duggasco's 40-CU unlock - our next experiment, stacked on the shmem patch.

Next experiments: the 40-CU unlock (duggasco) stacked on the shmem patch, and a graph-split audit for the MoE archs over RPC.

Happy to share configs, systemd units, or run benchmarks on request.