Qwen3.6-35B-A3B on two 8 GB cards: 5 coding agents by sizing slots from the attention layers only (and a FIM warm-up)

[RGlabs84]

Sharing something built on llama-server and ggml-rpc-server, in case the numbers help anyone running a hybrid model on small cards.

Setup. llama.cpp b11838, Qwen3.6-35B-A3B UD-IQ3_S, two 8 GB AMD BC-160s on Vulkan:

-c 163840 -np 5 -ngl 99 --n-cpu-moe 5 -ts 23,17 -fa on -ctk q8_0 -ctv q8_0 --cache-reuse 256 --ctx-checkpoints 4 --cache-ram 2048

Slot sizing for a hybrid model. Qwen3.6 keeps attention KV in only 10 of its 40 layers, so at q8_0 a token costs about 10,880 bytes, plus about 62.8 MiB of recurrent state per slot. Sizing slots from that, instead of from all 40 layers, took the same two cards from 2 slots to 5 at 32K each. Six slots fit too, but left the first card with under 500 MiB free, so I stopped at five. Measured: about 38.5 tok/s for one user and about 56 tok/s in total with five requests at once.

FIM latency. Sending the text before the cursor as its own request (max_tokens: 0) just ahead of each fill-in-the-middle request keeps the cache warm for the hybrid model. Median ghost-text latency went from 1.57 s to 1.0 s, with the same completion quality on a 36-case suite.

What it's for. DLNE is a free coding agent for VS Code-family editors that runs on GPUs its users lend. GPUs on the coordinator's own LAN join a sharded model over RPC. Since this week, a volunteer's GPU farther away (8 GB or more) can instead run the whole model by itself behind a small proxy and serve two agents, with requests sealed end-to-end in the editor (X25519, ChaCha20-Poly1305).

MIT licensed: https://github.com/RGlabs84/DistributedLLMEngine

Thank you for llama.cpp; none of this exists without it. Happy to share more numbers, and very open to a better split for two 8 GB cards.