Add optional expert bundle sidecar for contiguous MoE expert reads

[GiorgioOppo]

Summary

This proposal adds an optional .expertbundle sidecar for SSD-streamed MoE
models.

The GGUF file remains unchanged. The sidecar is a derived cache where each routed
expert's tensors are stored contiguously:

GGUF layout, simplified:

[gate_expert_0][gate_expert_1]...[gate_expert_N]
[up_expert_0]  [up_expert_1]  ...[up_expert_N]
[down_expert_0][down_expert_1]...[down_expert_N]

expert bundle layout:

[expert_0: gate up down]
[expert_1: gate up down]
...
[expert_N: gate up down]


When the router selects an expert and that expert is not already resident in the
expert cache, the engine can satisfy the miss with one large sequential read
instead of three scattered reads.
The model weights, quantization, and math do not change. This is only a disk
layout optimization for routed expert slabs.
Motivation
In SSD streaming mode, decode can become dominated by routed expert I/O. For
each selected expert, the engine needs the gate, up, and down projections. In
the GGUF layout those slabs are grouped by tensor kind, so loading one expert
requires multiple non-contiguous reads.
A sidecar organized per expert makes a cache miss look like:
pread(bundle_fd, dst, expert_total_bytes, expert_offset);
instead of:
pread(model_fd, gate_dst, gate_bytes, gate_offset);
pread(model_fd, up_dst,   up_bytes,   up_offset);
pread(model_fd, down_dst, down_bytes, down_offset);
This should improve the cold-miss path while preserving the existing GGUF format
and all current decode behavior.
Proposed behavior
The feature is opt-in, for example:
DS4_EXPERT_BUNDLE=1 ./ds4 model.gguf
On startup, if enabled, the engine looks for:
model.gguf.expertbundle
If the sidecar exists and matches the source GGUF, it is opened and used for
expert cache misses / expert gathers.

If the sidecar does not exist, the engine may build it once by reading the
routed expert tensors from the GGUF and writing them in per-expert order.

If there is not enough disk space, or sidecar creation fails, the engine falls
back to the current GGUF gather path.

Numerics must be identical, because the sidecar contains the same expert bytes,
only reordered.

Sidecar format sketch
A minimal header could be:
struct ds4_expert_bundle_header {
    char magic[16];           // "DS4EXPBUNDLE"
    uint32_t version;
    uint32_t n_layers;
    uint32_t n_experts;
    uint32_t n_parts;         // usually 3: gate, up, down
    uint64_t source_size;
    uint64_t source_mtime;
    uint8_t source_hash[32];  // optional but preferred
};
Then a table of entries:
struct ds4_expert_bundle_entry {
    uint32_t layer;
    uint32_t expert;
    uint64_t offset;
    uint64_t gate_bytes;
    uint64_t up_bytes;
    uint64_t down_bytes;
};
The data region stores:
layer 0 expert 0: gate bytes, up bytes, down bytes
layer 0 expert 1: gate bytes, up bytes, down bytes
...
layer L expert E: gate bytes, up bytes, down bytes
The exact metadata can be adjusted to match ds4's current tensor descriptors and
mixed-quantization handling. The important invariant is that one expert maps to
one contiguous byte range.
Validation
The sidecar should be ignored or rebuilt if:
source model size differs;
source model timestamp differs;
source hash / GGUF fingerprint differs;
layer count or expert count differs;
tensor byte sizes do not match the current model metadata;
sidecar version is unsupported.
The safe fallback is always the existing GGUF path.
Expected benefits
This does not reduce the number of expert bytes read. It reduces the number and
randomness of reads on cache misses.
Expected improvements are workload- and disk-dependent, but the optimization is
most useful when:
SSD streaming is active;
expert cache hit-rate is not high enough to hide misses;
the model is too large for RAM;
decode is gather/I/O-bound;
the platform has high sequential NVMe bandwidth but scattered reads are costly.
Tradeoffs
Requires extra disk space, potentially tens of GB for large MoE models.
First run may spend time building the sidecar.
Needs invalidation logic to avoid stale sidecars.
Does not help if expert cache hit-rate is already very high.
Does not change quality or model output.
Why a sidecar instead of changing GGUF?
Changing GGUF layout would make this a model distribution problem. A sidecar keeps
the optimization local and optional:
existing GGUFs continue to work;
no conversion is required for users who do not need it;
users can delete the sidecar at any time;
the engine can fall back automatically.
Implementation notes
The change can be kept narrow:
Add optional sidecar open/build code during model load.
Add a lookup table from (layer, expert) to sidecar offset/size.
In the expert gather/cache-miss path, if the sidecar is available, read the
contiguous expert block.
Otherwise use the existing GGUF tensor-offset path.
Add diagnostic logging reporting whether the bundle is used, built, skipped,
or invalidated.
Example log lines:
expert bundle: found model.gguf.expertbundle, using contiguous expert reads
expert bundle: missing, building sidecar...
expert bundle: skipped, not enough free disk space
expert bundle: invalid source fingerprint, falling back to GGUF
Correctness
This should be a performance-only change. The output must remain identical to the
current GGUF gather path when the same bytes are copied into the same expert
buffers.
Suggested tests:
create a tiny synthetic GGUF-like expert layout;
build a bundle;
verify every (layer, expert) gate/up/down byte range matches the original;
run decode with and without the bundle and compare logits on a fixed prompt;
verify fallback behavior when the sidecar is missing or invalid.

[GiorgioOppo]

Successfully tested in my Swift project; 30% improvement in token generation on a 16GB MacBook M1 Pro.
https://github.com/GiorgioOppo/ds4.git

[DanteCpp]

Hi @GiorgioOppo,
I am interested in the idea but unfortunately I was not able to reliably test your claim with your implementation. I think there might be some error in the computational graph of your implementation since when I prompt "Hi!" it responds in chinese.

I was thinking myself about implementing something along the lines you are proposing, however I have taken some measurements under memory pressure and I have realized that it might not be worth it. The following figure has been obtained running 8 prompts 3 times each for every expert cache budget datapoint.

Each run is and independent ds4 new process, and the os memory gets purged between runs. The execution is also randomized among cache budget, prompt and repetition, e.g. (prompt 3, cache budget 9, run 1 -> prompt 2, cache budget 7, run 1 -> etc). The measurement has been taken in a Macbook pro max M2 with 32GB of Unified Memory.

The pread throughput is calculated dividing the total bites read by the time it took to read them. You can see the pread throughput is higher than the SSD bus bandwidth for small values of expert cache budget. My take is that for small values of cache the os can very efficiently paginate the rest of the expert which do not fit in ram, providing an effective read speed higher then the hardware limit.

This thesis is also supported by what happens at the limit when the memory pressure is very high. Around the 15GB cache budget mark the pread throughput approaches the hardware limit because the OS does not have room to keep hot pages in memory. At 16GB of expert cache budget, the os starts to swap out part of the unrouted weights (attention, embeddings, routers, etc...) that results in a drop in generation throughput, while the pread gains a bit of room and start increasing again.

Bottom line, my intuition tells me that there is not much to gain reorganizing the gguf, the best we can do is to give some room to the OS VM management system to do its job. But I might be wrong here...

[GiorgioOppo]

Hi, the response in Chinese depends on the temperature. If you set it to 0, it will respond in en in my implementation, whereas it will respond in Chinese with a temperature of 0.6—at least based on the tests I’ve just run.

I noticed that reorganizing the GGUF file takes slightly less time to retrieve the weights; this marginally reduces latency and allows for faster generation.

[DanteCpp]

I doubt that a temp of .6 should generate Chinese when prompted with "Hi", Chinese characters should be very unlikely, on top of that with vanilla ds4 I do not get Chinese with same temp settings.

I drive a MacBook pro max M2 with 32 GB of RAM.

Dante

[GiorgioOppo]

Found the bug and fixed the temperature.
tanks

[GiorgioOppo]

@DanteCpp Can I ask you to try my fork of ds4 on your Mac to see if the generation and decoding speed scales well with more memory?

[giannisanni]

Some measured data from consumer hardware that supports this proposal (RTX 4060 Ti 16GB, Flash q2, CUDA streaming, details in #495):

• Moving the same 81GB GGUF from a 0.53GB/s SATA drive to a 1.5GB/s PCIe 3 NVMe improved generation 5-6x, not the 2.8x the bandwidth ratio predicts. The extra gain is exactly what you would expect if scattered per-tensor reads are the bottleneck rather than raw bandwidth, which is the problem this sidecar attacks.
• The current selected-expert path issues gate/up/down as 3 separate read+upload+sync round trips per expert (cuda_model_copy_to_device_streamed callers around ds4_cuda.cu:3219). A contiguous bundle would collapse that to one sequential read per expert with zero kernel changes.
• We prototyped a thread-pooled parallel fetch for the same hot path: flat on the bandwidth-saturated drive for generation, +17% on prefill. Bundling and parallel fetch should compose, and both matter more the faster the drive (we will have Gen5 numbers next week).

Happy to benchmark a sidecar prototype on this rig if that helps, it seems like the cheapest remaining win for disk-bound streaming.

[DanteCpp]

@GiorgioOppo as I tryied to illustrate in my previous comment, I do not think there is much to gain bundling the tensors, at least on apple silicon.

Anyway, I might be wrong and I'll be happy to test your feature if you implement it in isolation, as a branch from vanilla ds4. In your implementation there is something fishy going on, and it wouldn't be a reliableand fair benchmark against the baseline.

Best,
Dante

[GiorgioOppo]

@DanteCpp
Feel free to give it a try; unfortunately, I'm not very good at C, but Claude helped out.
I created a dedicated 'Sidecar' branch for it.

https://github.com/GiorgioOppo/ds4

use
DS4_EXPERT_BUNDLE=1

[gilbert-barajas]

Some measurements that speak to the contiguous-read premise here — might save
effort, or at least scope where the sidecar pays off.

Setup: GLM-5.2 (IQ2_XXS, ~211GB) under --ssd-streaming on a single M5 Max
(128GB, Apple NVMe). Spent a couple days trying to push single-stream decode
past ~2.4 t/s, and this sidecar was one of the first things I wanted — so I
measured whether contiguity actually buys throughput before building it.

Random vs sequential straight to the disk (F_NOCACHE to bypass the page
cache, reads at one-expert granularity ~9.3 MiB, on the real 211GB file):

read pattern       seq     rand-1thr   rand-9thr
expert ~9.3 MiB   11.0     11.1        14.1 GB/s   (penalty 0.78x)
subtensor ~3 MiB  12.1      8.2        14.5 GB/s   (penalty 0.83x)

At the 9-thread depth the pread pool already runs, random reads are as fast as
sequential — actually faster, because the NVMe queue depth saturates the
controller regardless of layout. So on this hardware, laying experts out
contiguously doesn't add throughput; a scattered 9.3 MiB read is already a
full-bandwidth read. Caveat: this is Apple-Silicon NVMe at high queue depth.
The 1-thread column shows the random penalty is real there, so on a
higher-latency drive or a single-threaded path the sidecar could still win.

Why it matters for the wall: at max throughput, 5.84 GB streamed/token ÷
14 GB/s ≈ 0.41s = 2.4 t/s, which is exactly what we measure. It's
bandwidth-bound and the fetch is already optimal — so the win has to be in
fetching fewer bytes, not fetching them faster.

A few more numbers from chasing that, in case they save someone the dig:

• Expert-cache sizing has a memory-pressure cliff. Pushing the streaming
  cache toward ~90GB on a 128GB box tanked decode to 0.84 t/s — the OS
  compressor starts fighting the streaming preads. Sweet spot was ~46GB;
  bigger is worse past ~half of RAM.
• Token-to-token expert reuse is low. Recorded the routed top-8 selections
  across a decode run: consecutive tokens reuse only ~30% of their experts
  (vs ~3% at chance, = 8/256). The routing genuinely churns, so caching and
  predict-ahead/dedup have less headroom than you'd hope — dedup across a
  token burst caps around 1.5x.
• Routing weight is flat. The 8 gate weights aren't peaked — top-4 of 8
  carries only ~67% — so dropping to top-k<8 to stream less isn't near-free.

Net on a single M5 Max: GLM-5.2 IQ2 is ~2.4 t/s and honestly bandwidth-bound.
The levers that actually move it look like more physical bandwidth (NVMe
striping) or a lighter model, not layout or caching.

Repro: the bench is a ~30-line F_NOCACHE pread loop (seq vs random, 1 vs 9
threads) on the model file; the reuse/weight numbers are small trace dumps of
the routed selections + gate weights. Happy to share the scripts or run more
if any of it's useful. Thanks for ds4 — a joy to poke at at this level.

[DanteCpp]

Hi @GiorgioOppo,

the expert bundling does produce an appreciable effect, in my setup I was able to measure a roughly 13% generation average speed-up, from 6.25 t/s to 7.25% which is not bad.

What I do not fully understand is the mechanism, the pread throughput seems to be worst on the average, and the same at best. So either I am not correctly measuring the pread throughput or there is something else going on...

Best,
Dante

[GiorgioOppo]

@DanteCpp
If you find the bug in the measurement, let me know; I get roughly the same gain with my 16GB M1 Pro, too.