Model load from the EFS-backed ModelCache is slow

[negz]

What problem are you facing?

Serving Qwen3-Coder-480B (894 GiB on disk, BF16) as a two-node Leader+Worker gang on EKS, vLLM spent ~43 minutes just reading the safetensors weights from the ModelCache RWX volume at startup — before CUDA graph capture and KV cache init even began. The cache itself hydrated fine (HuggingFace → EFS, ~960 GB in ~31 min, ~31 GiB/min); the bottleneck is the per-pod read of weights when the engine loads.

The per-shard load rate sawtooths: bursts of 1–2.5 it/s when a shard is already in the OS page cache (vLLM prefetches ahead with 8 threads), collapsing to 15–30 s/it when the loader catches the prefetch frontier and reads cold from EFS. Net sustained ~2–4 %/min across 241 shards. This read cost is paid on every pod start, so it dominates startup and can make the cache a net latency regression versus a direct pull for a single deployment.

Infra (us-east-2):

• 2× p5en.48xlarge (16× H200), AZ us-east-2b, EKS 1.36, AL2023 NVIDIA AMI.
• ModelCache RWX backing: Amazon EFS, throughputMode elastic, performanceMode generalPurpose, encrypted, ~960 GB. EFS CSI StorageClass modelplane-rwx-efs (provisioningMode=efs-ap). Same-AZ mount target (no cross-AZ).
• 16 vLLM workers (8 TP ranks/node × 2 PP nodes) read shards concurrently from the one filesystem.

EFS elastic throughput doesn't sustain the multi-GB/s a 16-GPU cold load wants; the page cache (8 prefetch threads into ~1.9 TiB RAM) hides latency but not the cold reads. EFS write during hydration was fine — it's the concurrent read fan-out that bogs down.

How could Modelplane help solve your problem?

The ModelCache's EFS-backed storage tier looks like the practical floor for cold-loading a ~1 TB model. Worth exploring faster options, but no specific proposal yet.

Related to #186 / #115 (cache footprint / hydration image).

[negz]

I looked into this and I think we're blaming the wrong layer. The EFS-backed cache wasn't the bottleneck on the #193 run — the vLLM weight loader was. Before we go build a faster storage tier, we should try a faster loader, because it's free and the dependency is already in the image.

I pulled the CloudWatch metrics for the run's filesystem (fs-0700573a53d10a0f3) over the weight-load window. EFS granted a flat 5,120 MiB/s the whole time (PermittedThroughput), but the loader only sustained about 906 MiB/s reading cold — roughly 18% of what was on offer. Averaged across the full 45.5-minute load it was worse, about 335 MiB/s, because of stalls and ramp. PercentIOLimit peaked around 3% and sat near 0.1%, so we weren't IOPS-limited either. EFS was sitting mostly idle while vLLM stalled at the cold read frontier — the same 15–30 s/shard sawtooth described in the issue. That makes sense: the default safetensors loader mmaps the shards and faults pages in on demand, and over NFS every cold fault is a network round trip that eight prefetch threads can't keep ahead of.

Worth noting this also rules out the EFS knobs. Provisioned throughput would actually lower the per-client read ceiling below the 5 GiB/s we already get and don't use, and maxIO is pointless at 0.1% IOPS (and isn't allowed on Elastic anyway). The fix has to be on the client side: read the existing headroom harder.

That's what --load-format runai_streamer does. Run:ai Model Streamer reads tensors with a pool of concurrent threads into a CPU buffer and streams them to the GPU in parallel, so it keeps many reads in flight and hides the NFS latency instead of paying it per fault. Crucially it reads the same safetensors files we already hydrate — no change to the PVC, the hydration job, the /mnt/models mount, or the RWX access mode.

I pulled both engine images from #193 and checked what's actually installed:

• vllm/vllm-openai:v0.23.0 — runai_streamer and fastsafetensors are both valid load formats, and both libraries ship pre-installed (runai-model-streamer 0.16.0, fastsafetensors 0.3.2). Nothing to add.
• lmsysorg/sglang:v0.5.10.post1-runtime — runai_streamer works out of the box (runai-model-streamer 0.15.8 is installed). fastsafetensors is a valid CLI choice but the package isn't installed, so that one would need a custom image. SGLang also only auto-picks runai_streamer for runai:// URIs, so for a local mount you have to pass the flag explicitly.

vLLM (the turnkey path already injects --model=/mnt/models, we'd add the loader flags):

args:
  - --model=/mnt/models
  - --load-format=runai_streamer
  - --model-loader-extra-config={"concurrency":16,"distributed":true}

SGLang (set it alongside the existing --model-path):

args:
  - --model-path=/mnt/models
  - --load-format=runai_streamer

On the speedup: I don't want to oversell it, but the envelope is wide. The same EFS already hit 2,735 MiB/s on warm reads during this run, so ~8x is physically on the table; saturating the full ceiling would be ~15x. Realistically I'd expect the streamer to pull the cold rate from ~900 MiB/s up toward 2–3.5 GiB/s, so call it 2–4x — 45 minutes down to somewhere around 12–20 — with the distributed flag likely pushing higher since a 16-rank gang reading one network filesystem is exactly its target case. Run:ai's own benchmarks claim 3–6x on network/S3 fileshares, which brackets that.

So my suggestion is: try runai_streamer on the vLLM run first and time the weight load against the 45.5-minute baseline, then add {"distributed":true,"concurrency":16} and compare. FSx for Lustre only becomes worth the build if the loader still can't saturate the mount — that'd be the signal the filesystem latency itself is the wall, rather than how we're reading it.

[negz]

Just tried the run_ai loader - it was a little over 4x faster. I was loading a quantized Kimi but I think it'd have bought our full precision Qwen 3 Coder load time down from ~45 mins to ~10 mins.

[negz]

we'd add the loader flags

^ I didn't have time to fully de-slop the above AI-generated comment. I think we should probably move away from injecting flags, not toward it. So I'd suggest we start by solving this with documentation.