What problem are you facing?
#65 planned three serving paths: native Kubernetes (single-pod), llm-d (multi-pod), and NVIDIA Dynamo (NVIDIA-optimized multi-pod). #99 landed native and llm-d but left Dynamo a dormant stub. A ModelReplica can't dispatch to Dynamo today.
Implementing the backend needs a concrete rule for when Modelplane picks Dynamo over llm-d. The two projects overlap on a lot, so the rule should rest on where they actually differ. From their own docs:
Both are orchestration layers over the same engines (vLLM, SGLang, TensorRT-LLM). Both do disaggregated prefill/decode, KV-/prefix-aware routing, and SLO-aware autoscaling, and both can run behind a Gateway API Inference Extension (GAIE) gateway. They also share plumbing: llm-d's disaggregation transfers KV over NIXL, which is an NVIDIA/Dynamo-org library, and llm-d lists Dynamo's KVBM as a compatible out-of-tree KV connector.
Where Dynamo has capabilities llm-d doesn't ship today:
- Grove (ai-dynamo/grove) — a Kubernetes scheduler for hierarchical gang scheduling, topology-aware placement within NVLink domains, and explicit startup ordering, with the pod group (not the pod) as the scaling unit. llm-d's multi-pod path is built on LeaderWorkerSet and has no equivalent gang scheduler or topology awareness.
- ModelExpress (ai-dynamo/modelexpress) — GPU-to-GPU weight streaming over NIXL/RDMA, so a new replica receives weights peer-to-peer from a seed GPU instead of loading from storage. llm-d has no equivalent; replicas load weights from storage on startup.
- KVBM (docs) — a built, unified GPU→CPU→SSD→remote-storage tiered KV hierarchy. llm-d's own native hierarchical offload is "under active development," with its CPU and storage tiers currently operating as independent options rather than one hierarchy (llm-d KV offloading).
These capabilities target NVLink/NVL72 rack-scale serving. Most of Dynamo's published throughput numbers are measured on GB200/GB300 NVL72 hardware. Outside that regime, llm-d and Dynamo cover similar ground.
How could Modelplane help solve your problem?
Add a Dynamo backend to compose-model-replica alongside native and llm-d. The open question is how Modelplane decides when to use Dynamo over llm-d, given the overlap above. We don't have a good answer yet.
What problem are you facing?
#65 planned three serving paths: native Kubernetes (single-pod), llm-d (multi-pod), and NVIDIA Dynamo (NVIDIA-optimized multi-pod). #99 landed native and llm-d but left Dynamo a dormant stub. A ModelReplica can't dispatch to Dynamo today.
Implementing the backend needs a concrete rule for when Modelplane picks Dynamo over llm-d. The two projects overlap on a lot, so the rule should rest on where they actually differ. From their own docs:
Both are orchestration layers over the same engines (vLLM, SGLang, TensorRT-LLM). Both do disaggregated prefill/decode, KV-/prefix-aware routing, and SLO-aware autoscaling, and both can run behind a Gateway API Inference Extension (GAIE) gateway. They also share plumbing: llm-d's disaggregation transfers KV over NIXL, which is an NVIDIA/Dynamo-org library, and llm-d lists Dynamo's KVBM as a compatible out-of-tree KV connector.
Where Dynamo has capabilities llm-d doesn't ship today:
These capabilities target NVLink/NVL72 rack-scale serving. Most of Dynamo's published throughput numbers are measured on GB200/GB300 NVL72 hardware. Outside that regime, llm-d and Dynamo cover similar ground.
How could Modelplane help solve your problem?
Add a Dynamo backend to
compose-model-replicaalongside native and llm-d. The open question is how Modelplane decides when to use Dynamo over llm-d, given the overlap above. We don't have a good answer yet.