You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Modelplane is a fleet-level control plane. Many of the features built by cluster-level orchestrators — routing, scheduling, caching, autoscaling — Modelplane does at the fleet level, i.e across clusters. What workload clusters are responsible for is serving: single-pod deployments, and multi-pod deployments with advanced topologies (LeaderWorker, disaggregated prefill/decode).
This issue proposes that Modelplane drop its KServe dependency and compose directly to llm-d (default for multi-pod cases) and NVIDIA Dynamo (for NVIDIA-optimized multi-pod with distinctive features). The rest — routing, scheduling, caching, autoscaling — happens at the Modelplane fleet layer.
This is primarily about getting the layering right. Feedback welcome before we land the composition logic in v0.1.
What Modelplane does at the fleet level
Modelplane's primary concerns are above the cluster:
Scheduling. Picks (cluster, pool) per replica based on declared capabilities and capacity across the fleet. Replicas of the same deployment can land on different clusters depending on placement policy.
Routing. A fleet-aware gateway (Envoy-based) routes requests across replicas. Replicas can span clusters; the gateway watches ModelEndpoint health and routes accordingly. ModelService rules compile to gateway configuration.
Caching. model weight caching should be a fleet-aware primitive for staging weights to clusters and pools.
Autoscaling. KEDA on the scale subresource of ModelDeployment, fleet-wide. Each ModelReplica is a fixed-topology serving instance; scaling adds or removes whole instances.
These are above the cluster. Within a cluster, Modelplane only needs the workload to serve.
What workload clusters need to do
Workload clusters need to run inference. Two cases:
Single-pod. One pod, possibly with multiple GPUs, running the engine container. This is just Kubernetes — Deployment + Service. No specialized orchestrator needed.
Multi-pod with advanced topologies. Multiple coordinated pods forming one logical inference unit: tensor or pipeline parallelism across nodes, disaggregated prefill/decode with KV cache transfer, expert parallelism. This needs a multi-pod orchestrator.
Where KServe fits — and why we don't need it
KServe is a comprehensive ML serving platform. For LLMs specifically, it provides LLMInferenceService (introduced in v0.16), which is built on llm-d's architecture — the substantive multi-pod orchestration work happens in llm-d.
Looking at what KServe LLMInferenceService brings beyond llm-d:
Lifecycle management (Modelplane handles at fleet layer)
Autoscaling hooks, Knative integration (Modelplane uses KEDA at fleet layer)
The LLMInferenceService CRD as a stable user-facing API (Modelplane composes; users don't write LLMInferenceService directly)
For Modelplane, all of these are either handled at the fleet layer or wrapped behind our own abstractions. The wrapper KServe provides on top of llm-d adds redundant layering — two reconciliation loops doing similar work, two lifecycle managers, two autoscaling integrations.
Where Dynamo fits
NVIDIA Dynamo is a multi-pod LLM orchestration project at roughly the same architectural layer as llm-d. Dynamo's distinctive features go beyond what llm-d offers:
KV Block Manager (multi-tier KV cache offload to CPU/storage)
ModelExpress (checkpoint restore, weight streaming over NVLink for fast replica startup)
NVL72 topology awareness via Grove
more
Dynamo's distributed runtime is NVIDIA-coupled in practice — the shipped runtime images are CUDA-targeted, and the distinctive features are tuned for NVIDIA hardware. Dynamo is the right backend for NVIDIA customers wanting these specific capabilities.
Proposed composition rules
Three paths, dispatched by topology and engine:
Topology
Composition
Single-pod (any GPU count, single node)
Native Kubernetes Deployment + Service
Multi-pod (LeaderWorker, Disaggregated)
llm-d by default. Dynamo when distinctive Dynamo features are needed and the engine is NVIDIA-compatible.
Modelplane infers the path from engine.name, engine.version, requested topology, and matched hardware capabilities. Users describe what they want to run; Modelplane picks the lightest composition path that satisfies it.
Modelplane is a fleet-level control plane. Many of the features built by cluster-level orchestrators — routing, scheduling, caching, autoscaling — Modelplane does at the fleet level, i.e across clusters. What workload clusters are responsible for is serving: single-pod deployments, and multi-pod deployments with advanced topologies (LeaderWorker, disaggregated prefill/decode).
This issue proposes that Modelplane drop its KServe dependency and compose directly to llm-d (default for multi-pod cases) and NVIDIA Dynamo (for NVIDIA-optimized multi-pod with distinctive features). The rest — routing, scheduling, caching, autoscaling — happens at the Modelplane fleet layer.
This is primarily about getting the layering right. Feedback welcome before we land the composition logic in v0.1.
What Modelplane does at the fleet level
Modelplane's primary concerns are above the cluster:
Scheduling. Picks
(cluster, pool)per replica based on declared capabilities and capacity across the fleet. Replicas of the same deployment can land on different clusters depending on placement policy.Routing. A fleet-aware gateway (Envoy-based) routes requests across replicas. Replicas can span clusters; the gateway watches ModelEndpoint health and routes accordingly. ModelService rules compile to gateway configuration.
Caching. model weight caching should be a fleet-aware primitive for staging weights to clusters and pools.
Autoscaling. KEDA on the scale subresource of ModelDeployment, fleet-wide. Each ModelReplica is a fixed-topology serving instance; scaling adds or removes whole instances.
These are above the cluster. Within a cluster, Modelplane only needs the workload to serve.
What workload clusters need to do
Workload clusters need to run inference. Two cases:
Single-pod. One pod, possibly with multiple GPUs, running the engine container. This is just Kubernetes — Deployment + Service. No specialized orchestrator needed.
Multi-pod with advanced topologies. Multiple coordinated pods forming one logical inference unit: tensor or pipeline parallelism across nodes, disaggregated prefill/decode with KV cache transfer, expert parallelism. This needs a multi-pod orchestrator.
Where KServe fits — and why we don't need it
KServe is a comprehensive ML serving platform. For LLMs specifically, it provides LLMInferenceService (introduced in v0.16), which is built on llm-d's architecture — the substantive multi-pod orchestration work happens in llm-d.
Looking at what KServe LLMInferenceService brings beyond llm-d:
For Modelplane, all of these are either handled at the fleet layer or wrapped behind our own abstractions. The wrapper KServe provides on top of llm-d adds redundant layering — two reconciliation loops doing similar work, two lifecycle managers, two autoscaling integrations.
Where Dynamo fits
NVIDIA Dynamo is a multi-pod LLM orchestration project at roughly the same architectural layer as llm-d. Dynamo's distinctive features go beyond what llm-d offers:
Dynamo's distributed runtime is NVIDIA-coupled in practice — the shipped runtime images are CUDA-targeted, and the distinctive features are tuned for NVIDIA hardware. Dynamo is the right backend for NVIDIA customers wanting these specific capabilities.
Proposed composition rules
Three paths, dispatched by topology and engine:
Modelplane infers the path from
engine.name,engine.version, requested topology, and matched hardware capabilities. Users describe what they want to run; Modelplane picks the lightest composition path that satisfies it.