Skip to content

Dynamo v1.4.0

Latest
Latest

Choose a tag to compare

View all tags
@dagil-nvidia dagil-nvidia released this 15 Aug 22:13
· 1 commit to release/1.4.0 since this release
v1.4.0
0301494
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Dynamo v1.4.0 - Release Notes

Dynamo v1.4.0 is the 17th feature release of the open-source distributed inference platform, spanning 640 merged PRs from 127 contributors. It takes Dynamo Router multi-datacenter with a sequenced KV relay, peer reservation replay, and endpoint-scoped event transport. It opens an experimental vLLM-compatible /inference/v1/generate token-in/token-out endpoint on the Dynamo Frontend alongside a default-on tokenizer prefix cache. vLLM-Omni pipelines gain NIXL RDMA disaggregation for true multi-node autoregressive-to-diffusion serving, the Spica discrete-event simulation framework joins the repository, and the engine fleet moves to vLLM v0.26.0, SGLang v0.5.16, TensorRT-LLM 1.3.0rc22, and NIXL v1.3.x. The release also ships a rebuilt documentation website with tab-based navigation and native Simplified Chinese localization.

Summary

Dynamo Router: Reservation Replay & Multi-DC Indexing

Dynamo Router adds experimental cross-datacenter prefix routing this release. A new in-process selection cache lets peers replay booking decisions instead of re-sending long prompts (#11416, #12335). A sequenced datacenter-scoped KV relay extends prefix routing across multiple datacenters (#11793), and event transport becomes endpoint-scoped so consumers aggregate explicit endpoints rather than inferring ownership from broadcasts (#11841). Cache-salt-aware KV routing isolates tenants: requests carrying a tenant namespace never reuse another tenant's KV cache while keeping prefix reuse within the tenant (#8197). Session affinity now synchronizes across Router replicas (#11750).

Dynamo Frontend: Generate Endpoint, Tokenizer Cache & Protocol 3.1

The Frontend registers an experimental vLLM-compatible /inference/v1/generate token-in/token-out endpoint (opt in with DYN_VLLM_ENABLE_INFERENCE_V1_GENERATE), letting vLLM and llm-d configured clients target Dynamo's token path without reconfiguration while preserving Dynamo-owned routing, tracing, and cache-salt metadata (#11108, #11167). The process-local L1 tokenizer prefix cache turns on by default with a 64 MiB budget and gains exact TikToken special-token support (#11078, #11499). Chat templates accept deployment-level thinking defaults so reasoning behavior is configured once per deployment instead of per request (#11691, #12440), and the dynamo-protocols wire crate ships at v5.0.1 for this release.

Multimodal & Omni: NIXL Disaggregation & Unified Serving

vLLM-Omni autoregressive-to-diffusion pipelines gain NIXL RDMA tensor transport, taking models like GLM-Image from shared-memory single-node serving to true multi-node disaggregation (#10071, #12984). The unified vLLM backend now serves image, video, and audio through one pipeline (#11267), with modality-aware routing so unified models such as Gemma 4 route each content type correctly (#10375). A pluggable custom vision encoder interface lets deployments swap in their own in-process encoders (#10910, #10832, #11847), and an experimental realtime worker streams transcripts and output audio (#10166).

Performance Modeling: Spica Simulation & Native Mocker G1

The experimental Spica discrete-event simulation framework moved into the Dynamo repository under aisimulate/, bringing capacity planning and what-if analysis of Dynamo deployments without GPUs into the main development loop (#11923). A new request_trace_to_satf CLI converts production request-trace logs into SATF 2.0 simulation artifacts, so recorded traffic drives that analysis directly (#11375). Dynamo Mocker defaults to a native G1 block manager that preserves KV event order (#12033, #12222) and gains streamlined SGLang radix scheduling (#12043), keeping simulated scheduling faithful to the engines it models. Constant-time block-pool bookkeeping raised the Mocker's median output-token throughput by 22.2% in a balanced A/B (#12244).

Kubernetes: Rollout Control, Diagnostics & Standalone EPP

DynamoGraphDeployment gained rollout control on Grove: spec.grove.updateStrategy selects RollingRecreate or OnDelete, with component-scoped Recreate via a pod-template annotation (#11085, #11909). The Ready condition now reports why a deployment is stuck (insufficient_capacity, pods_not_ready, updating), and placement-score fields landed in status (#11131, #11057). A convert_api_version.py utility migrates v1alpha1 manifests to the now-default v1beta1 API through the Operator's own conversion webhook (#10657). The GPU Memory Service added intra-pod shadow-engine failover counters with a Grafana dashboard (#11664) and a device-agnostic VMM layer (#9788), and XPU serving got ready-made DRA deployment YAMLs for aggregated and disaggregated vLLM (#9253). The inference-gateway EPP moved toward standalone operation, discovering workers directly from the GAIE InferencePool (#11074) and gaining a vLLM render client (#11827). Volcano scheduling landed for Grove-backed deployments (#8225), and the Operator no longer bundles CRDs in the Helm subchart (#11689).

Observability: Multimodal Metrics & Request Tracing

Prometheus histograms report images, videos, and audio parts per request (#11166) and calculated image-token counts (#12065). Inter-token-latency collection is buffered to cut its on-CPU cost by 71.9% while preserving the existing schema (#11569). Request payload auditing unifies into a single dynamo.request.trace.v1 stream configured by DYN_REQUEST_TRACE_* variables (#11180). The stream exports through a standard OpenTelemetry Collector (#9390) and captures operator-allowlisted HTTP headers so requests join with upstream gateway identifiers without leaking non-allowlisted headers (#11386).

Developer Experience & Docs

docs.nvidia.com/dynamo has been rebuilt around task-oriented tabs that take a deployment question straight to a working guide (#10855). Reference pages were corrected against shipped v1.4.0 behavior across the Router, KV routing, and Planner configuration (#11892, #12949, #12983, #11613), and release tags now publish documentation from the tagged commit itself (#11140). The versioned docs match the release you run. An interactive Get Dynamo selector produces the right install command for your backend (#11950), and Fern-native Simplified Chinese localization opens the same docs to zh-CN readers (#11161). For contributors, CODEOWNERS is generated from a single ownership definition, so pull requests route deterministically to the owning team (#10715, #11614).

Open-Source Contributions

Dynamo v1.4.0 includes contributions from 127 developers, 29 of whom made their first merged contribution in this release. Welcome to our first-time contributors:

Key Dependencies

Dynamo SGLang TensorRT-LLM vLLM NIXL UCX
v1.4.0 v0.5.16 v1.3.0rc22 v0.26.0 v1.3.2 (vLLM) / v1.3.1 (TensorRT-LLM) / v1.3.0 (SGLang) v1.21.0

CUDA Variants

Backend CUDA 12 CUDA 13
SGLang N/A 13.0
TensorRT-LLM N/A 13.1
vLLM N/A 13.0

Dynamo Ecosystem

AIConfigurator AIPerf Grove ModelExpress
v0.11.0 v0.10.0 v0.1.0-alpha.12-rc1 v0.5.0

Breaking Changes

Notable Behavioral Changes

Default Changes

  • Router queueing disabled by default (--router-queue-threshold default removed) (#11599).

    Migrate: Explicitly set --router-queue-threshold=16.0 (or DYN_ROUTER_QUEUE_THRESHOLD=16.0) to keep the previous queueing behavior.

  • DYN_SELF_HOST_METADATA now defaults to ON (#11417).

    Migrate: Set DYN_SYSTEM_PORT (e.g. 9090) on workers to enable self-hosted MDC serving, or set DYN_SELF_HOST_METADATA=0 to restore the previous shared-storage behavior.

  • Frontend no longer honors truncation config baked into tokenizer.json (#11792).

    Migrate: If you relied on tokenizer.json's baked-in truncation to shorten long prompts, truncate client-side or set an explicit context/length limit, since the Frontend now errors on over-length prompts instead of clipping them.

  • Session affinity now falls back to agent session headers when x-dynamo-session-id is absent (#11677).

    Migrate: Clients sending agent session headers (e.g. Claude Code/Codex/OpenCode) will now be pinned to a worker; omit those headers or set an explicit x-dynamo-session-id if the previous non-affinity routing is desired.

  • Multimodal load sites no longer force trust_remote_code=True (#10738).

    Migrate: Enable trust-remote-code on the engine (vLLM/SGLang --trust-remote-code, TensorRT-LLM extra_engine_args trust_remote_code: true, or --trust-remote-code for the trtllm mm_router_worker example) when serving multimodal models that require custom remote code.

  • Policy classes without a busy threshold no longer fall back to 16.0 (#11599).

    Migrate: Add an explicit numeric queue threshold to each policy class in the policy-config YAML to retain queueing.

  • Direct dispatch now bypasses local worker inhibition (#11993).

    Migrate: Remove the worker through service discovery if you depended on inhibited workers being skipped for direct-routed requests; direct dispatch does not consult the local inhibition set.

  • Decode workers now advertise local indexer support (#12084).

    Migrate: If decode workers should not advertise a local indexer, explicitly disable enable_local_indexer for those workers.

  • init_weights_update_group now bounded by 30s watchdog that terminates the worker (#11418).

    Migrate: If your RL weight-update rendezvous can take longer than 30 seconds, set DYN_RL_INIT_WEIGHTS_TIMEOUT_S to a larger value on the vLLM worker.

  • TensorRT-LLM video_url now enforced through shared SSRF/URL validation policy (#12002).

    Migrate: Set DYN_MM_ALLOW_INTERNAL=1 if you rely on internal/private/plaintext-HTTP video URLs with the TensorRT-LLM backend, otherwise switch to HTTPS public URLs.

  • Omitted/string thinking values now normalized; omitted thinking treated as enabled for kimi_k25 reasoning parsing (#11653).

    Migrate: If you relied on Kimi post-tool output arriving in content (including the </think> marker) or on string values like "false" being treated as truthy, set chat_template_kwargs.thinking explicitly and read reasoning from reasoning_content.

  • Tokenizer cache byte budget default increased (#11078).

    Migrate: Explicitly set DYN_TOKENIZER_CACHE_BYTES=52428800 to keep the previous 50 MiB budget.

  • EPP default RUST_LOG changed from debug/kv_router=trace to info (#11550).

    Migrate: Explicitly set RUST_LOG=debug,dynamo_llm::kv_router=trace on the EPP component if verbose KV-router trace logging is still required.

  • Monitoring scripts now target dynamo-system namespace by default instead of dynamo (#10771).

    Migrate: If your dynamo-platform release is installed in the 'dynamo' namespace, export DYNAMO_NAMESPACE=dynamo before running setup-monitoring.sh or cleanup-monitoring.sh.

  • arrivalIntervalMs default changed from 0.0 to None (#12062).

    Migrate: Explicitly set arrivalIntervalMs (or another arrival controller) instead of relying on the previous 0.0 default and its implicit 1 ms fallback.

API & Wire Format

  • Event-plane subjects/keys are now endpoint-scoped with no backward-compatible fallback (#11841).

    Migrate: Upgrade all workers, routers, and planner/consumer components together; mixed old/new versions will not see each other's events.

  • Embedding worker now returns pooled, L2-normalized vectors instead of per-token hidden states (#10248).

    Migrate: Update any client code that depended on the previous variable-length, un-normalized embedding payload to expect a single fixed-dimension normalized vector per input.

  • Audit subsystem migrated into request trace ("log_type":"audit" records and the "Audit sinks ready" readiness marker -> records carrying "schema":"dynamo.request.trace.v1" and the "Request trace sinks ready" marker). Payload logging is unified under the request-trace pipeline and the standalone audit module is gone (#9390, #11180).

    Migrate: Update log pipelines that match on the "Audit sinks ready" marker or the "log_type":"audit" field to the new marker and schema field. DYN_AUDIT_* variables remain legacy aliases of their request-trace replacements (DYN_AUDIT_SINKS -> DYN_REQUEST_TRACE_SINKS, DYN_AUDIT_FORCE_LOGGING -> DYN_REQUEST_TRACE_RECORDS=request_payload). Move configuration to the new names.

  • dimensions on /v1/embeddings now validated by vLLM pooler; requires Matryoshka-declared models (#10248).

    Migrate: Launch embedding workers for non-Matryoshka models (e.g. Qwen3-Embedding) with --hf-overrides '{"is_matryoshka": true}' if clients send the dimensions parameter.

  • max_tokens: 0 now rejected with HTTP 400 on /v1/chat/completions (#11394).

    Migrate: Send max_tokens >= 1 (or use max_completion_tokens) on /v1/chat/completions requests instead of 0.

  • Nested chat_template inside chat_template_args/kwargs now rejected with 400 (#11755).

    Migrate: Stop sending chat_template inside chat_template_args/chat_template_kwargs; configure the chat template server-side instead.

Helm & Charts

  • Bundled NATS subchart disabled by default (#11951).

    Migrate: Set --set global.nats.install=true (or configure dynamo-operator.natsAddr for external NATS) if your deployment relies on NATS transports, otherwise NATS resources and the NATS_SERVER env var are removed on upgrade.

  • Grove subchart/API dependency now requires v0.1.0-alpha.11 or later; v1.4.0 ships v0.1.0-alpha.12-rc1 (ClusterTopologyBinding API required) (#8225).

    Migrate: Upgrade externally-installed Grove to >= v0.1.0-alpha.11 in lockstep with Dynamo, since older Grove releases using the ClusterTopology API are no longer compatible.

  • Namespace-restricted installs must set dynamo-operator.upgradeCRD=false (#11689).

    Migrate: Set --set dynamo-operator.upgradeCRD=false for namespace-restricted installations and rely on the cluster-wide Operator for CRDs.

  • Snapshot chart init container now sets imagePullPolicy explicitly (default Always) (#10864).

    Migrate: If you override daemonset.initContainer.image with a pinned tag and rely on cached images, set daemonset.initContainer.imagePullPolicy: IfNotPresent in your values file.

  • Dev-pod script ConfigMap must now include managed_state.py key (#9790).

    Migrate: Add a managed_state.py key to the power-agent dev-pod script ConfigMap before upgrading to chart 1.3.0, per templates/dev-pod.yaml header.

Environment Variables

  • Tokenizer L1 prefix cache now enabled by default (#11078).

    Migrate: Set DYN_TOKENIZER_CACHE=0 to restore the previous disabled-by-default tokenizer cache behavior.

  • New required env var DYN_EPP_TOKENIZER_PROTOCOL for standalone EPP (#11827).

    Migrate: Set DYN_EPP_TOKENIZER_PROTOCOL=vllm-render in existing standalone EPP deployments, otherwise config parsing fails at startup.

  • New required env var DYN_EPP_TOKENIZER_SERVICE_URL for standalone EPP (#11827).

    Migrate: Set DYN_EPP_TOKENIZER_SERVICE_URL to the vLLM render endpoint base URL; validation rejects empty or non-absolute URLs.

  • HTTP header capture in trace records is an explicit fail-closed allowlist: request headers appear in request_payload records only when named in DYN_REQUEST_TRACE_HTTP_HEADER_CAPTURE_LIST; when the variable is unset or empty, no headers are captured (#11386).

    Migrate: Set DYN_REQUEST_TRACE_HTTP_HEADER_CAPTURE_LIST to the comma-separated header names you need recorded; captured values are unredacted, so avoid allowlisting credential-bearing headers.

  • DYN_HTTP_OVERLOAD_STATUS_CODE no longer accepts informational (1xx) status codes (#12738).

    Migrate: If DYN_HTTP_OVERLOAD_STATUS_CODE is set to a 1xx value, change it to a final status code in 200-999 (e.g. 503) or rely on the 529 default.

CLI Flags

  • Decode workers no longer suppress KV event publishing (#12084).

    Migrate: To keep KV event publishing off on decode workers, omit --kv-events-config (or disable prefix caching / set enable_kv_cache_events=false) instead of relying on decode mode.

  • --tag default changed from single weights tag to all production GMS tags (#11285).

    Migrate: Pass --tag weights explicitly to python -m gpu_memory_service if you rely on serving only the weights tag/socket.

  • --socket-path now requires exactly one --tag (#11285).

    Migrate: When using --socket-path, also pass a single --tag <name>.

  • Implicit 1 ms arrival-interval fallback removed from synthetic replay API/CLI (#12062).

    Migrate: Pass one of --request-rate, --arrival-interval-ms, or --replay-concurrency (or the equivalent Python kwarg) when running synthetic replay.

  • New --trust-remote-code flag (default off) in trtllm mm_router_worker example (#10738).

    Migrate: Pass --trust-remote-code to the trtllm mm_router_worker example when the model requires custom remote code.

Python API

  • Python binding router_queue_threshold default changed to None (#11599).

    Migrate: Pass router_queue_threshold=16.0 explicitly when constructing the router config to keep queueing enabled.

  • Missing media decoder now raises RuntimeError (MissingMediaDecoderError) instead of ValueError/ImportError (#12725).

    Migrate: Update callers/handlers that catch ValueError (or expect a 4xx client error) for undecodable media to also catch MissingMediaDecoderError/RuntimeError, which now signals a deployment/decoder-configuration failure.

  • load_vision_model / load_qwen_grid_params signatures gain trust_remote_code param with False default (#10738).

    Migrate: Callers of these multimodal utility functions that rely on remote code must now pass trust_remote_code=True explicitly.

  • Synthetic WorkloadSpec now requires exactly one arrival controller (#12062).

    Migrate: Update synthetic workload specs to set exactly one of concurrency, requestRate, or arrivalIntervalMs (remove redundant/duplicate arrival controllers).

Container Images

  • Chart 1.3.0 requires agent image v1.2.0 or newer (#9790).

    Migrate: Re-pin image.tag (or image.digest) to the v1.2.0 agent image; older images lack managed_state.py and CrashLoop with ModuleNotFoundError.

CRDs & Operator

  • Unresolved GMS extraClientContainers names now rejected by admission and reconcile (#12580).

    Migrate: Update DynamoGraphDeployment/DynamoComponentDeployment manifests so every experimental.gpuMemoryService.extraClientContainers entry matches a unique container name declared in podTemplate.spec.containers before upgrading the Operator.

Rust API

  • C API now requires an explicit endpoint name (#11841).

    Migrate: Update C API callers to pass the endpoint name explicitly instead of relying on the implicit serving endpoint.

Deprecated and Removed

  • Removed the deprecated vLLM worker role flags (--is-prefill-worker / --is-decode-worker -> --disaggregation-mode) (#12089).

    Migrate: Pass --disaggregation-mode prefill or --disaggregation-mode decode instead of the removed boolean flags.

  • Removed the deprecated multimodal worker flags (--multimodal-worker / --multimodal-encode-worker -> --enable-multimodal with --disaggregation-mode / --dedicated-mm-encoder) across the vLLM, SGLang, and TensorRT-LLM backends; deprecated in v1.3.0 (#11887).

    Migrate: On vLLM and SGLang, replace the removed flags with --enable-multimodal plus the appropriate --disaggregation-mode / --dedicated-mm-encoder combination; on TensorRT-LLM, use --modality multimodal, which remains the supported path.

  • Removed bundled software video decoders from the runtime images: H.264/H.265 video inputs decode in hardware via NVDEC (#11836). Other codecs (or H.264/H.265 where NVDEC is unavailable) fail with an actionable error naming the missing package and installer command instead of decoding in software (#12725).

    Migrate: Grant containers the video NVIDIA driver capability for H.264/H.265 input, or install software decoders with python -m dynamo.common.utils.install_media_decoders <vllm|sglang|trtllm>.

  • Legacy DGDR Compatibility Retirement: Stopped emitting legacy per-field nvidia.com/dgdr-* annotations during v1alpha1 to v1beta1 conversion, restoring v1alpha1-only profiling fields through typed API conversion instead (#11531). The remaining read-only annotation compatibility layer and its fixtures moved into dedicated files (#11598), and the legacy annotation readers were then removed, so conversion relies exclusively on the structural nvidia.com/dgdr-spec and nvidia.com/dgdr-status payloads (#11663). Reworked worker hashing to converge lazily to v2, computing the v1alpha1 hash only for incomplete migrations and avoiding an Operator-upgrade-induced rollout for unchanged Dynamo 1.2 worker generations (#11529).

    Migrate: Read DGDR state from the structural nvidia.com/dgdr-spec and nvidia.com/dgdr-status annotations. The per-field nvidia.com/dgdr-* annotations are no longer emitted.

  • Unified Backend Engine Removal: Removed the experimental per-backend unified backend engine implementations for vLLM, SGLang, and TensorRT-LLM in favor of the sidecar implementation (#11831). The --unified launch flag, the unified serve configs, and the engine-specific tests are gone, while the Python common/backend and Rust backend-common interfaces remain for the sidecar path. The removal was backported to release/1.4.0, preserving the release-specific vLLM and Fern documentation structure and deleting the stale unified-backend docs (#12862).

    Migrate: Launch each backend through its standard entry point (python -m dynamo.vllm, dynamo.sglang, or dynamo.trtllm). The --unified flag no longer exists.

  • Common backend interfaces are no longer a published integration surface: the Rust dynamo-backend-common crate is not published for v1.4.0 (last published as 1.2.1). The crate and the Python dynamo.common.backend module remain in-tree solely for the in-development sidecar path following the unified backend engine removal (#11831). Treat these interfaces as internal until the sidecar ships.
  • Go-based EPP deprecated in favor of the Rust-based EPP: the Rust ext-proc EPP introduced in v1.3.0 (#8783) replaces the Go Endpoint Picker under deploy/inference-gateway/epp (built on gateway-api-inference-extension), and the default switches in an upcoming release.
  • Namespace-restricted Operator mode deprecated: the mode will be removed in a future release, so run the cluster-wide Operator instead. Namespace-restricted installs already depend on the cluster-wide Operator for CRDs (dynamo-operator.upgradeCRD=false).

Features & Improvements

Frontend

Generate Endpoint

  • Experimental /inference/v1/generate Endpoint: Added a vLLM-compatible /inference/v1/generate token-in/token-out endpoint on the Dynamo Frontend (#11108), then dispatched real requests through it with Generate capability gating, tokenizer-free frontend registration, and sanitized cancellations and errors (#11167). The endpoint remains experimental and unary-only.

Tokenizer & Chat Templates

  • Tokenizer Prefix Cache Enablement: Enabled the process-local L1 tokenizer prefix cache by default unless DYN_TOKENIZER_CACHE is exactly 0, raising the default byte budget from 50 MiB to 64 MiB (#11078). Extended the cache to supported TikToken models by forwarding exact registered special-token boundaries (#11499).
  • Chat-Template Thinking Defaults: Added deployment-level thinking defaults for chat templates (#12440), and honored reasoning_effort for templates that gate reasoning through enable_thinking by deriving enable_thinking=false for reasoning_effort="none" and true for other supplied values, with explicit chat_template_args.enable_thinking values taking precedence and requests without reasoning_effort left unchanged (#11691).
  • Dynamo Protocols 3.1.0 Upgrade: Updated Dynamo to dynamo-protocols 3.1.0 with compatible parser, renderer, and tokenizer versions, propagated tokenizer-cache initialization failures instead of hiding them, and adapted Anthropic, OpenAI Responses, and media preprocessing to protocol 3.x optional media URL fields (#11978).

OpenAI API & Validation

  • Configurable Overload Rejection Status: Added DYN_HTTP_OVERLOAD_STATUS_CODE so operators can choose the HTTP status returned for Frontend overload and admission-control rejections, keeping 529 as the default and accepting 503 (#11249). Rejected informational status codes in the parser, falling back to 529 instead of surfacing HTTP 500 (#12738).

Request Plane & Extensions

  • Python Frontend Route Extensions: Added a dynamo.llm.FrontendRoute hook so external packages can register GET routes on the Frontend via --frontend-route-extension or DYN_FRONTEND_ROUTE_EXTENSIONS, without a custom binary. Routes run on a bounded pool and shed load with 503s, keeping handlers off the hot path (#11626).

General

  • Squeeze-Evolve Multi-Model Service: Added an experimental dynamo.squeeze_evolve component that serves verifier-free evolutionary test-time scaling as a chat model, routing candidate groups across model tiers via per-tier KvRouter instances. It is opt-in and not wired into any default frontend path (#10785).

Multimodal & Diffusion

  • vLLM-Omni Realtime Worker: Added a realtime bidirectional vLLM-Omni worker exposing a WebSocket /v1/realtime endpoint that streams transcript and output audio deltas (#10166), and fixed an AttributeError on newer vLLM-Omni versions when reading multimodal_output (#12667).
  • Unified vLLM Multimodal Pipeline: Added image, video, and audio handling to the aggregated unified vLLM backend (#11267) with modality-aware multimodal routing metadata (#10375). Enabled the vLLM multimodal CPU embedding cache (#11268), added frontend media discovery plus SHM and NIXL transfer fallback (#11269), and brought multimodal prefill and decode parity to the unified backend (#11270).
  • Pluggable Custom Vision Encoder: Added a VisionEncoderBackend ABC, the build_mixed_embeds assembler, and a --custom-encoder-class flag (#10910), then wired the contract into serving so an aggregated dynamo.vllm worker runs a user-supplied encoder in-process without a separate encode worker (#10832). Added adapters that build the final vLLM EmbedsPrompt or TokensPrompt directly (#11847).
  • Cached Multimodal UUID Passthrough: Added UuidOnly and multi_modal_uuids support, preserving user-provided image UUIDs and image-slot alignment through the vLLM worker (#11943), rejected non-null multimodal UUIDs on the SGLang and TensorRT-LLM paths (#11944), and parsed canonical and deprecated image UUID forms in the Rust and Python frontends (#11958).
  • Media Decoder Installer And Diagnostics: Added an operator-run installer for the media-decoder packages the runtime images omit, invoked as python -m dynamo.common.utils.install_media_decoders <vllm|sglang|trtllm> with --packages, --pip-args, --timeout-s, and --dry-run flags (#12051). Reworked decode errors to name the missing package, installer command, and hardware alternative (#12725).
  • NVDEC Hardware Video Decode: H.264/H.265 video inputs now decode on the GPU via NVDEC through PyNvVideoCodec, keeping decode work off the SMs beyond YUV-to-RGB conversion. DYN_DISABLE_NVDEC opts out and DYN_NVDEC_GPU_ID selects the decode GPU (#11836).
  • Encode-Worker Request Routing: Added a backend-agnostic encoder routing hop that sends multimodal requests to a surface-less Encode worker before aggregated or prefill execution, with an optional round-robin EncoderRouter (#11460). Forced the vLLM V1 model runner in the disaggregated EPD encode worker to work around a vLLM 0.25.1 startup crash (#11913).
  • Video Embedding Cache Benchmark: Added a video-understanding benchmark workload for SGLang multimodal embedding cache reuse, including video-single-turn and video-sliding-window JSONL generation and an E/PD sweep config comparing cache-off vs cache-on, letting users evaluate SGLang video cache performance (#10000).
  • NIXL Connector for vLLM-Omni: Added a NIXL RDMA-based tensor transport connector for vLLM-Omni, enabling autoregressive-to-diffusion (AR->DIT) pipelines such as GLM-Image to run disaggregated across multiple nodes and device types instead of requiring shared memory on a single machine (#10071).
  • Cross-Request Vision Batching: Added a ThreadedMicroBatcher and batcher-backed AsyncVisionEncoder that combine image forwards from concurrent requests into thread-affine, cost-bounded batches, improving GPU utilization for the custom encoder path. Canceled requests are tombstoned before dispatch so abandoned input cannot affect in-flight requests sharing a later batch (#11037).
  • TensorRT-LLM Video URL Support: Added video_url multimodal input handling to the TensorRT-LLM backend, decoding video URLs into VideoData instead of silently dropping them, mirroring the image_url path's validation and controlling frame count via DYN_MM_VIDEO_NUM_FRAMES (#11896).

Scheduling

Multi-DC Indexing & Relay

  • Radix Tree Memory Reduction: Replaced each positional radix bucket's inner DashMap with FxHashMap, since the outer DashMap already served as the concurrency boundary and inner sharding added no benefit. This reduced peak Mocker RSS by 5.51 GiB (35.3%) in a 32-worker A/B test (#12161).
  • CRTC Compressed Radix Tree Indexer: Replaced each CRTC node's eager DashMap with adaptive child storage, improving full-trace throughput from 9.04M to 13.49M block ops/s (#11459). Added a router-local Relay boundary for exact per-datacenter aggregation (#11435), renamed tenant_id to routing_group across APIs (#11471), closed stale-shape races (#11169), and skipped shape locks for already-internal nodes (#11102).
  • Arena-Backed Block Tracker Rework: Replaced per-block Arc ownership in the KV Router block tracker with a SlotMap-arena block chain (#11508), removed the PromptRegistry mutation lookup (#11515), and moved to compressed radix-tree edges (#11644). Streamlined BlockTracker::release (#11568), fixed a leak in ConcurrentRadixTreeCompressed (#11785), and preserved worker and DP-rank coverage on cold restores (#11190).
  • Missing KV Event Publisher Diagnostics: Added the optional kv_event_publishing_enabled runtime capability across backends with a warning-only source-health monitor and stable structured codes (#12095), plus a Router diagnostic that logs an actionable error when expected worker or DP query endpoints never appear on the local-indexer recovery path (#11194).
  • Endpoint-Scoped Relay Event Transport: Added endpoint-scoped EventPublisher/EventSubscriber constructors that thread namespace/component/endpoint identity through discovery and transport (#11841), added a domain-scoped KV Relay with a lane-sticky global indexer for multi-DC prefix routing (#11793), and fixed a startup TypeError in the relay's health endpoint (#12948).
  • Planner Predictor Data Inputs: Added a KV-cache-read dimension to the vLLM prefill self-benchmark grid, gated behind --benchmark-prefill-kv-read-granularity and DYN_BENCHMARK_PREFILL_KV_READ_GRANULARITY (#11150). Added native dynamo.request.trace.v1 support to Planner load-predictor warmup with schema auto-detection and rejection of malformed traces (#11842).
  • Pool-Scoped KV DC Relay: Added a pool-scoped KV DC Relay runtime mapping each serving endpoint to one Dynamo PoolId, with a PoolRegistry owning per-pool Relay actors and CKF state. This keeps independent endpoints serving the same model isolated and recovering from stream failures (#12199).

Selection & Reservation Replay

  • Active Sequence Replica Event Pipeline: Batched frontend active-sequence replica-sync lifecycle events on ZMQ with the shared 256-event and 1 ms policy (#11920), added a lifecycle-owned direct-ZMQ fan-in preserving per-source FIFO (#11967), and replaced the async publication API with a bounded synchronous enqueue that drops the newest event on overflow (#11992).
  • Standalone EPP Selector Mode: Added a standalone EPP mode selected by DYN_EPP_MODE with EppStandaloneConfig validation (#11070), embedded the dynamo-kv-router SelectionService with optional multi-EPP replication via DYN_EPP_PEER_SERVICE (#11541), wired an in-process EppRouter without DistributedRuntime, etcd, or NATS (#11542), and preserved token-id completion prompts in the EPP scorer request (#10872).
  • Selection Booking Replay By Id: Added a SelectionCache keyed by (model, routing_group, selection_id) so a follow-up POST /reservations replays the cached POST /select booking instead of re-sending and re-hashing a long prompt (#11416), with select_and_reserve returning the normalized booking fields so peer ingress replicas can replicate it (#12335).
  • Session Affinity Across Replicas: Propagated agent_context.session_id through live router selection and Mocker replay into SchedulingRequest (#11089), and synchronized session-affinity bindings automatically whenever the affinity TTL is configured, with first-live-binding-wins semantics (#11750). Moved replica tasks under the affinity coordinator and upgraded the end-to-end test to an adversarial two-frontend scenario (#11750).
  • Decode Affinity Scoring Path: Added decode affinity scoring to the Router's worker selection (#11720), with an experimental block-equivalent decode cost term plumbed through config, CLI, and bindings (#12158). Stamped request_id and worker_type on cost-breakdown log rows (#12370) and extended the same stamping to the decode-affinity branch (#12955).
  • Provider-Keyed Tracking Hashes: Added an opt-in keyed-xxh3-v1 identity mode for router tracking, backed by a zeroized 32-byte provider key and managed key ID, preserving hashes and wire formats. Standalone selection honors router_assume_kv_reuse=false with random tracking identities; rotation requires coordinated key changes and restarts (#11548).
  • ISL-Based Aggregate Routing: Added an optional input sequence length (ISL) dimension to aggregate GlobalRouter pool selection, allowing requests to route across pools based on token count in addition to existing TTFT x ITL configurations. Existing 2D TTFT x ITL routing remains unchanged when ISL settings are not configured (#11835).

KV Events & Source Health

  • Concurrent KV Event Ingestion: Delivered direct-ZMQ KV events through one FIFO task per publisher, bypassing the lossy broadcast hop (#11938). Preserved native SGLang and vLLM event-list boundaries (#11466), published ordered RouterEvent vectors in partitioned batches (#11776), batched deferred Mocker visibility boundaries (#11860), and restored KV-aware routing for multinode SGLang workers on Kubernetes (#12185).
  • Cache-Salt Multi-Tenant KV Isolation: Added cache-salt-aware KV routing so requests carrying a tenant namespace (via x-tenant-id, nvext.cache_salt, or a compatibility cache_salt field) hash and route independently across the Router, vLLM, and TensorRT-LLM KV events, preventing cross-tenant cache reuse while preserving existing unsalted behavior (#8197).
  • Forward-Pass Metrics Tracing: Added opt-in, best-effort persistence of forward-pass metrics to bounded rotating gzip JSONL traces for vLLM, SGLang, TensorRT-LLM, and Mocker publishers, exposed via --fpm-trace/--no-fpm-trace and DYN_FPM_TRACE, giving operators durable forward-pass records without adding disk I/O to the inference path (#11110).

Routing Policies

  • Router Queue Admission Strategy: Added exact-worker lanes to the policy-class queue (#11363) with deferred blocked-lane rechecks (#11484), a public PolicyClassAdmissionStrategy contract (#11434), and a live RequestProgress reader (#11615). Kept Router queueing off unless a threshold is configured (#11599) and replaced --admission-control with opt-in thresholds, ignoring the legacy flag and DYN_ADMISSION_CONTROL with a warning (#11276).
  • Conditional Disaggregation Routing Policy: Added a conditional disaggregation policy kernel to the Router (#11718) with backend handling for the conditional disagg bypass annotation (#11719). Deprecated --enforce-disagg and DYN_ENFORCE_DISAGG as accepted no-ops (#11160), emitting the deprecation warning immediately for both forms (#11800).
  • Configurable Active Request Expiry: Added the DYN_ROUTER_ACTIVE_REQUEST_EXPIRY_SECS environment variable to configure the KV-router's stale active-request cleanup guard, preventing long-context agentic requests from having their live record removed before completion when they exceed the previous hard-coded 300-second limit (#11303).
  • Native KV Offloading Capacity: Added a backend-neutral native_offloading_capacity.total_tokens metadata contract for Rust and Python routing consumers, accepting SGLang capacity only from the scheduler's realized host pool and translating it by write policy to avoid double counting GPU capacity in router budgets (#11321).
  • Amplified Overlap Score Credit: Allowed the KV Router's --router-kv-overlap-score-credit value to exceed 1.0 and removed the zero floor on adjusted prefill blocks, so amplified overlap credit can offset decode load and improve cache-hit routing without adding new configuration surface (#11714).
  • Native Multimodal KV Routing: Moved TensorRT-LLM aggregated multimodal KV routing onto Dynamo's Rust Frontend and KV Router, removing the separate Python MM-router sidecar and updating worker integration for TensorRT-LLM v1.3.0rc22. This simplifies deployment to a single native request path while preserving KV cache reuse across repeated image requests (#11737).

General

  • HiCache Host Capacity Budget: Updated ThunderAgent to derive its retention budget from GPU KV plus native HiCache host capacity instead of device KV alone, aligning pause/resume behavior with the memory SGLang can actually retain (#11185).

Performance Modeling & Replay

Spica Simulation

  • Experimental Spica Simulator Integration: Moved the experimental Spica simulator from aiconfigurator into Dynamo as an independent aisimulate distribution with import aisimulate.spica and CLI python -m aisimulate.spica (#11923), and stopped emitting synthetic arrival intervals for closed-loop workloads so Replay receives only replay_concurrency (#12824).
  • SATF Request-Trace Conversion: Added a Rust API and CLI (request_trace_to_satf) that converts Dynamo request-trace JSONL and gzip JSONL shards into SATF 2.0 performance-layer artifacts, preserving token counts, timing, KV block identity, request metrics, and tool-call structure for downstream simulation and analysis without altering the existing capture format (#11375).

Mocker

  • Native G1 Block Manager: Added a selectable native G1 KV manager for the shared vLLM and TensorRT-LLM mock schedulers (#12033), defaulted both backends to native G1 with fallback to KVBM when G2/G3/G4 offload is required (#12222), flattened native G1 request token representation (#12248), and fixed a build failure via resolved_g1_backend() (#12336).
  • SGLang Mocker Scheduling Fidelity: Streamlined SGLang radix scheduling in the Mocker by keeping each request in one append-only u32 token sequence so decode caching borrows the retained prefix (#12043), and modeled SGLang KV allocation by page instead of individual token-slot IDs (#12108).
  • Mocker KV Bookkeeping Overhead: Reduced Mocker KV-management overhead in high-concurrency disaggregated workloads by batching terminal dereference signals and using canonical immutable block handles (#11095). Replaced the native VllmBlockPool inactive-copy BTreeMap with an intrusive linked LRU for constant-time operations, with a balanced AgentX A/B measuring a 22.2% median output-token throughput improvement (#12244).
  • Mocker Max Model Length: Added explicit max_model_len support to the vLLM Mocker and replay so they match the Frontend, Router, and real worker sequence-length behavior. Requests exceeding the context limit are rejected up front, while mid-generation hits complete with finish_reason="length" (#11069).

Trace Replay & Benchmarks

  • Poisson Synthetic Replay Arrivals: Added Poisson synthetic replay driven by request rate, alongside fixed-interval and concurrency modes; an independent arrival seed means switching modes changes only timestamps. Replay now requires one of request rate, interval, or concurrency, removing the implicit 1 ms fallback (#12062).

Kubernetes Deployment

Inference Gateway (EPP)

  • Standalone EPP Worker Discovery: Added discovery of standalone EPP workers from the gateway-targeted GAIE InferencePool via a kube-rs reflector index of Ready, selector-matching workers (#11072), with a topology adapter that converts each ready worker into a WorkerRegistration and reconciles the worker set with the in-process Selector (#11074).

Operator & CRDs

  • Grove Update Strategy Controls: Added spec.grove.updateStrategy.type to DynamoGraphDeployment with RollingRecreate and OnDelete values, surfacing OnDelete rollout progress through a DGD condition and events (#11085), and added component-scoped Recreate support via the nvidia.com/deployment-strategy pod template annotation (#11909).
  • CRD apiVersion Converter Utility: Added deploy/utils/convert_api_version.py, a CLI that converts nvidia.com/v1alpha1 manifests to nvidia.com/v1beta1 by POSTing a ConversionReview to the Operator's existing conversion webhook (#10657). Fixed the power-agent Helm chart, where a whitespace trim broke the default helm install (#12651).
  • Grove Volcano Scheduler Support: Added Operator-controlled Volcano scheduler integration for Grove-backed DynamoGraphDeployment workflows, setting schedulerName: volcano on Grove clique pods and mapping the nvidia.com/volcano-queue annotation to Grove's PodCliqueSet metadata, with startup validation preventing simultaneous KAI-Scheduler and Volcano configuration (#8225).
  • DynamoGraphDeployment Placement Score Status: Added status.placementScore and status.placementScoreState fields to the DynamoGraphDeployment API, mirrored across v1alpha1 and v1beta1 with conversion wiring, and reported non-failing placement states during reconcile on the Grove pathway. Full score reporting awaits Grove exposing a consumable score field. (#11057).
  • Fine-Grained DGD Readiness Reasons: Added classification of the DynamoGraphDeployment Ready condition into specific reasons (insufficient_capacity, pods_not_ready, updating, mixed_not_ready_reasons) derived from Grove status fields, plus an optional scheduledReplicas field on component status, so operators can diagnose why a Grove-backed deployment is stuck without inspecting Grove resources directly (#11131).
  • Operator RuntimeVersionOverride Backport: Added runtimeVersionOverride support for DGD (DynamoGraphDeployment) components and DGDR-generated deployments, backported to release/1.4.0, letting users pin a specific runtime version for these components (#12427).

GPU Memory Service & Snapshot

  • GMS Shadow Failover Hardening: Added dynamo_component_engine_failover_* counters and state gauges for intra-pod shadow-engine failover with a Grafana dashboard (#11664), aliased all scratch KV allocations to a single granule of device memory (#11911), and surfaced scratch-KV engagement in shadow worker logs (#11724).
  • VMM Device Abstraction Layer: Introduced a device-agnostic Virtual Memory Management (VMM) abstraction in the GPU Memory Service with a --device-kind CLI option and VMMDevice protocol, preserving existing CUDA behavior while laying groundwork for XPU support, which raises NotImplementedError until Phase 2 (#9788).

General

  • XPU DRA Deployment Examples: Added XPU Dynamic Resource Allocation deployment YAMLs for aggregated and disaggregated vLLM serving, including tracing and Planner variants, giving users ready-made configs for aggregated serving, OpenTelemetry tracing, NIXL KV transfer, and Planner-based disaggregation on XPU. (#9253).

Fault Tolerance & Observability

  • OTLP Log Export Sink: Added an opt-in otel audit sink that exports chat-completion audit records as OTLP LogRecords, enabled via DYN_AUDIT_SINKS (#9390), and decoupled OTEL_EXPORT_ENABLED from DYN_LOGGING_JSONL so OTLP export initializes regardless of the local log format (#10817).
  • Unified Request-Trace Payload Capture: Folded audit-style payload logging into the request trace pipeline, emitting a single dynamo.request.trace.v1 stream with payload rows selected via DYN_REQUEST_TRACE_RECORDS (#11180), and added allowlisted HTTP request-header capture through DYN_REQUEST_TRACE_HTTP_HEADER_CAPTURE_LIST (#11386).
  • Frontend Multimodal Input Metrics: Added per-request counts of image_url, video_url, and audio_url content parts as frontend Prometheus histograms and request-completed log fields (#11166), followed by the dynamo_frontend_image_tokens_per_request histogram of calculated image-placeholder token counts (#12065).
  • Local Resource Monitor Caching: Cached process selection and aggregation in the local resource monitor so dashboard deltas no longer rescan every retained process series, and disconnected hidden dashboard tabs from Socket.IO until shown again, reducing background traffic and CPU use (#11336).
  • Tokenizer Cache Token Metrics: Added per-model dynamo_frontend_tokenizer_cache_cached_tokens_total and dynamo_frontend_tokenizer_cache_uncached_tokens_total counters alongside the existing aggregate hit and miss metrics, letting operators measure token-level cache reuse per served model instead of inferring it from aggregate counts (#11353).
  • Buffered ITL Metrics Collection: Reduced inter-token latency metrics overhead by buffering observations in a request-local Prometheus histogram, flushing every 64 samples instead of synchronizing on every output token, cutting the ITL histogram on-CPU cost by 71.9% while preserving the existing Prometheus schema (#11569).
  • Dynamic Log Filter Warning: Added a stderr warning emitted when a dynamic span or field selector forces logging to fall back from the lock-free Targets filter to EnvFilter, explaining the streaming-performance risk and how to restore the fast path (#11974).
  • Configurable Worker Inhibition Duration: Added DYN_RUNTIME_INHIBITED_DURATION_SECS to configure the local worker inhibition window, keeping the five-second default when unset and allowing 0 to disable local inhibition(#11993).

Engines

vLLM

  • Graph-Aware Self-Benchmark Sampling: Replaced linear per-request sweeps in the vLLM self-benchmark with deterministic prefill and decode sampling derived from the resolved PIECEWISE/FULL CUDA Graph configuration (#11509). Added an option to override the benchmark grid (#11713) and filtered infeasible EAGLE prefill KV-read coordinates (#12836).
  • Unified Backend Elastic EP: Added headless multi-node worker support and live elastic expert-parallel (EP) scaling via scale_elastic_ep to the vLLM unified backend, bringing it to parity with the legacy path and letting deployments scale data-parallel size up or down without restarting service (#10834).
  • vLLM Multiple Served Names: Enabled the vLLM backend to accept multiple --served-model-name values, registering the first as primary and the rest as aliases, matching SGLang. A single worker now serves several model names, with /v1/models and chat completions resolving aliases to the primary (#11611).
  • vLLM Render Client Support: Added a pooled HTTP client in the standalone External Processor (EPP) that calls vLLM's /v1/chat/completions/render endpoint for tokenization, configurable via DYN_EPP_TOKENIZER_SERVICE_URL and DYN_EPP_TOKENIZATION_TIMEOUT_MS, replacing an earlier offline tokenizer approach and classifying failures as typed Unavailable, Timeout, UpstreamStatus, or InvalidResponse errors (#11827).

SGLang

  • Multiple Served Model Names: Added support for registering an SGLang worker under multiple served model names via --served-model-name, canonicalizing alias requests to the primary name so engine routing, Prometheus metrics, and the OpenAI response.model field stay consistent (#11141).

TensorRT-LLM

  • Engine-Owned Conversation-Aware ADP Routing: Added opt-in TensorRT-LLM engine-owned conversation-affinity attention-DP routing, deriving the conversation id from agent_context.session_id and passing it as ConversationParams (#11609), with a --conversation-affinity flag and DYN_ENGINE_CONV_AFFINITY environment variable forcing engine-side assignment (#11705).
  • Prefill Handoff And Event Batching: Added generic Rust and Python KvEventPublisher.publish_batch APIs so one ordered source batch reaches the KV event processor as a single input (#11783), and skipped duplicate prompt metadata in the legacy TensorRT-LLM prefill handoff for text-only requests (#11202).

Infrastructure Modernization

  • Dependency and Service Refresh: Aligned aiohttp to 3.14.3 or later across all container images, including the TensorRT-LLM runtime's system-site install (#13132, #13195, #13227, #13256), bumped etcd to v3.5.33 and NATS Server to v2.12.14 (#13016, #13225), moved vLLM images to NIXL v1.3.2 (#12882, #13185), and adopted the published AIConfigurator v0.11.0 release (#13280).
  • TCP Request-Plane Fast Path: Removed two large request-payload copies from the TCP request-plane frontend send path for payloads of at least 4096 bytes via vectored socket writes (#10519), and replaced the Tokio mutex guarding the TCP stream registry with parking_lot::Mutex (#11065).
  • Python Request-Plane Codec Transcoding: Added a statically dispatched ingress payload adapter that transcodes request-plane wire bytes directly to and from Python objects, making MessagePack the default with JSON as an explicit fallback (#11104), interned the six annotated-response envelope keys (#11331), and added per-worker codec negotiation via discovery metadata (#12658).
  • Client TCP Address Lookup: Added Client.instance_tcp_addresses() to the Python bindings, returning a dict mapping instance id to TCP transport address for an endpoint's registered instances, letting callers reach peer worker nodes without reading the discovery/etcd registry layout directly (#11540).

Hardware

  • XPU Deployment Sanity Checks: Extended dev/sanity_check.py to detect and report Intel XPU systems alongside NVIDIA systems, and added xpu-smi to XPU Docker images, giving users GPU, framework, and filesystem diagnostics for Intel XPU deployments (#7702).
  • vLLM XPU Support Enablement: Resolved the accelerator device from vLLM's current platform in the multimodal embedding cache connector so CPU-cached embedding loads work on both CUDA and XPU (#9938), and updated the XPU aggregated examples for vLLM XPU 0.24.0 by removing the --block-size override (#11229).

KV Block Manager

  • Unified KV Block Clearing: Added a POST /engine/control/clear_kv_blocks endpoint for unified vLLM workers that resets the prefix cache and connector state without preempting active requests. Supported aggregated, prefill, and decode workers, and propagated cached-token usage so clients can verify cache hits and post-clear misses (#11451).

General

  • vLLM v0.26.0 Upgrade: Upgraded the pinned vLLM to v0.26.0 with refreshed CUDA and CPU runtime images, FlashInfer 0.6.14, and NIXL v1.3.1, later moved to v1.3.2 (#12202, #11606), and moved the XPU container to the vllm/vllm-openai-xpu v0.26.0 image (#12190). The upgrade brings reasoning-parser configuration and structured-output handling to the Rust vLLM backend.
  • TensorRT-LLM v1.3.0rc22 Upgrade: Upgraded TensorRT-LLM to v1.3.0rc22 across supported runtime configurations and platforms with updated baseline SBOM records (#11086, #11763, #12071), including improved multimodal request handling and image-token range detection for TensorRT-LLM KV event processing.
  • SGLang v0.5.16 Upgrade: Upgraded the SGLang backend and CUDA 13 runtime to v0.5.16 on the lmsysorg/sglang:v0.5.16-cu130-runtime image with NIXL v1.3.0 pinned for disaggregated serving (#11596, #12081), setting incremental_streaming_output before SGLang resolves its immutable startup record and removing the expired 0.5.14 profiling compatibility path.
  • dynamo-parsers 7.0.1 Upgrade: Upgraded dynamo-parsers to 7.0.1 in the workspace and Python bindings (#12147, #12451, #12509), aligning the frontend crate graph on dynamo-protocols 4.0.0 and dynamo-renderer 3.0.0 and picking up the DeepSeek fix that stops tool-call markup from leaking into reasoning content.
  • Inline Compliance Artifact Coverage: Extended the inline license/SBOM compliance pipeline to the Planner and Operator images (#11328) and the frontend and snapshot images (#11533). Added per-build OSRB CSV diffs against context-dependent baselines (#11240) and passed the IRSA sccache secrets to compliance-extract rebuilds (#11181).
  • Power Agent DCGM Actuator: Added an opt-in DCGM actuator for the Power Agent, letting operators set agent.actuator=dcgm to write power caps through the cluster nvidia-dcgm hostengine, with NVML remaining the default and cap writes anchored to a transaction-verified GPU UUID (#9790).
  • Self-Hosted MDC Default: Flipped DYN_SELF_HOST_METADATA to on by default so workers advertise their model card (MDC) files over the system_status_server instead of requiring shared storage, and added a startup hard-fail when self-hosting is enabled but DYN_SYSTEM_PORT is unset instead of silently degrading hf:// MDC advertisement (#11417).
  • Hugging Face Hub LoRA Source: Added native hf://<repo>[@revision] support for dynamic LoRA loading, downloading adapters into the standard Hugging Face Hub cache with revision pinning, timeout limits, and integrity checks instead of requiring a separate copy under DYN_LORA_PATH (#11431).
  • Instance Transport Object Model: Added Client.instances() to the Python bindings, returning read-only Instance snapshot objects with instance_id, namespace, component, endpoint, device_type, and transport attributes, preserving all transport types including NATS instead of flattening addresses to opaque TCP-only strings (#11617).
  • Static Logging Filter Fast Path: Switched to a metadata-only Targets filter for static DYN_LOG and configured log filter directives, falling back to EnvFilter only for dynamic span or field directives, removing a shared span-matcher lock from the request-span lifecycle and reducing frontend CPU overhead on streaming responses (#11820).
  • EFA Installer Version Bump: Updated the AWS EFA Installer to version 1.49.0 and reverted the prior libfabric override so the AWS container now uses the installer's provided libfabric, improving compatibility with EFA networking (#12010).

Recipes

  • Model-Agnostic Accuracy Check: Added recipes/accuracy/, an AIPerf-based Kubernetes Job that runs a public benchmark against any deployed Dynamo recipe and grades the answers, with a methodology README (#12495, #13215).
  • GPT-OSS-120B vLLM Deployment Recipes: Added GPT-OSS-120B recipes for aggregated and disaggregated vLLM serving on B200 and H200 GPUs, covering speculative decoding, KV-cache routing and offload, reasoning, tool calling, automated model downloads, and an AIPerf trace-replay benchmarking workflow with documented prerequisites, quick start, performance results, and known limitations (#11506). Fixed the disaggregated deployments so they no longer force etcd discovery at a hardcoded endpoint, letting the cluster discovery backend injected by the Operator apply, while keeping the NATS message bus setting (#12070).
  • Hugging Face Hub Pin Bump: Upgraded the huggingface_hub CLI pin from 1.11.0 to 1.16.4 in model-cache and vLLM LoRA helper job YAML files, fixing a missing click dependency that broke the hf download command (#10986).
  • ARM Recipe Build Tools: Added build-essential installation to ARM/GB200-facing benchmark recipes so aiperf>=0.8.0 can build its crick dependency from source, since crick lacks a Linux aarch64 wheel (#11278).
  • DSV4 Agentic vLLM Recipes: Added DeepSeek-V4-Flash and DeepSeek-V4-Pro deployment recipes combining Dynamo with vLLM, covering aggregated and P/D-disaggregated profiles on B200 and H200 for agentic workloads, plus model-cache manifests and an AIPerf Moontrace perf job. Disaggregated variants move KV over NIXL GPU-Direct RDMA (#11429).
  • TokenSpeed FlashInfer Cache Fix: Pinned the TokenSpeed recipe's base image by immutable digest, fixed the source revision, and aligned the FlashInfer JIT cache version required before building kernels, preventing version mismatches in kimi-k2.5 builds (#11465).
  • Inkling NVFP4 B200 Recipe: Added a recipe for thinkingmachines/Inkling-NVFP4 with Dynamo plus SGLang aggregated serving on one 8x B200 node, including NVFP4 quantization, FA4 attention, EAGLE speculative decoding, and a model-cache download job. It documents a required DYN_FORWARDPASS_METRIC_PORT override avoiding a scheduler crash (#11743).
  • GLM-5.2 SGLang Recipes: Added GLM-5.2 SGLang aggregated and disaggregated serving recipes for B200 and H200 clusters, including model-cache setup, EAGLE speculative decoding, HiCache CPU offload, NVFP4/FP8 quantization, and an AIPerf trace-replay benchmark workflow, along with fixes for reasoning-parser configuration, a non-stream logprobs 500 error, and trace-staging symlink handling. (#11926).
  • Kimi-K3 Recipe Publication: Brought the Kimi-K3 recipe assets (README, model-cache config, and vLLM deploy manifests for GB200/GB300 aggregated and disaggregated setups) onto main and added its row to the Available Recipes table, so users on main can now discover and deploy Kimi-K3 without switching to the release branch (#12228).

Bug Fixes

Frontend

  • LoRA Unload Resource Release: Fixed the Frontend to release the worker connection and associated memory after each LoRA unload, eliminating unbounded RSS growth of roughly 95 MB per load/unload cycle (#12743, #13203).
  • FinishReason Error Propagation: Fixed the Frontend to read the string form of FinishReason that workers emit, so request-level error messages reach the caller instead of a bare HTTP 500 (#13118, #13212).
  • MDC Version Compatibility Fix: Preserved compatibility for Model Discovery Card (MDC) versions across rolling upgrades by accepting legacy v1.2 context_length fallbacks and omitting default-valued post-v1.3 router fields so v1.3 frontends can deserialize v1.4 MDCs, while keeping removed v1.3 router behavior like durable_kv_events=true failing closed (#12801).
  • Hidden Disabled Batch Routes: Fixed the OpenAPI specification to omit Batch and Files API routes when their internal enable flag is off, so disabled surfaces no longer advertise themselves (#12760, #13122).
  • Chat Template Kwargs Plumbing: Fixed the vLLM chat processor so per-request chat_template_kwargs reach the chat template (#11729), threaded the server-wide --default-chat-template-kwargs flag through with per-request values overriding defaults (#11731), and rejected a chat_template nested inside kwargs, closing a validation bypass (#11755).
  • Streaming Tool-Call Emission Fixes: Reworked the streaming converter for the Anthropic Messages and OpenAI Responses formats to accumulate fragmented tool-call identities and argument deltas, publishing each complete block on finish (#10988), and synthesized a finish_reason: "tool_calls" chunk when the upstream stream omitted its terminal reason (#11045).
  • Guided JSON Reasoning Parsing: Fixed guided decoding for force-reasoning parsers, preserving both reasoning_content and structured tool calls for tool_choice=required and named tool calls (#11205), and routed response_format: json_schema output to content rather than reasoning_content with SGLang reasoning enabled (#11512).
  • Token Usage Accounting Fixes: Corrected Anthropic /v1/messages input_tokens accounting so cached tokens are no longer double-counted (#11030), propagated completion_tokens_details including reasoning_tokens into Chat Completions responses (#11027), and added DYN_ENABLE_FORCE_INCLUDE_USAGE to require usage in streaming responses even when clients omit include_usage (#12017).
  • Frontend Request Validation Hardening: Mapped InvalidArgument errors from the annotated first-event path to HTTP 400 (#11453) and converted preserved aggregation errors on /v1/audio/speech so worker rejections return 4xx (#12832). Restored boundary validation so max_tokens: 0 returns HTTP 400 (#11394), and tolerated raw control bytes in JSON string fields on the OpenAI endpoints (#11114).
  • Responses API Event Semantics: Mapped length-limited generation to incomplete response states, emitting response.incomplete with max_output_tokens as the reason (#12182), emitted requested reasoning summaries through the full ordered streaming lifecycle (#12183), and rejected completion_token_ids and prompt_logprobs requested via nvext.extra_fields with a field-specific 501 (#11253).
  • Release 1.4.0 Frontend Backports: Backported validation of converted OpenAI Responses and Anthropic requests, returning 400 for malformed inputs and 501 for unsupported features (#12904), and rejected non-progressing guided JSON cycles (#12853). Forwarded vLLM tool-call and structured-output constraints to workers (#12877), preserved explicit top_logprobs=0 (#12681), and converted SGLang log_probs and top_logprobs to the OpenAI contract (#12988).
  • Model Registry Reconciliation: Added periodic reconciliation that queries all discovered models every 30 seconds and retries incomplete registrations, so a frontend's local model registry recovers from missed discovery events or transient registration failures without manual intervention (#11092).
  • Async Tokenization Offload: Moved prompt tokenization off the Frontend's tokio event loop onto the blocking thread pool, matching the embedding-path offload. This prevents BPE encoding of long prompts from stalling the event loop and wrongly marking healthy workers as down in disaggregated serving (#11200).
  • MiniMax M2 Tool-Calling Fix: Fixed forced and named tool calling for MiniMax M2 with reasoning by wiring minimax_m2 into Dynamo's force-reasoning and guided-output parsing paths, so tool output is separated from reasoning instead of misparsed. Docs now recommend minimax_m2 over deprecated minimax_append_think (#11554).
  • Kimi Multi-Turn Reasoning Fix: Fixed post-tool reasoning parsing for Kimi K2.5/2.6 so private reasoning and the </think> marker no longer leak into content, and normalized thinking/enable_thinking values consistently across prompt rendering, the SGLang reasoning gate, and the postprocessor (#11653).
  • SGLang Reasoning Delimiter Fix: Fixed the custom SGLang chat processor to preserve special delimiter tokens during incremental detokenization when a tool-call or reasoning parser is active, so reasoning content populates correctly in streaming output (#11748).
  • Startup File-Descriptor Limit: Raised the process's soft RLIMIT_NOFILE at frontend startup, bounded by the environment's hard limit, to prevent the TCP accept() loop from spiraling into EMFILE errors and flooding logs under high connection concurrency (#11802).
  • Tool-Result Media Preservation: Fixed Anthropic tool results so embedded images are no longer reduced to text-only content, and added support for image, video, and audio URL parts in OpenAI-compatible tool messages, routing them through the existing user-media preprocessing path (#12180).
  • Parallel Tool Calls Enforcement: Added a post-parse fallback that caps chat completion responses to the first tool call when parallel_tool_calls: false is set, covering streaming and non-streaming responses across vLLM, SGLang, and TensorRT-LLM backends (#12201).
  • Mistral Tokenizer Chat Template Fix: Fixed an AttributeError in the vLLM chat-processor path where MistralTokenizer lacks a chat_template attribute, which previously blocked Mistral models from registering and caused chat requests to return HTTP 404 (#12659).

Multimodal & Diffusion

  • Preserved Media URL Client Errors: Fixed audio and video loaders to preserve URL-validation and HTTP-status errors during per-item and batch handling, so intentional client rejections keep their 4xx status instead of flattening into generic failures (#12407, #13193).
  • GLM-Image NIXL Deploy Config: Migrated the GLM-Image AR to DiT NIXL disaggregation stage config off vllm-omni's legacy stage_args schema to the stages deploy schema (#12984), following a fix that declared the entry stage's engine_input_source after vllm-omni v0.26.0rc1 removed its default (#12709).
  • Multimodal Trust-Remote-Code Fix: Fixed multimodal and encode workers across vLLM, SGLang, and TensorRT-LLM to thread trust_remote_code from the engine config instead of hardcoding True, so custom model repo code no longer executes on load unless the operator explicitly enabled it (#10738).
  • SSRF-Blocked Media URL Status: Fixed multimodal image loading to return HTTP 400 instead of 500 when an image URL is blocked by the SSRF guard, preserving the UrlValidationError so clients get the block reason (#11312).

Scheduling

  • SGLang Global Router Support: Fixed the prefill router's execute_prefill logic to detect SGLang workers and return a Bootstrap outcome with bootstrap host, port, and room instead of hanging, so decode workers now receive requests when using SGLang with the global router (#7897).
  • Planner Worker Discovery Hardening: Added runtimeNamespace to replica status so Planner reads the runtime namespace from DGD status (#11352), resolved workers by DGD component type (#11578), and matched model cards against .status.components[*].componentNames (#12047). GlobalPlanner read the v1beta1 spec.components schema (#11990) and rejected scaling with DynamoGraphDeploymentNotReadyError on unready deployments (#10526).
  • Exact Push Router Dispatch: Fixed PushRouter::direct() to enforce exact-dispatch behavior, returning a typed CannotConnect error instead of silently rerouting to another worker when the selected instance disappears before transport resolution. Transport resolution now runs before overload checks, so stale overload metadata cannot mask a missing exact target (#10429).
  • EAGLE3 KV Routing Fix: Fixed enable_eagle detection in SGLang to derive from SpeculativeAlgorithm.is_eagle() instead of a hand-maintained name set, correcting KV-aware routing for EAGLE3 and FROZEN_KV_MTP workers that previously reported zero cached blocks due to hash misalignment (#10982).
  • KvRouter Cancellation Scope Fix: Fixed KvRouter to cancel a child token instead of the runtime's primary token on drop, so rebuilding the Router after all workers exit no longer tears down the entire runtime with an etcd lease error (#11390).
  • Prefill EAGLE KV Routing Fix: Fixed prefill KV routing to detect EAGLE/MTP support from the prefill endpoint's runtime config instead of always disabling EAGLE hash mode, preventing zero cache-block predictions and restoring correct prefix cache hits when SGLang prefill workers enable EAGLE (#11764).

Kubernetes Deployment

  • CRIU Checkpoint Image Mechanics: Added an imageIoMode CRIU knob (writeback or direct, defaulting to direct) and bumped go-criu to v8.4.0 (#11476), pinned the snapshot image CRIU build to a fixed revision (#11659), made rootfs diff capture atomic with zero-byte-archive rejection (#11883), and fixed the DRA nvidia-smi probe to enter the workload PID namespace (#12227).
  • DGD Admission Webhook Migration: Served both v1alpha1 and v1beta1 DynamoGraphDeployment admission endpoints via the v1beta1 hub (#11117), restored pre-existing v1alpha1 behaviors with per-container CEL scoping (#11126), and fixed volume-mount composition so service PVC mounts survived (#11263). Made cluster-wide handlers Lease-aware (#11597) and gated Operator readiness on the webhook server with a TLS-checked /readyz (#12255).
  • DGDR Profiling And Status Hardening: Added a reusable Go package that applies versioned overrides.dgd patches using Kubernetes structural merge semantics (#11124), an idempotent profiling-to-deployment handoff so a DynamoGraphDeploymentRequest no longer sticks in Deploying (#11250), and a MessageValidationPassed constant on the Validation=True condition (#11261).
  • Explicit Service Mesh Override: Added a serviceMesh.enabled option to the Operator configuration so users can explicitly enable or disable Istio integration, fixing startup failures that occurred when Istio's API group was partially present but the DestinationRule resource was unavailable (#10773).
  • Docker Secret Index Resilience: Fixed the Operator's controller manager crashing on startup when the initial Docker secrets index refresh encountered malformed kubernetes.io/dockerconfigjson secrets, now skipping invalid entries and logging aggregated errors while continuing to index valid ones (#10789).
  • LWS Worker GPU Validation: Removed the hardcoded GPU limit check from LWS worker pod template generation, letting Kubernetes and device admission decide pod validity so CPU-only deployments no longer fail to render (#11115).
  • Duplicate Compilation Cache Mounts: Fixed pod generation to reuse an existing writable compilation-cache mount instead of appending a duplicate one, and rejected mount-path conflicts and read-only cache mounts, preventing Grove PodCliqueSet validation failures on converted v1alpha1 components (#11264).
  • EPP DiskPressure Resilience: Set default CPU, memory, and ephemeral storage requests for the EndPoint Picker (EPP) and lowered the default RUST_LOG level to info, preventing kubelet from evicting EPP under node DiskPressure and stalling InferencePool deployments (#11550).
  • Multi-Node TensorRT-LLM Env Relay: Fixed multi-node TensorRT-LLM launch commands to forward the CPATH and TRITON_PTXAS_PATH environment variables and avoid duplicate forwarding, preventing build and compilation failures on multi-node deployments (#11999).
  • DGDSA Replica Ownership Fix: Fixed a reconciliation race between the DynamoGraphDeployment and DynamoGraphDeploymentScalingAdapter (DGDSA) controllers by replacing full-spec DGDSA synchronization with a patch that preserves DGDSA-owned replica counts, preventing stale DGD values from overwriting scaling adapter replicas during serve, unserve, and scaling operations (#12087).
  • Event Channel Metadata Compatibility: Fixed DynamoWorkerMetadata deserialization by reconstructing legacy namespace/component fields alongside the current scope field for Kubernetes event channels, preventing rolling upgrades from rejecting worker records due to version mismatches (#12708).

Fault Tolerance & Observability

  • Bounded Endpoint Drain Shutdown: Fixed a zombie-worker failure mode where a lost etcd lease with a stuck in-flight request left a worker Running but unable to serve; per-endpoint drain now respects the graceful shutdown timeout and lease loss routes through Runtime::shutdown() (#11068).

Engines

vLLM

  • Writable vLLM Cache Directories: Pre-created the vLLM cache root in the container image with group 0 write access so pods running under a UID other than 1000 with gid 0 no longer fail engine initialization with EACCES on the first torch compile (#13040), cherry-picked into the release branch (#13045).
  • Lifecycle-Aware Disaggregated LoRA Prefill: Registered and served LoRA lifecycle endpoints on vLLM prefill workers, with metadata-only initial registration, adapters activated from the inference-time LoRARequest, and endpoint deduplication by physical pod (#11520). Fixed dynamic LoRA registration to publish adapter model cards with the engine-actual main-attention KV block size instead of the raw CLI block_size (#11867).
  • vLLM Embedding Pooling Correctness: Set the embedding worker's pooling task to embed in PoolingParams so vLLM returns one pooled, L2-normalized embedding per input (#10248), and propagated truncate_prompt_tokens to vLLM encode as tokenizer kwargs for raw-text embedding inputs (#11099).
  • Embeddings Dimensions Type Validation: Fixed the /v1/embeddings endpoint to reject boolean values passed as the dimensions parameter, raising a TypeError instead of silently truncating embedding vectors to one dimension. Prevents corrupted embeddings from propagating undetected into downstream vector stores or RAG systems (#9873).
  • Cache Salt Propagation Fix: Fixed the vLLM text-mode path to propagate nvext.cache_salt from the request into the prompt object sent to the vLLM engine, ensuring KV cache keys reflect the requested cache salt instead of losing it. (#11082).
  • Qwen3-TTS Prompt Estimator Fix: Fixed Qwen3-TTS prompt-length estimation to call the estimator through Qwen3TTSPromptEmbedsBuilder instead of the stale talker-model import, avoiding the generic 2048-token fallback that could produce invalid or poor-quality audio (#11595).
  • vLLM EngineCore Shutdown on Failed Startup: Fixed vLLM decode workers to shut down the EngineCore process when Dynamo-side initialization fails after engine startup, so failed workers exit cleanly instead of leaving a live engine process behind (#12867).

SGLang

  • HiCache Capacity Publication: Fixed SGLang workers to publish ratio-based HiCache offload capacity, so Dynamo Router can account for SGLang CPU-offload cache tiers (#12953, #13060).
  • Writable FlashInfer Cubin Directories: Marked the installed flashinfer_cubin/cubins directories writable in the SGLang runtime image before switching to the non-root dynamo user (#11154), added matching group-write permissions in the vLLM runtime images for arbitrary OpenShift UIDs in group 0 (#12770), and carried the vLLM fix into the release branch (#12976).
  • SGLang Token Usage Reporting: Populated prompt_tokens_details.cached_tokens in terminal completion usage from final SGLang response metadata, restoring cache observability for warm requests including HiCache (#10925), and treated streamed usage metadata as optional so final chunks without token counts no longer raise KeyError (#11062).
  • SGLang Response Format Forwarding: Fixed the SGLang chat processor to forward the OpenAI response_format parameter to decode and prefill engine calls, enabling SGLang-native JSON schema and JSON object structured output constraints when using --dyn-chat-processor sglang. (#10259).
  • SGLang Multimodal Audio Forwarding: Fixed forwarding of Dynamo's --enable-multimodal setting to SGLang's server arguments and added extraction and forwarding of audio_url/audio_data alongside image and video inputs across aggregate, disaggregated prefill, and disaggregated decode paths. Unextracted audio content is now rejected using the same guard applied to other modalities (#11747).
  • SGLang Bootstrap Host Discovery: Fixed prefill bootstrap endpoint discovery to prefer the concrete, non-loopback SGLang host from GetServerInfo before falling back to dist_init_addr or local-IP resolution. This prevents decode workers from receiving unreachable loopback or wildcard addresses for bootstrap (#12026).
  • SGLang Conditional-Disagg Error Propagation: Fixed the SGLang rejection of the x-bypass-remote-prefill annotation so clients receive the intended unsupported-feature message instead of a bare Internal Server Error (#12616).
  • SGLang NVExt Token Validation: Rejected out-of-vocabulary IDs from pre-tokenized nvext.token_data before reaching SGLang's embedding lookup, while preserving valid request-scoped multimodal sentinel token IDs, backported to release/1.4.0 (#12852).

TensorRT-LLM

  • TensorRT-LLM Mpmath Version Cap: Capped mpmath below 1.4 in the trtllm extra to prevent the resolver from pulling mpmath 1.5.0a0, which removed bitcount and caused the TensorRT-LLM backend to crash on import. (#11402).
  • TensorRT-LLM Multimodal Request Handling: Scoped NIXL Connect initialization to multimodal disaggregated workers so plain text workers no longer load the nixl bindings (#11373), sized an omitted max_tokens from the expanded prompt length instead of the default 32 (#11513), and routed video_url inputs through Dynamo's URL validation policy before local decoding (#12002).
  • TensorRT-LLM RC22 Compatibility Fixes: Updated disaggregated serving for TensorRT-LLM rc22's (node_id, process_id) machine-ID pair and stopped dispatching GEN when a context request finished during prefill without KV-cache handoff IDs (#12203), with engine argument defaults read through the changed TensorRT-LLM API (#11741).

Infrastructure Modernization

  • ZMQ Subscriber Socket Leak: Fixed a leak where a dropped ZmqSubTransport left its subscriber pump running and SUB socket open, accumulating sockets until "Too many open files". Pumps now stop when their final owner is dropped, and cancellation propagates to discovery watch tasks (#11730).

KV Block Manager

  • KVBM Decode KV Cache Alignment: Configured KVBM prefill and decode workers with --disable-hybrid-kv-cache-manager so both sides share the same vLLM KV-cache layout, marking decode workers with --disaggregation-mode decode across launchers, examples, and the integration test (#12821). Stopped suppressing an explicit kv_events_config on disaggregated decode workers and set up the Dynamo KV-event relay for them (#12084).
  • HMA Disable For PdConnector: Fixed KVBM and LMCache disaggregated serving startup failures by disabling vLLM's hybrid KV cache manager (HMA) for PdConnector configurations whose child connectors, such as DynamoConnector or LMCacheConnectorV1 paired with NixlConnector, do not all support HMA, preventing the MultiConnector assertion during EngineCore startup (#11608).

General

  • etcd Connection Resilience Fixes: Added a cancellable startup retry loop for the initial etcd connection and primary lease creation, retrying up to 120s with exponential backoff (#10799), and introduced a WatchEvent::Resync snapshot after watch reconnects so KvCache, TypedPrefixWatcher, and session-affinity claim watchers rebuild state and stale instances are removed (#11146).
  • Shared ZMQ Context Fix: Fixed the event plane's direct (broker-less) PUB/SUB transport to reuse a single process-wide ZMQ context instead of creating one per socket, preventing unbounded OS-thread growth on frontends as worker fleets scaled up, which had starved the async runtime and triggered liveness-probe restart loops (#11246).
  • JSON5 Config Parsing Fix: Fixed max context length derivation to parse config.json with JSON5 instead of strict JSON, preventing registration failures for models like Nemotron-H whose configs contain non-finite literals such as Infinity in unused fields (#11445).
  • Triton CUDA 13 Image Fix: Updated the Triton example's default server image to tritonserver:25.10-py3 for CUDA 13.0.2 compatibility, fixed the DCGM library bind-mount path and permissions, and added a build-time import tritonserver smoke check (#12577).

Recipes

  • DeepSeek-V4 H200 Checkpoint Download: Fixed the deepseek-v4 and deepseek-v4-pro H200 recipes to download the FP8 checkpoint instead of the B200 NVFP4 variant by parameterizing the model-download Job (#13034, #13213).
  • DeepSeek-R1 Disagg Decode Mode: Fixed the deepseek-r1 vLLM disaggregated recipe to declare --disaggregation-mode decode on the decode worker, which previously never became ready (#13046).
  • Removed Stale Router Flags: Removed the retired --router-reset-states flags from affected recipes so they launch cleanly against the shipped Router CLI (#13116, #13117).

Developer Experience

Documentation Website Refactor

  • Tab-Based Documentation Navigation: Restructured the documentation site from a single flat sidebar into top-level Home, User Guide, Developer Guide, Recipes, Reference, Blog, and Community tabs, with an explicit redirect table of 357 redirects so every previously live URL still resolves (#10855). Restored the per-backend SGLang, TensorRT-LLM, and vLLM multimodal deployment guides to the navigation and fixed a stale anchor in their cross-references (#11064), and consolidated Kubernetes content while correcting remaining navigation, cross-reference, styling, and release and observability pages missed by the initial pass (#12243).
  • Simplified Chinese Docs Localization: Registered all 13 existing zh-CN translations in the Fern nav under the hidden Additional Resources section so they publish at real URLs and the language switcher links resolve instead of serving soft-404 stubs (#11088), then replaced the raw HTML switcher badge with a plain Markdown link and dropped duplicate body H1s on 5 translated pages (#11148). Migrated the manual switcher to Fern native localization, moving translations to a fern/translations/zh-CN mirror tree with a translations: config that adds the header language dropdown and falls back to English for untranslated pages on the dev version (#11161).
  • Tagged-Commit Release Snapshots: Fixed the Fern docs release workflow to build versioned pages and navigation from the tagged commit's own docs, navigation, and converter instead of whatever content was on main, preventing release tags from publishing mismatched content, and added a local release dry-run script under fern/ for pre-publish validation (#11140).

Documentation

  • Kubernetes & Deployment Docs: Restructured the Gateway API (GAIE) documentation into overview and quickstart pages with standalone EPP on-ramp guidance (#10956, #10993, #11119, #12480), corrected the Dynamo Operator's default termination grace period (#11120), aligned install examples with current platform versions (#11486), and documented DGDR spec.overrides.dgd merge semantics (#11487), scale-to-zero behavior and its recovery limits (#11536), and the TensorRT-LLM snapshot support matrix (#11674).
  • Recipe & Model Pages: Added recipe pages for the Inkling NVFP4 day-0 launch (#11749, #11807), GLM-5.2 on SGLang across B200 NVFP4 and H200 FP8 targets (#11972), Kimi-K3 (#12210), and vLLM agentic targets for GPT-OSS-120B (#11980), plus a Model Early Access Builds reference page standardizing early-access terminology (#11898).
  • Router & KV Routing Reference: Corrected the Router reference pages to match shipped v1.4.0 behavior (#11892, #12949, #12983), clarified that KV routing requires explicit worker-side event publishing and distinguished the fallback modes (#11505), added a KV offloading support matrix across backends and cache tiers (#11973), updated SGLang HiCache minimum-version requirements (#11183), and corrected session-radix ownership claims (#11241).
  • Backend Setup & Troubleshooting: Directed TensorRT-LLM users to the container-based install path (#11066), recommended the PyTorch multiprocessing backend over Ray for vLLM multi-node deployments (#11259), documented the CUDA 13 FlashInfer sampler failure and its workaround (#10954), the NVENC hardware requirement for TensorRT-LLM video diffusion MP4 output (#11457), corrected the TensorRT-LLM UCX environment-variable note and EFA image facts (#13049), vLLM worker-role flags and --kv-transfer-config (#12666), and XPU deployment paths for Intel GPUs including DRA requirements (#11894).
  • Planner & Profiler Docs: Documented rapid-profiling behavior when no SLA-feasible configuration exists and corrected the unachievable-SLA constraints claim (#11612, #12774), and corrected the Planner configuration reference covering all four optimization targets and TLS verification behavior (#11613).
  • Simulation Docs: Reframed the DynoSim documentation around deterministic offline replay and live Mocker with a dedicated simulation-model page (#12172, #11430), corrected the load-generator ISL/OSL preset semantics (#10791), and corrected Spica's kv_load_ratio support guidance (#12714).
  • Observability Docs: Documented the actual OTLP endpoint resolution chain and sampling controls, removing a stale fallback claim that caused silent log loss (#11337), and corrected tokenizer-cache metric absence semantics while adding the v1.4.0 deprecations section (#12775), and marked the CKF counter as registered but never incremented in this release (#13149).
  • Cleanup: Replaced references to the removed experimental unified backend engines with current guidance (#11833).

Known Issues

Backend (vLLM)

  • Omni Cases Fail On GB200: On ARM64 (GB200) hosts, every vLLM Omni case fails at startup with ModuleNotFoundError: No module named vllm_omni, covering the aggregated text, audio, image, video, and image-to-video paths as well as disaggregated GLM image serving. The cause is that the vllm_omni module is not shipped in the ARM64 container image; ARM is not a supported platform for vLLM Omni, as noted in the installation section of the vLLM-Omni installation guidance. No workaround is available yet; Omni workloads must run on x86_64 images.
  • Blocked vLLM Engine Reports Healthy: On RL vLLM workers, when the engine execution loop is blocked (for example during a long weight transfer), the /health endpoint continues to report ready and the engine monitor emits no error, so requests stall and the worker never restarts or otherwise self-heals. This is the current intended behavior: weight transfer over IPC/ZMQ legitimately runs long on larger models such as GLM 5.2 and Kimi K3, and Dynamo does not interrupt an in-flight transfer; detection plus restart or resume responsibility is planned to move to the RL framework, with Dynamo providing only the signal. No workaround is available yet.
  • Stale Weight Version After Update: In reinforcement learning workflows with the vLLM worker, the reported weight version stays at its initial value after weights are refreshed, so clients cannot tell that new weights were loaded. Weight-version tracking is only incremented on the /engine API path, so any out-of-band weight update that bypasses that endpoint leaves the version surface unchanged, and an undeclared version is indistinguishable from a declared one. No workaround is available yet.

Backend (TensorRT-LLM)

  • Killed Engine Worker Reports Healthy: After the TensorRT-LLM engine core dies from an out-of-memory kill or SIGKILL, check_health() continues to report the worker as healthy and TrtllmEngineMonitor never trips, so the dead worker keeps attracting traffic and requests routed to it hang indefinitely. Carried over from v1.3.0.
  • Partial EFA Allocation Transfer Failure: On GB200 (arm64, p6e) nodes, disaggregated requests return an empty response with finish_reason=cancelled and the decode worker logs a NIXL KVReceiver exception when a worker is given fewer EFA devices than the node provides (for example 2 EFA devices with 4 GPUs); libfabric reports a rail 0 completion-queue abort. Workaround: allocate all EFA devices present on the node so each GPU is matched 1:1 with an EFA device, and do not request a subset of the node's EFA devices. Carried over from v1.3.0.

Backend (SGLang)

  • Prefill Worker Startup Assertion: An SGLang prefill worker fails to start when configured with data parallelism of 2 and max-running-requests=1, aborting during initialization with AssertionError: max_running_request is zero. Carried over from v1.3.0.

Multimodal & Diffusion

  • Wan2.2 Video Generation Worker Crash: Video generation requests served through the vLLM-Omni diffusion worker terminate the worker with a CUDA illegal memory access fault raised in the Wan2.2 transformer cross-attention path, and the request fails without a response. Carried over from v1.3.0.

Router & Runtime

  • Disaggregation Stays Off Permanently: When the Router briefly sees an ambiguous endpoint-scoped prefill/decode topology, disaggregated serving is turned off and never re-enabled after the topology resolves to a valid P/D layout, so all subsequent requests are served aggregated for the lifetime of the Router. Requests are not failed: they continue to return HTTP 200, only the disaggregation path stays inactive. The cause is the Router's worker removal and deactivation path, which is removed by a desired-state discovery refactor on the main branch that is too broad to backport to this release. No workaround is available yet.
  • Explicit Prefill Pin Request Failure: A request that explicitly pins a prefill worker fails hard when that target is locally inhibited: it is neither dispatched to the pinned worker nor redirected, even when other healthy prefill workers are available. This is intended behavior for 1.4, since an explicit prefill pin is treated as a hard pin with fail-closed semantics, and no fallback or retry path is implemented; a soft-pin mode with fallback is under consideration for a future release after the routing refactor. No workaround is available yet.
  • Long-Running Request State Eviction: The KV Router can discard the tracked KV state for a request that is still running if that request stays active longer than the absolute-age expiry window, whose default is 5 minutes. Once the state is removed, the Router no longer accounts for that sequence's blocks when scoring workers, so subsequent routing decisions and cached-prefix accounting for long-lived requests become inaccurate. This is expected behavior of the current stale-state guard rather than a regression: the absolute-age cleanup does not check whether the sequence is still lifecycle-active, and raising the default expiry is under consideration for a future release. No workaround is available yet.
  • KV Source Recovery Diagnostics Incorrect: After a model worker drops off the Router's KV event source and then rejoins, the Router logs the source-health warning a second time and never emits the kv_event_source_recovered signal, so health diagnostics continue to report the KV event source as degraded even though it has recovered. The defect is limited to diagnostics and observability: KV-aware routing recovers on rejoin and serving stays healthy. No workaround is available yet.

Planner & Profiler

  • SGLang Prefill Queue Depth Underreported: When running SGLang in disaggregated serving, the prefill role reports a queued_prefill_tokens value that stays at or near zero even while requests are backlogged waiting for chunked-prefill compute admission, so the Planner scales on an artificially low prefill load signal and under-provisions prefill workers. Carried over from v1.3.0.
  • SLA Planner Load Scaler Stall: The SLA Planner's load scaler stays in the insufficient_data state and never issues scaling decisions, so replica counts are not adjusted in response to load. Carried over from v1.3.0.
  • Mocker AIC Perf Model Crash: Starting dynamo.mocker with the --aic-perf-model option crashes immediately: the Mocker's memory estimation path expects an sdk.memory module that is not present in aiconfigurator 0.9.0, so startup fails before the Mocker begins serving. Carried over from v1.3.0.
  • Mocker Thorough Profiling Pod Failure: Running the pre-deployment Profiler with the Mocker backend and the thorough sweep mode attempts to schedule a real interpolation pod, which fails on GPU-less or capacity-limited clusters because the interpolation step is invoked without a Mocker guard. Carried over from v1.3.0.
  • Decode Sweep Returns No Results: During DGDR profiling with the vLLM backend, thorough decode candidates retain the --disaggregation-mode decode argument after their prefill peer is deleted, so the model never registers and the decode sweep always yields no results, reproduced in six independent test cases across three model sizes on H200 with 1.4.0-rc.4. The decode branch of the Profiler never strips that argument, which became a failure once the prefill peer is removed ahead of the decode sweep. Note that this affects the v1 Profiler code path, which is being replaced by the new Profiler and is out of scope for this release. No workaround is available yet.
  • Model Sizing Fails Non-Llama Configs: Automatic model sizing fails and the caller raises an error for models whose HuggingFace config uses non-Llama key names, affecting seven architecture families: GPT-2, OPT, BLOOM, GPT-J, GPT-Neo, Falcon, and CodeGen. The naive fallback estimator in the raw-config sizing path reads only hidden_size, num_hidden_layers, vocab_size, and intermediate_size, so configs using the aliases n_embd, n_layer, ffn_dim, and n_inner produce a null weight size and the sizing call fails. No workaround is available yet.

Kubernetes & Operator

  • Thorough Mode Exceeds GPU Capacity: When a DynamoGraphDeploymentRequest for a Mixture-of-Experts model is run in thorough mode, the configuration enumeration can emit candidate DynamoGraphDeployments whose combined tensor, expert, and data parallel sizes require more GPUs than the target cluster provides, so the recommended deployments fail to schedule. Carried over from v1.3.0.

KV Block Manager

  • Deprecated KVBM Example Fails Deployment: Deploying examples/backends/vllm/deploy/agg_kvbm.yaml does not come up: the KVBM-backed aggregated worker stalls during initialization and never reaches a serving state. KVBM is deprecated in this release, so this is not being fixed as a runtime defect, and the manifest is still shipped in the examples tree even though it no longer deploys. No workaround is available yet.

Observability & Simulation

  • Responses Endpoint Trace Mislabeling: Request-trace payload records emitted for /v1/responses requests are attributed to the wrong endpoint, appearing as openai.chat_completion, and the recorded request snapshot is not pristine, with the streaming flag reflecting the translated request rather than what the client sent. Payload capture was scoped to /v1/chat/completions only, and /v1/responses requests are translated into chat completion requests before reaching the preprocessor capture point, so the records inherit both the chat endpoint label and the post-translation streaming flag. A partial fix that labels records by route is in review; the distorted streaming flag in the request snapshot is still outstanding. No workaround is available yet.

Looking Ahead

Standalone Router & Runtime-Free EPP

The standalone Router work continues toward a runtime-free selection service and inference-gateway EPP that deploy without the full Dynamo runtime, extending the endpoint-scoped transport and reservation-replay foundations shipped in this release. The Rust ext-proc EPP becomes the default gateway integration in an upcoming release, replacing the Go-based Endpoint Picker.

Custom Routing Strategies

Router scoring and selection open up behind native WorkerScore and WorkerPick policies already merged on main, so deployments can bring a routing strategy tuned to their workload. The trace-capture and simulation tooling shipped in this release gives those strategies a place to be validated against recorded production traffic before they route real requests.

Cross-Datacenter Routing

Cross-datacenter prefix routing shipped experimental in Dynamo v1.4.0. The work ahead hardens the datacenter-scoped KV relay and sequenced indexing toward production use, and workers move toward advertising their own router configuration so fleets negotiate routing behavior without frontend flag coordination.

Voice & Custom Multimodal Encoders

Multimodal work extends toward custom encoder support, frontend-decoded encode-prefill-decode pipelines, and realtime voice serving, building on the unified vLLM multimodal serving and transcription infrastructure in this release.

Assets 2
Loading