Skip to content

Dynamo v1.6.0

Latest
Latest

Choose a tag to compare

View all tags
@dagil-nvidia dagil-nvidia released this 22 Oct 22:31
v1.6.0
701da25
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Dynamo v1.6.0 - Release Notes

Dynamo v1.6.0 is the 19th feature release of the open-source distributed inference platform, spanning 654 merged PRs from 148 contributors. Dynamo Router gains pluggable flow control: request classifiers can defer, reorder, or steer requests, and every selection path now runs on one shared core with scoring knobs consolidated into one policy file. On Kubernetes, LPX brings LPU-backed inference to DynamoGraphDeployments, multinode components take role-specific pod templates, and Dynamo Snapshot backs engine failover. Dynamo Planner reconfigures GPU budgets at runtime, and the engines move to SGLang v0.5.21, TensorRT-LLM v1.3.0rc29, and vLLM v0.31.0.

Summary

Dynamo Router: Pluggable Flow Control & Unified Selection

Dynamo Router opens request intake to plugins. A request-classifier hook lets a plugin defer a request, override validated scheduling inputs, or suggest a worker without overriding explicit worker pins (#14123, #14177). Every request now flows through the policy queue, which orders requests by deadline (#14176). ThunderAgent ships as a builtin plugin (#15164). Dynamo Frontend, the standalone selection service, and the Gateway API endpoint picker (EPP) now share one worker-selection core (#14570). Scoring knobs move into --router-policy-config, and the older flags are deprecated (#15202). An opt-in session prefix index tracks session-to-block lineage for agentic workloads (#13807), and a versioned gRPC relay exposes KV pool state to consumers outside the data center (#14379).

Dynamo Frontend: Structured Parsing, Reranking & Error Fidelity

Dynamo Frontend adds opt-in V2 tool and reasoning parsers for eight model families, selected with DYN_PARSER_VERSION (#14950), with structural tags on by default for supported parsers (#15916). DeepSeek V4.1 gains reasoning and tool-call support (#15030). An OpenAI-compatible /v1/rerank endpoint serves SGLang cross-encoder models (#14032), and a root-level thinking_token_budget field caps reasoning tokens on SGLang and vLLM (#12624, #15120). Errors map to protocol-correct OpenAI and Anthropic responses (#14396, #14002). The Frontend HTTP server now supports mutual TLS (#14173).

Multimodal & Omni: Video-Aware Routing & New Speech Models

Video-aware KV routing extends from vLLM to SGLang for Qwen3-VL and Qwen3.5 (#15014), and Nemotron 3 Nano Omni gains exact image and video routing (#14564, #14653). vLLM-Omni adds Nemotron Audex text-to-speech (#12788) and MiniMax-H3 text-to-video-and-audio generation (#13589, #14702). External encoders can reuse Dynamo's image extraction and prompt reconstruction through a public handoff API (#13293). An opt-in shared cache of encoded images in Redis Cluster or Dragonfly saves repeated media fetches (#14676).

Kubernetes: LPX, Role Pod Templates & Snapshot Failover

LPX components bring LPU-backed inference to DynamoGraphDeployments, with partition selection for hybrid builds and rolling build updates on Grove (#15062, #15563, #15607). Multinode worker, prefill, and decode components can take separate leader and worker pod templates (#14468), and the Operator injects portable DYNAMO_RANK and DYNAMO_LEADER_ADDRESS variables (#15374). Dynamo Snapshot backs active and standby engine failover for SGLang and vLLM (#15924) and captures CUDA shared memory by default (#15791). The GPU Memory Service adds Intel XPU support (#12484). The Rust Endpoint Picker scales past one replica (#14758), and GA Helm charts publish to the NGC OCI registry (#13657).

Dynamo Planner & Performance Modeling

Dynamo Planner reconfigures endpoint floors and GPU budget bounds at runtime through /v1/min-endpoints (#14218, #14231), caps how many replicas one throughput decision changes (#14296), and keeps pending capacity when demand falls while a worker is still starting (#14671). An experimental Planner Gym compares Planner, KEDA-style, reactive, and static autoscaling policies against replayed or live traffic (#15576). Performance-model consumers across the Planner, Profiler, Router, Frontend, Mocker, and replay move to AISimulate's canonical configuration (#15063). Replay gains typed agentic traces (#13586), Codex and Claude trace export (#15601), and host-memory KV offload in the Mocker (#15647).

Observability & Transport

Dynamo and engine metrics can export over OTLP to an OpenTelemetry Collector (#14071), and opt-in lifecycle tracing models each request from arrival to completion (#14101). The runtime warns on sustained Tokio queue pressure (#13027). An opt-in batched QUIC response plane joins the default TCP path (#11996), and IPv6-only fixes cover listeners, ZMQ, SGLang KV events, and vLLM multinode (#15383).

Open-Source Contributions

Between v1.5.1 and v1.6.0 the project merged 654 PRs from 148 contributors, 58 of them from outside NVIDIA. Thank you to the external community contributors in this release (organization identified through the commit-author email domain or the public GitHub profile):

New Contributors: welcome to the 45 first-time contributors in this release:

  • @Victor49152 matched the Qwen vision utilities to the DeepStack vision embedding size (#11244).
  • @Stelath preserved raw reasoning output in the Responses API (#11604).
  • @RealNicolasBourbaki propagated the worker ID into snapshot-restored vLLM engine cores (#12545).
  • @gitover22 published KV cache gauges from vLLM prefill workers (#12559).
  • @aswinvisva forwarded mm_processor_kwargs to the TensorRT-LLM engine and enabled NVDEC decoding of data: URI videos (#13856, #15219).
  • @ShaneGZhu made the ThunderAgent cost model aware of data-parallel ranks (#14000).
  • @aylonsk bounded the KV router state-agent discovery wait at worker startup and four more Planner and router fixes (#14003).
  • @Champollion9012 recorded the router prefill start only for phases that run prefill (#14019).
  • @sbak5 added per-request lifecycle instrumentation to the runtime (#14101).
  • @ZenAlexa rejected oversized TCP request frames before send (#14110).
  • @Vinya567 added FP16 and BF16 tensor protocol variants and Triton /v1/classify support (#14150, #15396).
  • @cyclinder preserved forced SGLang tool guidance in the frontend (#14179).
  • @trevor-m fixed SGLang compatibility for get_model_config and use_mla_backend (#14234).
  • @SeungminHeo added A.X-K2 H200 vLLM recipes with a disaggregated benchmark (#14318).
  • @Efeselimsurekli documented that the operator compilationCache is version-specific (#14381).
  • @git-jxj rejected unsupported tools in the Responses API before dispatch (#14440).
  • @diffractiongoo shared the process runtime with model fetching (#14534).
  • @nvjullin forwarded session_id and parent_session_id to the SGLang engine (#14611).
  • @tianhaox fixed the decode warm-up for hybrid KDA and Mamba models, validated on GLM-5.3-Flash (#14614).
  • @jacksonrnewhouse drained SSE streams after graceful generation stops (#14645).
  • @zhaohuizh shared prompt tokens across request clones (#14701).
  • @Arsene12358 settled vLLM KV warm-up stages through a per-step round on every attention data-parallel rank (#14728).
  • @Prudctual carried typed pre-stream errors on the QUIC response plane and passed response_format into vLLM-Omni video inputs (#14789, #14844).
  • @Sumit884-byte sent inline media once on the TCP request plane and prepended the KServe batch dimension for Triton metadata (#14801, #14996).
  • @bojiang-li used the root cache for container user overrides with a group and three more operator, cache, and multimodal fixes (#14820).
  • @host452b enforced media download limits (#15005).
  • @andrewpaprotsky added the operator LPX integration and four follow-up LPX changes (#15062).
  • @Atharva-Kanherkar preserved URL path case in the multimodal image cache key (#15101).
  • @sherlockwu gated SGLang auto tool-call grammars on reasoning and forwarded tool-message media in Python chat processors (#15227, #15237).
  • @livingshade added SGLang EAGLE3 speculative decoding tests and a guide (#15251, #15268).
  • @c2w-sea declared the ModelExpress startup weight version for vLLM (#15252).
  • @junuxyz moved vLLM runtime configurations, examples, and recipes to explicit NIXL roles (#15256, #15366).
  • @surajssd broke a KV router request-lease ownership cycle and retried NATS connections at startup (#15258, #15269).
  • @JasperChan929 charged and gated explicit hard pins at ThunderAgent admission (#15290).
  • @dundysm kept multi_modal_data on non-Qwen vLLM decode for image KV identity (#15371).
  • @charan-rathore placed the vLLM-Omni video negative_prompt in the prompt entry (#15402).
  • @awesome-pro generated the nightly release ledger for the docs (#15410).
  • @sunil1511 added shared NGC model fetching with SGLang and vLLM integration (#15441).
  • @neilkg passed the checkpoint tokenizer to SGLang parsers and eight more frontend and SGLang fidelity fixes (#15509).
  • @kathleenfrench dropped the stale dynamoGraphDeploymentRef from LPX bindings (#15522).
  • @geelen let recipe cards without logos use the full header width (#15546).
  • @joshhay95 selected GPU-local partitions for hybrid and HX LPX builds (#15563, #15706).
  • @ericcurtin rejected vLLM-Omni TTS tasks the checkpoint cannot serve (#15591).
  • @mattwittwer rejected non-numeric logit_bias keys in the frontend (#16322).
  • @Jianhong-Zhang upgraded SGLang XPU to v0.5.19 on oneAPI 2026.0 (#16684).

Dynamo Special Interest Groups (SIGs)

Dynamo now has six Special Interest Groups (SIGs). Each SIG runs a public meeting, reviews the Dynamo Enhancement Proposals (DEPs) and PRs in its area, and sets that area's roadmap. Anyone in the community can join, bring a PR or DEP, and be involved in the technical direction of Dynamo.

  • #sig-router: KV-aware routing, Global Router, and Gateway API routing.
  • #sig-engines: SGLang, TensorRT-LLM, vLLM, TokenSpeed, and Dynamo Sidecar.
  • #sig-frontend: Frontend APIs, tool calling, and parsers.
  • #sig-deploy: Kubernetes Operator, Helm charts, DRA, and Grove.
  • #sig-agents: Agentic inference, agent hints, and request tracing.
  • #sig-rl: RL integration, rollouts, and weight updates.

How to Join

  1. Join the ai-dynamo Slack and find the channel.
  2. Subscribe to the community calendar for meeting times.

For how to contribute code, docs, and recipes, see the Contribution Guide.

Key Dependencies

Dynamo SGLang TensorRT-LLM vLLM NIXL UCX
v1.6.0 v0.5.21 v1.3.0rc29 v0.31.0 v1.5.0 (Frontend, SGLang) / v1.3.1 (TensorRT-LLM) / v1.4.1 (vLLM) v1.21.0

CUDA Variants

SGLang TensorRT-LLM vLLM
13.0 13.1 13.0

Dynamo Ecosystem

AIPerf AISimulate Grove ModelExpress
v0.13.0 v0.13.0 v0.1.0-alpha.14 v0.7.0

Breaking Changes

Notable Behavioral Changes

Kubernetes and Operator

  • vLLM worker services use short names (VllmWorker / VllmPrefillWorker / VllmDecodeWorker -> worker / prefill / decode), matching the SGLang, TensorRT-LLM, and Mocker backends and keeping generated names under Grove's 45-character limit (#9848).

    Migrate: Update DynamoGraphDeployment manifests, overrides, scripts, dashboards, and alerts that reference the old vLLM service names. Pod names, Kubernetes Service names, and the dynamo_component metric label change with them.

  • Dynamo Operator no longer adopts resources it does not own (#13990). An existing object that shares a name with an Operator-managed child but has no controller owner, or a different one, is no longer adopted, updated, or deleted; the deployment reports an ownership conflict instead.

    Migrate: Before upgrading, delete or rename any pre-existing Deployments, Services, or other objects that collide by name with Operator-managed children.

  • Multinode settings are validated more strictly (#14468, #14533). multinode is accepted only on worker, prefill, and decode components, roles[] takes at most two entries, and a declared role must carry a complete PodTemplate that owns its launch command.

    Migrate: Remove multinode from frontend, Planner, EPP, and other non-engine components, trim each roles list to a leader and a worker, and give every declared role a full PodTemplate.

  • Init container named runtime is reserved (#14927). Declaring it with restartPolicy: Always switches the component into Dynamo native sidecar mode, and the runtime version then resolves from that container's image.

    Migrate: Rename any unrelated init container called runtime before upgrading. In sidecar mode, make sure the runtime image carries a semantic-version tag or set runtimeVersionOverride.

  • Grove v0.1.0-alpha.14 is required for new multinode deployments created by Dynamo Operator v1.6.0, and the Operator now owns the DYNAMO_RANK and DYNAMO_LEADER_ADDRESS environment variables on those components (#15374, #15704). Existing deployments keep their current environment and command lines.

    Migrate: Upgrade an externally managed Grove and its CRDs to v0.1.0-alpha.14 before creating new multinode deployments, and rename any user-defined DYNAMO_RANK or DYNAMO_LEADER_ADDRESS variables in component pod templates.

  • Grove Coherent updates are opt-in (#15903). Without the nvidia.com/grove-update-strategy: Coherent annotation, every DynamoGraphDeployment uses Grove's RollingRecreate default.

    Migrate: Set the nvidia.com/grove-update-strategy: Coherent annotation on deployments that should keep coherent rollouts.

  • Operator-managed namespace discovery is strict for Dynamo v1.6.0 runtimes (#15561). A frontend in deployment foo no longer discovers workers from deployment foo-bar; it still admits foo, its foo-<8 hex> worker generations, and foo-legacy.

    Migrate: If you relied on discovery across deployments that share a name prefix, rename the deployments or set DYN_NAMESPACE_PREFIX_STRICT=false on the affected frontend and EPP containers.

  • GB200 nodes report GPU SKU gb200 instead of gb200_sxm (#15711).

    Migrate: Update automation and profiling configuration that reads the Operator-populated gpuSku for GB200 nodes.

  • Snapshot enables CUDA shared-memory capture by default (#15791). Automatic checkpoint jobs set nvidia.com/cuda-shared-memory-support: enabled, the checkpoint target container must set command, existing checkpoints are recaptured once because the compatibility hash now includes the annotation, and snapshot mode no longer forces the NCCL and TensorRT-LLM zero-copy variables off.

    Migrate: Add an explicit command to the checkpoint target container, or set nvidia.com/cuda-shared-memory-support: disabled on the DynamoGraphDeployment to keep the previous behavior. Set NCCL_CUMEM_ENABLE, NCCL_NVLS_ENABLE, or TLLM_NCCL_SYMMETRIC_ZERO_COPY to 0 yourself if you depended on snapshot mode disabling them.

  • The platform chart blocks the Dynamo Operator update when an externally managed Grove is incompatible or still rolling out (#16891). A preflight Job checks the Grove controller before the Operator installs or upgrades, and it pulls docker.io/library/python:3.12-slim by default.

    Migrate: Upgrade the external Grove controller and its CRDs to v0.1.0-alpha.14 before upgrading the platform chart, and override dynamo-operator.grovePreflight.image for internal registries. Set dynamo-operator.grovePreflight.enabled=false only when a separate compatibility gate exists.

Frontend and API

  • Remaining OpenAI routes return JSON errors (#14002). Malformed requests to embeddings, image generation and edits, and audio now receive the standard JSON error envelope instead of a plain-text body.

    Migrate: Update clients that parse plain-text error bodies from these endpoints.

  • Non-streaming /v1/responses no longer infers tool calls from assistant text (#14846). Only structured tool calls from the backend become function-call output items; <tool_call> markup in plain text stays text.

    Migrate: Enable a tool-call parser on the backend (for example --dyn-tool-call-parser hermes) so tool calls arrive structured.

  • Forced tool choice wins over response_format on SGLang (#14179). tool_choice="required" or a named tool now constrains output to a tool call, and requests that cannot satisfy a forced tool choice, such as required with no tools, are rejected.

    Migrate: Send a non-empty tools array with tool_choice="required", and use tool_choice="auto" when response_format should shape the output.

  • Boundary-only reasoning parsers fail at startup with the vLLM chat processor (#14991). Parsers such as openai_gptoss previously started and then failed each request with HTTP 500.

    Migrate: Remove or replace boundary-only reasoning parsers when using --dyn-chat-processor vllm.

  • Anthropic billing preambles are stripped by default (#15826). The x-anthropic-billing-header preamble that Claude Code adds is removed from requests unless disabled, reducing tokens and prompt-cache variation.

    Migrate: Set DYN_STRIP_ANTHROPIC_PREAMBLE=0 or pass --no-strip-anthropic-preamble to keep preambles intact.

  • Structural tags are on by default for supported parsers (#15916). DYN_ENABLE_STRUCTURAL_TAG now defaults to true and DYN_STRUCTURAL_TAG_SCOPE to always, so valid tool schemas that use constructs the grammar backend rejects, such as regex lookahead, can fail requests.

    Migrate: Set DYN_ENABLE_STRUCTURAL_TAG=false if your tool schemas use constructs the grammar backend rejects.

  • /v1/models/<model>/ready returns 503 for a known but unready model (#16943). It previously returned HTTP 200 with ready: false. The diagnostic JSON body is unchanged.

    Migrate: Parse the JSON body on both 200 and 503 responses in diagnostic clients, and treat 503 as unready and retry.

Router and Planner

  • Dynamo Router policy YAML takes precedence over scoring flags (#15202). Parameters under worker_selection.instances[].parameters in --router-policy-config override the deprecated command-line flags and environment variables, and shared-cache hits now earn a default multiplier of 0.5 when shared cache is enabled through YAML. The overlap-query response no longer returns router_credit_blocks; use the raw cache-hit fields.

    Migrate: Audit policy YAML for parameters that now override existing flags, and set shared_cache_multiplier: 0.0 explicitly if shared-cache hits should earn no credit.

  • Router requests always pass through the policy queue (#14176). Uncontended requests no longer bypass the queue, and a policy class with no registered workers returns a no-endpoints (service unavailable) error instead of a queue-rejected error.

    Migrate: Update error handling that matched the queue-rejected error for an empty worker set.

  • Dynamo Planner caps throughput scaling at 8 replicas per decision per component (#14296).

    Migrate: Raise max_throughput_scaling_replicas in the Planner configuration if you rely on larger single-step scaling.

  • Performance-model options renamed from AIConfigurator to AISimulate (--aic-* -> --ais-*, aic_perf_model -> ais_perf_model.roles) across the Planner, Mocker, and replay tools (#15063). Retired DYN_AIC_* environment variables now fail at startup.

    Migrate: Rename --aic-* flags to their --ais-* equivalents, replace DYN_AIC_* environment variables with their AIS equivalents, and move Planner performance-model settings under ais_perf_model.roles. The old CLI spellings are accepted only temporarily.

  • KV routing rejects text-input workers at discovery (#16938). vLLM workers started with --use-vllm-tokenizer, and other text-input workers, are no longer admitted when the effective Router mode requires KV routing.

    Migrate: Remove --use-vllm-tokenizer so the frontend tokenizes, or select a non-KV Router mode for those workers.

Runtime, Transport, and Metrics

  • Invalid DYN_REQUEST_PLANE values fail at startup (#12612). Empty or unrecognized values previously fell back to TCP silently.

    Migrate: Unset DYN_REQUEST_PLANE or set it to tcp or nats.

  • Invalid DYN_EVENT_PLANE values fail at startup (#16712). A misspelled value previously logged a warning and fell back to ZMQ.

    Migrate: Unset DYN_EVENT_PLANE or set it to nats or zmq.

  • Unbracketed IPv6 ZMQ endpoints are rejected (#15383). An address such as tcp://2001:db8::10:5555 now fails fast instead of retrying forever, IPv4 and hostname endpoints bind as IPv4 again, and a :: system-status bind advertises a non-loopback IPv4 address when no non-loopback IPv6 address exists.

    Migrate: Bracket IPv6 hosts in event-plane endpoints (tcp://[2001:db8::10]:5555), and set an explicit advertise host if you relied on ::1.

  • Histogram bucket variables take effect under their documented names (DYN_HISTOGRAM_DYN_METRICS_* -> DYN_METRICS_*) (#14514). The documented DYN_METRICS_<HISTOGRAM>_MIN, _MAX, and _COUNT variables were previously ignored and now change bucket boundaries.

    Migrate: Audit existing DYN_METRICS_* histogram settings.

  • NIXL telemetry settings are validated at startup (#14625). Unrecognized NIXL_TELEMETRY_ENABLE values and invalid ports are rejected, and the default NIXL Prometheus port changes from 19090 to 9090.

    Migrate: Set NIXL_TELEMETRY_ENABLE to a boolean value such as y or n, and set NIXL_TELEMETRY_PROMETHEUS_PORT explicitly if your scrape configuration expects 19090.

Backends

  • --publish-kv-events no longer turns on TensorRT-LLM performance metrics (#12162, #14181). The flag now publishes KV events only; iteration statistics and forward-pass metrics have their own opt-in.

    Migrate: Add --publish-metrics to TensorRT-LLM workers that relied on --publish-kv-events for performance or forward-pass metrics.

  • TensorRT-LLM sidecar speaks the OpenEngine gRPC API (#14349). Engines that serve only the older TensorRT-LLM gRPC service no longer work with the sidecar.

    Migrate: Run a TensorRT-LLM build that serves the OpenEngine (openengine.v1) gRPC API.

  • vLLM-Omni diffusion options no longer carry hardcoded defaults (#14702). --boundary-ratio no longer defaults to 0.875, and the offload, VAE slicing, and VAE tiling switches stay unset unless provided, so each model's own defaults apply.

    Migrate: Pass --boundary-ratio 0.875 or the matching --no-* flags explicitly if you depended on the previous values.

  • SGLang accepts top-logprob requests without an opt-in and validates top_logprobs strictly (#16927). logprobs: true without top_logprobs returns only the sampled token, and a positive top_logprobs without logprobs: true, or a negative, boolean, or non-integer value, returns HTTP 400.

    Migrate: Send logprobs: true with any positive top_logprobs, set top_logprobs explicitly to receive alternatives, and remove DYN_SGL_ALLOW_TOP_LOGPROBS from deployments.

Deprecated and Removed

  • Removed the TensorRT-LLM AutoDeploy backend, which TensorRT-LLM v1.3.0rc29 deleted upstream with no replacement (#15418). PyTorch is now the only TensorRT-LLM backend, and backend: _autodeploy raises a clear error.

    Migrate: Remove backend: _autodeploy from TensorRT-LLM extra engine arguments.

  • Removed the dynamo.mocker._worker compatibility module that v1.5.x retained for existing configurations (#14209).

    Migrate: Launch live Mocker workers with python3 -m dynamo.mocker.

  • Deprecated DYN_ENABLE_EXPERIMENTAL_PARSERS_V2 in favor of DYN_PARSER_VERSION (#14950). The old variable still works with a warning, and a value that conflicts with DYN_PARSER_VERSION stops startup.

    Migrate: Replace DYN_ENABLE_EXPERIMENTAL_PARSERS_V2=1 with DYN_PARSER_VERSION=2, or leave both unset to use each model family's default.

  • Deprecated component minAvailable in favor of native provider overrides, value.spec.minAvailable for standalone cliques and value.minAvailable for scaling groups (#15903).

    Migrate: Move component minAvailable settings into the matching provider override.

  • Deprecated the Router scoring and selection flags in favor of --router-policy-config, including the six scoring flags, --load-aware, the prefill and decode selection and queue-threshold flags, and their environment variables such as DYN_SHARED_CACHE_MULTIPLIER (#15202). They are slated for removal in Dynamo v1.7.0.

    Migrate: Move these settings into the policy YAML passed to --router-policy-config.

  • Deprecated --publish-events-and-metrics for TensorRT-LLM workers in favor of --publish-kv-events and --publish-metrics (#12162). The combined flag works for one more release.

    Migrate: Replace it with one or both of the split flags.

  • Deprecated the legacy worker names in DynamoGraphDeploymentRequest overrides (VllmWorker, VllmPrefillWorker, VllmDecodeWorker, and the matching SGLang and TensorRT-LLM names) (#9848).

    Migrate: Target worker, prefill, and decode in DGDR overrides.

  • Deprecated the gb200_sxm GPU SKU in DGDR v1beta1 in favor of gb200 (#15711).

    Migrate: Set spec.hardware.gpuSku: gb200.

  • Deprecated the planner.powerAwareness.enabled Helm value (#14671). The Planner service account now always receives namespace-level pods/list permission, and power awareness is controlled by enable_power_awareness in the Planner configuration.

    Migrate: Remove the value from your Helm values file.

  • Deprecated the doubled DYN_HISTOGRAM_DYN_METRICS_* variable names, which still work but log a warning (#14514).

    Migrate: Use the documented DYN_METRICS_* names.

Features & Improvements

Frontend & Agents

  • Responses Tool Selection Hardening: Rejected unsupported tools and tool_choice variants on /v1/responses with an actionable HTTP 400 instead of dropping them silently (#14440), supported function-only tool_choice.allowed_tools (#14990), removed text-based tool call inference from non-streaming responses (#14846), and restored original function names and namespaces (#15401).
  • Parsers V2 Structural Tags: Added opt-in V2 parsers for GLM, DeepSeek V4/V4.1, Kimi K2/K3, Gemma 4, Muse, and Qwen3 (#14950), configured through --dyn-structural-tag (#16217). Structural tags are on by default for supported parsers (#15916), and vLLM tool-call tags can start inside reasoning (#16220).
  • KServe Tensor Protocol Fidelity: Added half-precision Float16 and BFloat16 KServe tensor variants (#14150), reported ModelMetadata datatypes by KServe v2 wire names (#14957), and translated Triton ModelConfig.dims to KServe shapes (#14996).
  • Semantic Error HTTP Mapping: Centralized semantic error mapping to HTTP status codes in Dynamo Frontend for protocol-correct OpenAI and Anthropic failures without internal diagnostics (#14396), and extended the JSON error envelope to the remaining OpenAI routes (#14002).
  • DeepSeek V4.1 Frontend Support: Added mainline frontend support for DeepSeek V4.1 Flash reasoning and tool calling, enabled when workers pass both --dyn-tool-call-parser deepseek_v41 and --dyn-reasoning-parser deepseek_v41 (#15030), with native V4.1 image inputs and tool-result image ordering aligned to the rendered prompt by consuming the selected formatter's media order from dynamo-renderer (#15097).
  • SSE Streaming Lifecycle Hardening: Reworked SSE streams to end on a hard kill, not a graceful stop, so delayed events still finish (#14645), and made streaming handlers wait for the first backend event so an exception before output returns the non-streaming HTTP status (#14662).
  • Tokenizer Cache Lifecycle Hardening: Replaced the process-global tokenizer pool with a weak-reference registry that shuts down each tokenizer's thread pool once collected (#13368), and turned DYN_TOKENIZER_CACHE_BYTES into one combined budget per process, keeping the 64 MiB default (#15387).
  • Root-Level Thinking Token Budget: Added a root-level thinking_token_budget field to chat completion and Responses requests (#12624) and carried it into SGLang, which requires strict-thinking mode and a reasoning parser (#15120).
  • SGLang Cross-Encoder Reranking Endpoint: Added an OpenAI-compatible POST /v1/rerank endpoint for SGLang text-only cross-encoder models with a dedicated --rerank-worker mode and support for top_n (#14032), and fixed scheduler crashes in rerank workers serving XLM-RoBERTa cross-encoders (#16286).
  • Frontend HTTP Mutual TLS: Added a --tls-client-ca-cert-path flag and DYN_TLS_CLIENT_CA_CERT_PATH environment variable to the frontend HTTP server, requiring a CA-signed client certificate when set, with validation that rejects a client CA supplied without both a server certificate and key. Server-only TLS behavior remained unchanged (#14173).
  • Opt-In Jemalloc Preload: Added an opt-in jemalloc allocator preload for python -m dynamo.frontend, activated by DYN_FRONTEND_JEMALLOC=1, that continues with the default allocator when the library is missing (#14678), with libjemalloc2 installed in the SGLang, TensorRT-LLM, and vLLM runtime images so the switch takes effect (#13101) and documentation on bounding frontend host memory growth (#13099).
  • Typed Chunk Metrics Path: Carried chunk metrics in the typed llm_metrics field instead of annotation comments, reducing mean inter-token latency by 6.21% with a 5.57% gain in output token throughput in a local A/B run (#15115).
  • Opaque Agent Header Preservation: Replaced the agent-compaction field with an agent_headers map of the x-claude-code-* and x-codex-* header families, so new coding-agent hints no longer require an envelope schema change (#15367).
  • Raw Completion Stop Preservation: Added an optional no_stop_trim field to /v1/completions that retains matched stop strings and stop/EOS token text so callers can see terminal markers, defaulting to false (#15519).

Scheduling

  • Session Prefix Indexing and Lineage: Added an opt-in session prefix index (--enable-session-prefix-index) that tracks the KV blocks each session has built per worker and DP rank from vLLM KV events (#13807, #13806), hardened worker-removal handling and capped session identifiers at 256 bytes (#15362), and grouped subagent requests under their parent session so they land on one worker (#14521).
  • Standalone EPP Routing Enablement: Passed expected_output_tokens and policy_class from the standalone endpoint picker (EPP) into its selection service (#13595), routed runtime EPP requests by the x-dynamo-meta-policy-class header (#14480), added cache namespace and salt resolution (#13714) and sglang-renderer tokenizer support (#14501), and made native EPP atomically reserve KV-routed prefill capacity (#13042).
  • Pluggable Router Flow Control: Added a pluggable request-classifier hook to the KV Router that can defer a request, override scheduling inputs, or suggest a worker without overriding explicit pins (#14123), with a deadline-ordered policy queue for every request (#14176), async classifier lifecycle events (#14177), and one plugin registration catalog (#14149). Skipped capacity checks for policy classes with queueing disabled (#15423).
  • ThunderAgent Builtin Router Plugin: Shipped ThunderAgent as a builtin router plugin enabled through --router-policy-config (#15164), made its cost model DP-rank aware so SGLang data-parallel attention workers no longer fail program-lifecycle requests (#14000), and made it honor the Router's pinned worker and check that worker's live capacity (#15290).
  • KV Router Slot Tracker Recovery: Cut slot tracker overhead by sharding its locks (#15531), tied standalone replica sync to each request's reservation so peer routers no longer see a re-added request's prefill as complete (#15578), reworked the recovery snapshot cache (#15551), and restored buffer-first gap recovery that keeps the existing index on a KV event gap (#14627).
  • Planner GPU Budget Enforcement: Added runtime reconfiguration of endpoint floors and min_gpu_budget/max_gpu_budget bounds through PATCH /v1/min-endpoints (#14218), treated min_gpu_budget as a scale-down guard (#14231), and preserved the decode replica count when a raised GPU floor would reduce it (#15466). Applied min_endpoint in single-pool load scaling (#15530) and distinguished startup from drain and rollout so falling demand cannot cancel pending capacity or drain serving workers (#14671).
  • Planner Gym Autoscaler Evaluation: Added an experimental Planner Gym under gyms/planner-gym that compares Planner, KEDA/HPA-style, reactive, and static autoscaling policies (#15576), resolved its defaults through the production Planner configuration (#15678), and removed stale offline replay restrictions so Planner policies run on agentic Mooncake and Dynamo or Weka traces (#15435).
  • Phase-Aware Prefill Accounting: Gated prefill-start recording off for decode-only requests (#14019), moved SGLang prefill completion to after the background prefill stream ends (#14968), and fixed output-block undercounting across block boundaries and batched those updates (#14613, #15448).
  • Direct ZMQ Subscriber Multiplexing: Added opt-in fan-in via DYN_ROUTER_ZMQ_ENDPOINTS_PER_SUB so direct-ZMQ subscribers share one SUB socket across many publisher endpoints (#14219), and coalesced pending KV load metrics (#15587).
  • Standalone Indexer Lifecycle Delegates: Added ownership callbacks to the standalone KV indexer that notify consumers when a block hash gains its first owner or its last owner releases it (#14945), made cleanup safe for absent blocks and backpressure (#14652), and bounded the KV state-agent discovery wait at vLLM startup (#14003).
  • Unified Selection Core: Unified Dynamo Frontend, standalone selection service, and EPP worker selection on one shared core (#14570) and reduced scheduler and dispatch overhead for large worker pools (#14666).
  • TokenSpeed Disaggregation with KV Routing: Added prefill/decode disaggregation for the TokenSpeed backend over Mooncake and published native KV-cache events, so the Router can send prompts to the prefill worker already holding their prefix. Included bootstrap validation and transfer-error handling (#9237).
  • Native KV Cache Events: Added a native TensorRT-LLM KV cache event mode that subscribes directly to each attention-DP rank's ZMQ stream, bypassing the legacy polling and republishing path (#12167).
  • Throughput Scaling Replica Cap: Added the max_throughput_scaling_replicas Planner configuration option, defaulting to 8, and applied it per component to both throughput scale-up and scale-down targets, with capped targets fitted against GPU and power ceilings while preserving explicit endpoint and out-of-band budget recovery (#14296).
  • Relay WAN gRPC Transport: Added a dynamo.kvrelay.v1 Protobuf contract and WAN adapter so consumers outside a data center can subscribe to cache locality, readiness, and KV occupancy streams through an opt-in listener (#14379).
  • Router KV Hints Envelope: Passed Router-produced KV hints to vLLM through vLLM's dedicated KV-hints request argument, keeping hint data separate from the KV connector's kv_transfer_params on token, text, and prefill paths. In prefill/decode deployments the hint is applied once on the prefill worker, and decode handoff requests no longer replay it (#14890).
  • Centralized Router Policy Configuration: Moved Router scoring knobs into --router-policy-config with a router section for shared cache settings, deprecated the six scoring flags and --load-aware while their behavior remains available for migration, and removed router_credit_blocks from the overlap-query response (#15202). Added a soft-pin-repin policy example for soft session-affinity rebinding (#14152).
  • Worker KV Reuse Telemetry: Added router metrics for KV cache reuse per routing attempt across SGLang, TensorRT-LLM, and vLLM, labeled by phase and model, so operators can compare routed prompt tokens against engine cache hits (#15443).

KV Block Manager

  • File-Based Relay Source Mode: Added a --sources-file option to KV DC Relay that selects Dynamo namespaces from a versioned JSON file with live reload, and an /engine/state route reporting readiness (#14807).

Multimodal & Diffusion

  • Nemotron Omni Multimodal Routing: Added exact multimodal KV routing for Nemotron 3 Nano Omni images by modeling vLLM dynamic-resolution geometry and model-specific delimiters (#14564), and for video by sampling and decoding frames and grouping them into temporal tubelets (#14653).
  • Cascaded Nemotron Speech Example: Added a self-contained deployment example exposing Nemotron ASR, LLM, and TTS endpoints through /v1/realtime transcription and streaming /v1/audio/speech, with a three-GPU DynamoGraphDeployment and a smoke client (#12567), and added a client-driven ASR-to-LLM handoff for speculative prefix-cache warming (#14327).
  • Omni Video Parameter Plumbing: Preserved joint video-and-audio output from vLLM-Omni with per-output frame rate and audio sample rate metadata (#13707), cloned sampling parameters so request-specific video controls do not leak across requests (#13708), and forwarded response_format so b64_json is no longer dropped (#14844). Also fixed the HTTP 500 on video requests carrying nvext.negative_prompt and carried the value to the first-stage engine prompt (#15402, #15541).
  • MiniMax-H3 FastH3 Video Enablement: Added a four-B200 aggregated vLLM-Omni launcher for MiniMax-H3 text-to-video-and-audio generation with the four-step Dense FastH3 startup adapter (#13589), then enabled MiniMax-H3 video sparse attention (VSA) on vLLM-Omni with the --fastvideo-vsa-topk option and B200 and GB300 hardware guards (#14702).
  • Shared Encoded-Image Cache Hardening: Added an opt-in shared cache of encoded image bytes in Redis Cluster or Dragonfly, failing open on any cache error (#14676). Fixed case-differing S3 and GCS paths collapsing onto one image cache entry (#15101), made the embedding cache evict only while it needs space (#15497), and stopped the NIXL embedding ring buffer from overlapping a live embedding (#15498).
  • vLLM External Encoder Handoff: Added a public ExternalEncoderHandoff API so external encoders built on VisionEncoderBackend reuse Dynamo's image extraction and prompt reconstruction (#13293), forwarded preprocessed multimodal features to vLLM's gRPC API (#14244), and moved multimodal tensor transfer from pickle to typed msgpack (#14433).
  • Multimodal UUID and Media Deduplication: Extended client-provided multimodal UUID forwarding from images to all modalities for vLLM-based deployments (#14300), and stopped inline data: media from being sent twice on the TCP request plane (#14801).
  • Nemotron Audex Speech Synthesis: Added Nemotron Audex text-to-speech support to the vLLM-Omni backend on the existing /v1/audio/speech endpoint, covering nvidia/Nemotron-Labs-Audex-2B and nvidia/Nemotron-Labs-Audex-30B-A3B. Added cfg_scale to the frontend and worker request models for classifier-free guidance, and fixed cumulative waveform assembly so Audex requests no longer return silent or over-length audio (#12788).
  • SGLang Qwen Video KV Routing: Added video-aware KV cache routing for Qwen models on SGLang, mirroring SGLang's frame sampling in the frontend so repeated video requests hit cached prefix blocks (#15014). Extended the Multimodal KV Routing guide with video identity handling, documenting vLLM support for Qwen3-VL, Qwen3.5, and Nemotron 3 Nano Omni (#14958).

Kubernetes Deployment

  • LPX Integration and Partition Selection: Added the LPX component type for LPU-backed inference across API, controller, runtime, scheduler, Grove, and Helm (#15062), let hybrid XT and HX (LP30) builds run some or all partitions on the Cyborg GPU via lpx.experimental.localPartitions (#15563, #15706), packed LPU-only PropSync partitions into whole-node allocations (#15583), and fixed HX hybrid deployments (#15804).
  • Rolling LPX Build Updates: Routed compatible lpx.buildId changes through Grove rolling updates so remaining replicas keep serving (#15607), and excluded RollingUpdate from the layout comparison so an unchanged PodCliqueSet is no longer recreated (#15672).
  • Role-Specific Multinode PodTemplate API: Activated the roles[].podTemplate API for multinode worker, prefill, and decode components so leader and worker Pods can come from different complete PodTemplates (#14468, #14544), added an explicit roles[].replicas surface for role cardinality (#14477), and restricted multinode to inference-engine component types (#14533).
  • Snapshot CUDA Shared Memory Capture: Enabled CUDA shared-memory Snapshot capture so workers no longer disable NCCL cuMem, NVLS or TensorRT-LLM zero-copy, and existing checkpoints are recaptured because the compatibility hash includes the annotation (#15791), with SGLang, TensorRT-LLM, and vLLM (v0.31.0 and later) releasing only the KV cache before capture instead of moving weights to host memory, which shrinks checkpoints and speeds restores (#16358).
  • Snapshot-Backed Engine Failover: Added snapshot-backed engine failover for SGLang and vLLM, including single-node multi-GPU workers, that replenishes a failed engine without replacing the worker Pod, with unsupported snapshot failover backends rejected at admission (#15924), and propagated the compatibility hash to WaitForCheckpoint references (#16687).
  • Intel XPU GMS Support: Added Intel XPU support to the GPU Memory Service (GMS), using Level-Zero interop for inter-process memory sharing, a C++ extension for tensor reconstruction, and SGLang and vLLM integration paths. XPU support runs on native torch.xpu with no IPEX dependency and requires PyTorch 2.13 or newer (#12484).
  • GA Helm OCI Publishing: Added a push of GA Helm charts to the NGC OCI registry alongside the existing ChartMuseum push, so consumers on OCI-only registries such as ACR, ECR, and Harbor can mirror released charts (#13657).
  • KVCR Resiliency Deployment Example: Added a two-node vLLM KV Cache Reuse (KVCR) example over RDMA. The memory-service sidecar keeps the KVCR Guard available across engine restarts (#14695).
  • EPP Multi-Replica Support: Removed the admission rule that restricted the Dynamo-mode endpoint picker (EPP) to exactly one replica, so native Rust EPP deployments can scale out and tolerate pod loss like the Frontend-hosted Router, while legacy Go EPP deployments still require a single replica. DYN_ROUTER_REPLICA_SYNC remained off by default, matching the Frontend behavior (#14758).
  • Portable Multinode Topology Variables: Added DYNAMO_RANK and DYNAMO_LEADER_ADDRESS to new multinode components so SGLang and vLLM commands no longer depend on Grove or LeaderWorkerSet syntax. Injection is gated on the operator origin version, so existing deployments keep their current environment and command lines, and Grove was upgraded to v0.1.0-alpha.14 (#15374, #15704).

Dynamo Sidecar

Dynamo Sidecar remains experimental in Dynamo v1.6.0.

  • vLLM Sidecar LoRA Lifecycle: Added dynamic LoRA load, unload, and per-request adapter selection to the vLLM sidecar, with local, Hugging Face, and S3 adapters and no silent base-model fallback for unavailable adapters (#13068). Added aggregated and NIXL prefill/decode launchers for local LoRA serving (#14574).
  • TensorRT-LLM OpenEngine gRPC Sidecar: Moved the TensorRT-LLM sidecar onto TensorRT-LLM's OpenEngine (openengine.v1) gRPC API and added disaggregated serving (#14349). Raised the shared sidecar gRPC startup deadline from 300 to 1,800 seconds for large model loads (#14825).
  • vLLM Sidecar EPD Encode Role: Added opt-in Encoder-Prefill-Decode support to the native vLLM gRPC sidecar with E+PD and E+P+D launchers (#13966). Extended the Encode worker to accept video and combined image plus video requests (#15287), and shipped the FFmpeg binaries video decode needs for VP8 and VP9 (#16556).
  • SGLang Sidecar KV Routing: Added a disaggregated KV-aware routing launcher for SGLang sidecars (#13862) and node-local KV event relay for multinode headless sidecars (#14908). Published HiCache capacity so capacity-aware routers no longer treat HiCache workers as device-only (#14313) and an EAGLE flag so KV routers use the engine's bigram hashing (#14374), and moved the prefill bootstrap handoff into a nonterminal response (#14260).
  • Multimodal-Aware vLLM KV Routing: Resolved the image placeholder token for the vLLM KV event publisher, so repeated image requests match cached image blocks instead of only the text prefix. The token resolves once at startup with a 30 second fetch limit, and image requests fall back to text-prefix routing if resolution fails (#13360).
  • Standalone Decode Sidecar Contract: Added a standalone decode sidecar for the inference gateway that validates x-prefiller-host-port metadata so client-supplied prefill metadata cannot reach endpoint selection (#13669).
  • Engine-Primary Dynamo Native Sidecars: Added operator support for separate inference engine and Dynamo runtime containers, selected per component via an init container named runtime, while combined mode remains the default (#14927).
  • Shared Sidecar Transport Flags: Added --discovery-backend, --request-plane, --response-plane, and --event-plane to the SGLang, TensorRT-LLM, and vLLM sidecars, so workers select file, tcp, quic, or zmq directly (#15653).

SGLang

  • Sharded SGLang Worker Gateway: Added --gateway-workers to shard the dynamo.sglang gateway across processes so tokenizer intake and output relay no longer share one GIL (#15078), and fixed a startup AttributeError (#16337).
  • SGLang Cache Salt Routing: Added cache salt support for SGLang text workers so differently salted requests use separate KV cache entries. Non-empty salts are rejected when the installed engine or request type cannot honor them. Upgrade all workers and routers before enabling salted traffic (#14803).

vLLM

  • Generate Endpoint RL Parity: Added reinforcement learning (RL) serving parity for the dynamo.vllm worker in aggregated deployments, adapting native Generate token-in/token-out requests and publishing RL administration routes (#15179). Carried typed /inference/v1/generate controls such as sampling, stop-token IDs, and logprobs into Dynamo's internal request, preserving requests from v1.4 frontends (#14398), with a fix that sent top_k and min_p as unset when clients did not disable them (#15286).
  • LMCacheMPConnector PD Support: Registered a connector adapter so vLLM prefill/decode disaggregation works with LMCacheMPConnector, which previously failed at request time with an unsupported connector error. Wrapper resolution now prefers a connector that produces KV transfer parameters, so MultiConnector setups combining LMCacheMP with NixlConnector still resolve to NIXL (#13258).
  • Incremental Realtime Text Input: Added text-only /v1/realtime sessions to aggregated vLLM workers, with experimental input events so clients can forward ASR text during transcription and commit the final transcript (#14326).

Triton

  • FP16/BF16 Tensor Input Support: Dropped the dynamo.triton guard that rejected FP16 and BF16 inputs (#14835), reinterpreted their values from raw uint16 bit patterns (#16683), and passed BF16 as a raw-byte tensor (#16558).
  • KServe Classification Output Support: Added support for the KServe classification requested-output parameter on the Triton worker and forwarded requested outputs from the frontend (#14618), so the worker returns only the named tensors (#14783).
  • Defunct Triton Example Removal: Removed the out-of-date Triton Inference Server backend example, including its Dockerfile, launch scripts, model repository, and worker implementation. The Dynamo Triton Runtime under components/src/dynamo/triton replaces it (#15170).
  • Triton Classification Endpoint Support: Added OpenAI-compatible POST /v1/classify serving to the Triton backend, opt-in through --task classify with input and output tensor names auto-detected from config.pbtxt and overridable via --classify-input-name and --classify-output-name. The default --task tensor KServe path is unchanged, and text-only input is supported while token-ID variants and encoding_format requests return HTTP 400 (#15396).

Model Loading & LoRA

  • Native LoRA Capacity Reporting: Exposed native vLLM LoRA capacity through shared model registration, so a worker reporting max_loras=4 carries max_gpu_lora_count=4 into discovery (#14754), and corrected LoRA allocation simulation with capacity-bounded adapter residency and churn, hit, and occupancy metrics (#13758).
  • Shared NGC Model Fetching: Added shared NGC model fetching through ModelExpress for SGLang and vLLM, reusing cached models (#15441). Local models served under custom names no longer trigger hf:// metadata fetches on offline hosts (#15964).

Performance Modeling & Replay

  • Typed Agentic Replay Adapters: Added a streaming Weka importer in dynamo-data-gen, lowered Weka and Claude/Codex/Dynamo request traces into versioned Agentic Mooncake v2 rows, and added Dynamo replay entrypoints for typed agentic lanes (#13586).
  • Codex and Claude Trace Export: Added an offline Codex trace exporter for agentic replay alongside the Claude exporter and fixed Claude export so a full local corpus replays. Claude export on a 216 MB project dropped from 49 s to 2.6 s and peak RSS from 2.3 GB to 1.3 GB (#15601).
  • Mocker Host-Memory KV Offload: Wired AISimulate's vLLM host-memory KV offload into the Mocker with tier-aware KV routing, so the Router tracks host-resident blocks and --num-host-blocks alone enables host offload (#15647).
  • AISimulate Canonical Configuration: Moved performance-model consumers across the Planner, Profiler, Router, Frontend, Mocker, and replay to AISimulate's canonical configuration interface, carrying complete configurations through Planner roles and the related CLI consumers (#15063).

Fault Tolerance & Observability

  • Frontend NUMA Affinity Warning: Added a startup check that warns when the frontend's allowed CPUs span multiple NUMA nodes, a likely source of degraded performance (#11184).
  • Tokio Queue Overload Warnings: Added detection of sustained Tokio runnable-queue pressure in the runtime metrics loop, with worker-scaled thresholds, hot-worker detection, hysteresis, and sample-gap checks. Warnings log once per overload episode and repeat at most every five minutes while it continues, and the count is exported as dynamo_tokio_queue_overload_warnings_total for operators (#13027).
  • OTLP Metrics Export: Added a periodic OTLP exporter that ships Dynamo and engine metrics to a collector over gRPC, configured through standard OTEL_* variables and off unless OTEL_METRICS_EXPORTER=otlp is set (#14071).
  • Per-Request Lifecycle Instrumentation: Added opt-in runtime tracing that models each inference request as a finite-state machine, recording state timing and terminal outcome for aggregated and disaggregated serving (#14101).
  • Buffered JSONL FPM Recording: Added --output, --log-metrics, and --flush-interval options to the forward pass metrics (FPM) receiver, writing one JSON record per message to a new file and flushing on an interval (#14511).
  • NVTX Ranges in Wheels: Compiled the runtime's NVTX ranges into published wheels, so the preprocess.*, route.*, transport.*, and detokenize ranges appear in Nsight reports when DYN_ENABLE_RUST_NVTX is set, and removed the dependency on libnvToolsExt.so, which CUDA 12.9 dropped (#15242).

Runtime & Transport

  • Request-Plane Transport Configuration Hardening: Added DYN_EVENT_PLANE_HOST to override the address advertised by direct ZMQ event publishers (#11870). Restricted DYN_REQUEST_PLANE to default to TCP only when absent, so invalid values surface as startup errors (#12612), and defaulted the event-plane context to four libzmq I/O threads with a DYN_ZMQ_IO_THREADS override (#14517).
  • Configurable Etcd Startup Retry: Made the etcd startup retry timeout configurable, keeping 120 seconds as the default and bounding each individual connection attempt by the remaining retry window (#13742).
  • Batched QUIC Response Transport: Added an opt-in batched QUIC response plane selected with --response-plane quic or DYN_RESPONSE_PLANE=quic on each frontend and worker. TCP remains the default (#11996).
  • Packed Int32 Token IDs: Added an opt-in DYN_TOKEN_IDS_AS_BYTES flag that serializes token_ids as a packed int32 blob on request-plane codecs. It must stay off until all msgpack workers run a release containing the new reader (#15326).
  • Unary Endpoint Helper APIs: Added unary endpoint utilities that serve a handler result and collect unary responses and LLM token streams while preserving request context (#12920), then promoted them to dynamo.runtime and dynamo.llm (#15195).
  • Shared Prompt Token Storage: Made copies of a preprocessed request share one prompt-token buffer, reducing clone cost for disaggregated prefill and migration, though direct Rust consumers of token_ids may need adaptation (#14701).
  • Unary Chat Model Publisher: Added UnaryChatModel to dynamo.llm, which registers and serves a Chat model endpoint, plus LLMUnaryClient.connect, which waits for a routable instance (#15552).

Recipes

New Recipes

  • DeepSeek-V4-Pro-0813 vLLM Recipes: The GB200 recipes require the custom vllm-runtime:1.5.0-deepseek-v4-pro-0813-dev.1 container. Added aggregated and disaggregated recipes for deepseek-ai/DeepSeek-V4-Pro-0813 on H200 (#13783), relaxed GB200 prefill liveness probing so vLLM JIT kernel compilation no longer triggers a crashloop (#14735), and replaced the removed use_fp4_indexer_cache with indexer_kv_dtype: mxfp4 (#15427).
  • A.X-K2 vLLM Recipes: Requires the custom vllm-runtime:1.4.1-a.x-k2-post.1 container. Added FP8 H200 targets with 4xTP8 aggregated and 2P2D TP8 disaggregated layouts, KV-aware routing, and EAGLE3 (#14318), and A.X-K2-NVFP4 B200 recipes on vLLM v0.26.0 with TP4 aggregated and disaggregated deployments (#14635).
  • Solar Open2 250B NVFP4 Recipes: Requires the custom vllm-runtime:1.4.1-solar-open2-250b-post.1 container. Added aggregated and disaggregated B200 recipes with a reasoning parser fix for the checkpoint's special tokens (#14376), then repointed benchmark manifests at the public AIPerf v0.12.0 image (#14956).
  • K-EXAONE 2.0 vLLM Recipes: Requires the custom vllm-runtime:1.4.1-k-exaone-2.0-750b-post.1 container. Added vLLM recipes for LGAI-EXAONE/K-EXAONE-2.0-750B-A37B-NVFP4 on B200 with aggregated TP=4 and disaggregated 1P1D variants (#14822), then moved recipe notes into the Fern page and corrected the InfiniBand component, image, and resource settings (#15015).
  • Gemma 4 31B Recipes: Requires the custom tensorrtllm-runtime:1.5.0-gemma-4-31b-dev.1 container. Added Gemma 4 31B deployment recipes for B200, GB200, and H200 GPUs with TensorRT-LLM aggregated configurations, model cache setup, and benchmarking workflows, plus documentation in the recipe catalog and a Google provider filter (#13221).
  • Inkling-NVFP4 GB300 Agentic Recipes: Requires the custom vllm-runtime:1.5.0-inkling-dev.1 container. Added aggregated (2 replicas x TP4) and disaggregated (1P1D, TP4 per role) Dynamo plus vLLM recipes for thinkingmachines/Inkling-NVFP4 on GB300, tuned for an agentic workload with KV-aware routing, MTP-8 speculative decoding, and 1M context, along with an AIPerf trace-replay benchmark Job and Kustomize overlays for generic and AWS RoCE clusters (#13798).
  • Qwen3.6-35B-A3B SGLang Recipes: Added SGLang multi-token prediction deployment recipes for Qwen3.6-35B-A3B on B200, GB200, and H200 GPUs, along with model-download configurations, smoke tests, benchmark results, and catalog and documentation entries (#13958).
  • Qwen3.8-Flash-Next vLLM Recipes: Requires the custom upstream vllm/vllm-openai:qwen38-flash-next image, which installs ai-dynamo at pod startup. Added aggregated (4-GPU TP4, 8-GPU 2xTP4, 12-GPU 3xTP4) and disaggregated 1P1D and 2P1D vLLM deployment recipes for Qwen3.8-Flash-Next on B200, including model-cache setup, AIPerf trace-replay benchmarks, and a docs catalog entry (#14033).
  • DeepSeek-V4.1-Flash SGLang Recipes: Requires the custom sglang-runtime:1.6.0-deepseek-v4.1-flash-dev.1 container. Added day-0 aggregated and disaggregated recipes on 8x GB200 with KV-aware routing on the aggregated target. Both serve up to 1,048,576 tokens of context and neither is benchmarked (#14682).
  • Motif-3 NVFP4 Serving Recipe: Requires the custom vllm-runtime:1.5.0-motif-3-dev.1 container. Added the Motif-3 NVFP4 aggregated serving recipe for vLLM on B200, along with model-cache and benchmark manifests, a Fern documentation page, and a catalog entry that brings the recipe catalog to 25 model families and 125 deployable configurations (#15534).
  • Deployment Recipe Starter Templates: Added a catalog of copyable deployment recipe templates covering aggregated and disaggregated serving for SGLang, TensorRT-LLM, and vLLM, along with Kustomize components for caching, placement, scheduling, probes, networking, and registry credentials, plus contribution and validation guidance for building new recipes (#13724).

Recipe Updates

  • Provider Networking Kustomize Components: Ported the disaggregated provider-networking Kustomize Components for GKE RoCE and InfiniBand to the v1beta1 copy-and-fill scaffold, and extended the recipe validator to enforce the networking-slot ordering and worker-only networking-delta contract, so recipes carry no cluster-specific values in the base (#14142).
  • Kimi Recipe Configuration Fixes: Set DYN_SELF_HOST_METADATA=1 on every Kimi K3 worker role in SGLang and vLLM configurations (#14617), and removed --request-plane nats from the Kimi K2.5 aggregated recipe to use the default TCP transport (#15428).
  • Dynamo Frontend Image for Recipes: Switched the GLM-5.3-Flash recipes' Frontend to the nvcr.io/nvidia/ai-dynamo/dynamo-frontend image, dropping the inline pip install ai-dynamo step (#14229), and applied it to Qwen3.6 SGLang MTP recipes with --dyn-reasoning-parser qwen3 so reasoning lands in message.reasoning_content (#14472).
  • GB300 Max Model Length: Corrected the SGLang chat configuration in the Qwen3 recipe README to document a 262,144-token context length for the GB300 aggregated and disaggregated profiles, while GB200 values remained unchanged (#14048).
  • Shared DGD Environment Merging: Fixed the generated Kustomize OpenAPI schema so environment variables on DynamoGraphDeployment resources merge by name instead of producing duplicate entries, preserving existing values when recipes patch shared environment variables (#15064).
  • Qwen3-32B-FP8 Decode Mode Flag: Added the --disaggregation-mode decode flag to the decode worker in the Qwen3-32B-FP8 vLLM disaggregated recipe, which previously started as an aggregated worker and left the Frontend reporting no model despite a Ready deployment (#16821).
  • Qwen3-VL Worker Startup: Fixed the Qwen3-VL-32B-FP8 vLLM recipes so their worker containers pass command-line arguments correctly; the disaggregated encode and decode workers previously exited cleanly without ever starting, leaving no error explaining the missing workers (#16855).
  • Refreshed Kimi-K3 Recipes: Requires the custom vllm-runtime:1.5.0-kimi-k3-dev.1 or sglang-runtime:1.5.0-kimi-k3-dev.1 container. Expanded the Kimi-K3 recipes with SGLang and vLLM aggregated and disaggregated targets for H200, GB200, and GB300 and added Mooncake trace benchmarking workflows (#12944).

Bug Fixes

Frontend & Agents

  • Anthropic Tool-Call Conversion Fidelity: Fixed Anthropic Messages conversion so buffered tool calls are emitted after an early finish reason (#15129) and the single-call limit from tool_choice.disable_parallel_tool_use is applied (#14605). Preserved tool.strict (#16206), mapped output_config.format to response_format (#16207), and suppressed tool calls whose tool_use.input is not a JSON object (#16214).
  • Anthropic Messages Protocol Corrections: Reworked Anthropic streaming so text after a tool call opens a new block (#15467), reported stop_reason: "stop_sequence" with the matched string (#16417), and stripped the Claude Code billing preamble by default (#15826).
  • SGLang Structured-Output Reasoning Gating: Extended the reasoning gate to structured response formats so JSON no longer returns entirely in reasoning_content (#14465), gave forced tool_choice precedence over response_format (#14179), gated tool_choice="auto" grammars on reasoning to avoid responses that contain neither an answer nor a tool call (#15227), and bypassed the tool parser for ungated answer JSON (#15515).
  • SGLang Chat Processor Parser Fixes: Passed the checkpoint tokenizer to SGLang parsers so reasoning and tool calls are detected when checkpoint delimiters differ from defaults (#15509), retained parser-declared tool closing delimiters during stop handling (#15510), honored explicit skip_special_tokens=false (#14602), and returned chat-template errors as invalid-argument errors (#15511).
  • Truncated Tool-Call Finish Reasons: Preserved finish_reason: "length" at max_tokens rather than a completed tool call (#13791), suppressed the incomplete final tool call in Chat Completions and Anthropic Messages (#13792), and reported streams ending inside GLM-4.7 tool-call blocks as length-limited (#15184).
  • Chat and Completion Logprobs Reporting: Returned prompt logprobs at the root of non-streaming chat completion responses (#13957), emitted completion top_logprobs as token-to-logprob maps (#15521), and mapped chat logprobs onto vLLM sampling parameters instead of rejecting them (#15670).
  • Request Parameter Validation Hardening: Returned HTTP 400 instead of generic 500 for conflicting guided-decoding options (#13790), and aligned stop-sequence limits so excess stop or stop_token_ids entries fail with 400 (#14462). Also rejected an explicit empty tools array (#16231) and logit_bias keys that are not non-negative 32-bit integers (#16322), and validated strict: true function parameter schemas before inference (#16219).
  • Configurable HTTP Listen Backlog: Replaced the frontend HTTP listener's 128-entry queue with a configurable backlog defaulting to 4096, preventing Connection reset by peer during connection bursts (#15080), and restored the TCP call-home listener's 4096 backlog (#15270).
  • Multimodal Media Forwarding Fixes: Fixed SGLang and vLLM chat processors to read media from tool messages too, avoiding image count mismatches (#15237), converted Responses function_call_output image parts instead of rejecting them (#15306), and closed validation for audio, image, and video requests (#15420).
  • Raw Reasoning Output Preservation: Preserved backend reasoning_content in the /v1/responses API, with non-streaming responses returning reasoning_text content instead of summary_text and streaming responses emitting response.reasoning_text.delta plus the corresponding added and done events. Reasoning, text, and tool-call ordering is retained across interleaved reasoning spans, and summary remains as a compatibility fallback (#11604).
  • Scoped Tool-Call Array Bound: Fixed guided decoding so the maxItems=1 constraint applied when parallel_tool_calls=False is limited to tool_choice="required" schemas instead of also hitting named function choices. Array-typed function parameters no longer get truncated to a single element, and the caller's request dictionary is no longer mutated in place (#13242).
  • Speculative Prefill Warmup Lifetime: Bounded prefill warmup tasks to their preprocessor's lifetime with a 60-second warmup deadline after the client response finishes and a process-wide limit of four concurrent speculative preprocessing operations, so stalled backends or never-ending response streams no longer retain request resources indefinitely (#13438).
  • Incremental Tool-Call Streaming: Fixed vLLM preprocessing to stream tool-call deltas as the parser emits them instead of buffering them until the end of a completion, so consumers receive incremental SSE frames (#13975).
  • Client Token ID Bounds: Added client token id validation for nvext.token_data, completion prompt, and embedding input, so an id at or above the vocabulary bound returns HTTP 400 instead of reaching the engine (#14475).
  • SGLang Worker Routing Hints: Fixed the SGLang chat processor to forward backend_instance_id, decode_worker_id, prefill_worker_id, dp_rank, and prefill_dp_rank from nvext into the request routing fields. Requests pinned to a specific worker or data-parallel rank previously reached the Router unpinned and could be sent to a different worker (#14696).
  • Kimi Dynamic Tool Declarations: Added support for Kimi K3 message extensions so tools declared only in system.tools are treated like top-level OpenAI tools across request validation, tool-choice handling, guided decoding, parser definitions, stop-token behavior, and response gating, with dynamic declarations preserved at their original message positions to maintain prompt ordering and prefix-cache behavior (#14755).
  • Null Optional Completion Parameters: Fixed the frontend to treat explicit null values as omitted for recognized passthrough fields such as cache_salt and stop_token_ids, and for stream_options.include_usage and stream_options.continuous_usage_stats on /v1/chat/completions and /v1/completions. OpenAI-compatible clients that serialize unset optional parameters as null no longer receive HTTP 400 validation errors (#14847).
  • Responses API Metadata Echo: Fixed the Responses API to echo request metadata in both non-streaming and streaming responses, which was previously replaced with an empty map when Dynamo built the response. Requests without metadata continue to return an empty object (#15102).
  • Renderer Message Rewrite Preservation: Made the frontend render transformed messages, preserving tool argument normalization and the continue_final_message marker instead of falling back to the original history. Ordinary requests keep the typed fast path (#15163).
  • Reordered Message Continuation Guard: Rejected continue_final_message requests with an InvalidArgument error before tokenization when the prompt formatter moves the final input message ahead of other messages, preventing silently truncated prompts and discarded tool results. Continuation still works for ordered tool results and assistant prefixes (#15247).
  • Anthropic Hosted Tool Rejection: Changed /v1/messages to return HTTP 400 with invalid_request_error instead of HTTP 501 for unsupported hosted tools, so clients stop retrying a deterministic rejection, and aligned the behavior with the /v1/responses hosted-tool policy while preserving NotImplemented request metrics (#15468).
  • Refusal Text Preservation: Fixed aggregated non-streaming chat responses dropping refusal text by accumulating refusal fragments per choice and retaining them in the final message, so clients receive the full refusal content. Responses without refusal text remain unchanged (#15516).
  • Streaming Cached-Token Usage: Fixed SGLang and vLLM streaming so sparsely reported cached-token usage reaches the client and explicit zero cache counts are preserved. Streaming usage now follows stream_options.include_usage and continuous_usage_stats (#16944).

Scheduling

  • Planner Load Predictor Correctness: Clamped Kalman forecasts at zero so they no longer go negative (#14497), scored failed simulation presets honestly (#14515), weighted frontend histogram averages by observation count (#14705), and reported unreadable request counts as unknown rather than zero (#14766).
  • Per-Rank DP Capacity Handling: Normalized aggregate vLLM KV capacity to per-rank values and rejected zero data-parallel rank counts (#14258), and hardened the Router for empty prompts, DP ranges, and large KV events (#15303).
  • KV Router Lookup Repair: Fixed lazy cross-worker reverse-lookup repair in the KV Router compressed radix tree so repair updates only existing entries, preventing scrubbed keys from being recreated and reviving phantom tracked blocks. Store paths retained their insertion behavior for newly stored blocks (#11891).
  • Non-Blocking Prometheus Metrics Collection: Moved Planner traffic, KV-only, and accept-length collection to a background thread so slow Prometheus queries no longer block the event loop, keeping control API requests responsive. Shared metrics are published on the event loop only after a successful collection, and collect_accept_length() is now async across provider and environment protocols (#14377).
  • Atomic Planner Decision Publishing: Fixed the virtual Planner to publish worker counts and decision ID in a single etcd transaction, so clients no longer observe partial scaling decisions (#14809).
  • Pending Virtual Connector Decisions: Fixed VirtualConnectorClient.wait() in the Planner to return immediately when an unacknowledged scaling decision already exists in etcd, instead of only reporting decisions published after the watch started. Consumers no longer miss a decision published before they began waiting (#14841).
  • Rust Worker Topology Policy: Added topology policy loading in dynamo-backend-common so Rust workers read topology settings before model registration and publish the same topology fields as Python workers. The loader waits up to 30 seconds for delayed projection (#14909).
  • KV Router Lease Leak Fix: Fixed a reference cycle between the KV Router scheduler and lease manager, so dropped or failed routers no longer leak scheduler and lease state (#15258).
  • Non-Finite Router Weight Rejection: Made Router configuration reject infinite and NaN values for overlap_score_credit_decay, prefill_load_scale, router_temperature, and decode_active_request_weight at validation time instead of failing on every request after startup. Finite values, including the largest representable value, behave as before (#15529).
  • SGLang Mooncake Key Prefix: Fixed shared-cache lookups with --shared-cache-type hicache so the Router matches SGLang's Mooncake keys, which are prefixed with the backend tag and model name (#16427).
  • Prefill Activation Fail-Closed: Fixed disaggregated serving during rollouts so requests arriving while the prefill binding is activating receive a retryable 503 instead of being forwarded straight to decode without bootstrap_info, so clients receive the 503 instead of a 500 (#16942).
  • Stale Router Discovery Record Removal: Removed the obsolete router-discovery advertisement, whose stale transport could prevent a KV-routing frontend from initializing after a container restart. Worker discovery and request and response formats are unchanged, and a KV frontend without another RPC endpoint no longer binds DYN_TCP_RPC_PORT. The last release that consumed the record was Dynamo v1.3.1 (#16828).

Multimodal & Diffusion

  • Qwen3-ASR Realtime Example Support: Reworked the vLLM realtime launch example to use vLLM's native Qwen3ASRRealtimeGeneration instead of forcing the Voxtral architecture (#15481), and set Qwen/Qwen3-ASR-0.6B as the default realtime ASR model (#16438).
  • Multimodal Request Error Propagation: Surfaced image generation failures instead of returning HTTP 200 with empty data (#13000), mapped timeouts and connection failures on user-supplied image URLs to 408 and 400 (#14240), preserved malformed data: URIs as 400 rather than collapsing to 500 (#14603), and returned 400 for multimodal requests to text-only vLLM prefill workers (#15305). The Rust frontend with --frontend-decoding now returns HTTP 400 naming the modality instead of HTTP 500 for image_url and video_url parts that fail (#16829).
  • Shared HTTP Client Media Fetching: Routed remote media downloads through the shared HTTP client with URL validation and size limits for TensorRT-LLM .safetensors embedding URLs (#14373) and vLLM-Omni ref_audio voice-cloning downloads (#14372), and enforced the DYN_MM_MAX_FILE_SIZE_MB cap, 64 MiB by default, on remote image, audio, and video downloads (#15005).
  • EPD Encode Worker Hardening: Fixed the vLLM E/P/D encode worker answering 503 forever instead of reporting healthy (#15434), and enabled per-worker port isolation and readiness checks in the multimodal launch scripts (#14751).
  • Qwen DeepStack Embedding Size: Fixed Qwen multimodal placeholder generation in vLLM to use the model's actual vision embedding dimension instead of a fixed value, resolving tensor splitting errors when running DeepStack-based models such as the Qwen3-VL series (#11244).
  • Multimodal Processor Kwargs Forwarding: Fixed the TensorRT-LLM backend dropping per-request mm_processor_kwargs, which were hardcoded to an empty dictionary before reaching generate_async while the request still returned HTTP 200. The value is now read from the top-level field with an extra_args fallback, matching the precedence used by the vLLM backend (#13856).
  • Data URI Video Decoding: Fixed TensorRT-LLM multimodal handling so inline data: URI videos are base64-decoded and passed through the same codec probe and NVDEC path as fetched HTTP(S) bytes, instead of failing with an undetected codec error. Malformed or non-base64 URIs and payloads exceeding max_file_size_bytes now return HTTP 400 (#15219).
  • Idle Media-Loader CPU Fix: Fixed media-loader NIXL agent busy-polling by setting a 1 ms progress-thread delay, dropping idle frontend CPU use from 200% to 1.9% per multimodal worker set, and drained per-read notifications that leaked about 110 bytes per image (#15289).
  • Multimodal Decode KV Identity: Fixed non-Qwen vLLM decode workers clearing multi_modal_data, which let same-size images collide in the decode prefix cache so a request could reuse another request's image KV (#15371).
  • TTS Task Type Validation: Fixed vLLM Omni to reject a task_type the Qwen3-TTS checkpoint cannot serve with a 400 instead of terminating the stage, so later valid requests no longer fail (#15591).
  • Unsupported Diffusion Layer Count: Fixed vLLM-Omni image and video worker startup by omitting the obsolete layerwise_num_gpu_layers field by default, which the diffusion configuration validator rejected before engine initialization completed. Explicit legacy CLI or environment values are now rejected during Dynamo configuration validation with guidance to use --enable-layerwise-offload without a layer-count override (#16238).

Kubernetes Deployment

  • Unified vLLM Flag Parsing: Consolidated the Operator's vLLM argument readers so every flag spelling resolved identically (#15028), and resolved repeated flags to the last occurrence, correcting world size derivation (#15224).
  • DGDR GPU SKU Discovery: Added canonical SKU identifiers to DGDR GPU discovery, starting with b300_sxm for NVIDIA B300 variants, so an omitted spec.hardware.gpuSku no longer persisted an enum-invalid product name (#15216). Canonicalized GB200 to gb200, retaining gb200_sxm as a deprecated value with an admission warning (#15711), and added Vera Rubin NVL72 support as vr_nvl72 (#15996).
  • SGLang Snapshot Memory Saver Fixes: Enabled the SGLang memory saver during argument parsing when Dynamo snapshot mode is active and suppressed raw forward-pass metrics during snapshot startup including restored pods carrying DYN_FORWARDPASS_METRIC_PORT (#14555).
  • DGD Adoption and Metadata Validation: Hardened DGDR adoption to reject an unrelated DGD with the same name and refuse to replace its confirmed deployment (#13885), and validated durable metadata on terminating DGD updates, which previously accepted every mutation (#13486).
  • Oversized Metadata Header Status: Changed the standalone endpoint picker provider (EPP) to return HTTP 431 Request Header Fields Too Large for oversized scheduling metadata headers instead of 400, and kept the too-many-entries and too-large cases distinguishable, matching the integrated frontend behavior (#14612).
  • Worker Hash Input Normalization: Fixed the Operator worker hash computation to canonicalize semantically equivalent configurations, including empty Grove wrappers, forceScalingGroup: false, disabled checkpoint settings, and explicit backend-default compilation-cache mount paths. Representation-only differences no longer change the nvidia.com/dynamo-worker-hash label, though deployments using the explicit forms may see a one-time hash change after upgrading (#14721).
  • Init Container Pull Secrets: Fixed image pull secret discovery in the Operator to scan init container images in addition to regular containers, so components with a public main image and a private initialization image no longer fail to start without explicitly configured credentials. Explicit pull secrets, deduplication, and discovery-disable behavior were preserved (#14922).
  • Bounded Dynamo Namespace Labels: Fixed derived Dynamo namespaces that exceeded the 63-byte Kubernetes label limit by shortening them to a readable prefix plus a hash, preventing Grove from rejecting a generated PodCliqueSet (#16414).
  • Shadow-Mode KV Sizing Fix: Reserved the co-located peer engine's non-KV memory in the shadow-mode vLLM scratch-KV budget, charging each engine 2 GiB for allocations outside PyTorch's allocator plus 1.25 GiB for the CUDA context, so the failover winner's KV cache fits instead of timing out while the peer is still resident (#16710).
  • Authored Frontend Command Handling: Fixed the Operator so an authored command on a frontend container or frontend sidecar owns the entire argv, dropping the default frontend Args that previously duplicated -m dynamo.frontend and crash-looped the pod. Legacy Go EPP deployments keep their generated pool, config-file, and gRPC arguments (#16814).
  • v1alpha1 envFromSecret Preservation: Fixed DGDR overrides that set v1alpha1 envFromSecret on a service whose v1beta1 main container already had envFrom, such as hf-token-secret, which previously dropped the requested Secret during conversion. envFrom is now additive like env and volumeMounts, so the result matches the equivalent v1beta1 override (#16966).
  • GMS V1 Snapshot Loader Restart Loop: Fixed the gms-loader container exiting after a successful load with DYN_GMS_USE_V1=true, which made the restarted loader crash-loop and left the pod Ready=False (#16915).

Dynamo Sidecar

Dynamo Sidecar remains experimental in Dynamo v1.6.0.

  • vLLM Sidecar Parser Passthrough: Accepted --dyn-tool-call-parser and --dyn-reasoning-parser in the vLLM sidecar so the frontend parses output as in-process workers do (#15539), and started vLLM with --reasoning-parser none so thinking requests no longer returned 400 (#15684).
  • Sidecar Streaming Payload Fidelity: Fixed the vLLM sidecar to forward buffered empty text deltas as empty strings instead of dropping them, so the frontend waits for vLLM to release its stop-string buffer rather than detokenizing the token IDs and duplicating completion text (#14699), with corrected hidden stop-token reporting and preserved cancelled prefill responses (#15243).
  • Effective Attention Block Size: Fixed the vLLM sidecar to register and publish KV events with the engine-reported effective attention block size, so routing retains KV cache information under decode context parallelism (DCP) (#14673).
  • Decode Worker KV Indexing: Fixed the shared Rust worker forcing enable_local_indexer off for decode workers, so sidecars and other callers can create local KV indexes and recovery endpoints for declared event sources with matching model metadata. Explicit enable_local_indexer=false and the encode-role exclusion remain unchanged (#14981).
  • Early Sidecar Health Probes: Started sidecar health probes before waiting for the inference engine, so Kubernetes can pass the sidecar startup probe and launch the engine container while readiness stays independent of model loading (#15051).
  • SGLang Incremental Streaming Validation: Added a discovery check that rejects SGLang sidecar startup unless GetServerInfo reports incremental streaming output as enabled, with an error directing users to relaunch SGLang with --incremental-streaming-output. This prevents duplicated output and inflated completion-token counts caused by cumulative output IDs being treated as new tokens (#15150).
  • SGLang Sidecar Refusal Messages: Fixed SGLang sidecar request validation so refusals return their specific reason in the HTTP 400 response body instead of a generic "Invalid request" (#15285).
  • SGLang Cached Token Reporting: Fixed the SGLang sidecar to read cached_tokens from the engine's meta_info and report it as usage.prompt_tokens_details.cached_tokens, clamped to the prompt length, so OpenAI completions and chat responses reflect prefix-cache hits instead of always reporting zero (#15602).
  • Guided Decoding for Tool Choice: Fixed TensorRT-LLM sidecar launch scripts, deploy examples, and the README engine command to set guided_decoding_backend: xgrammar, so chat completions requests with a required or named tool_choice return a parsed tool call instead of a 500 error (#15649).
  • Explicit Sidecar Launcher Resolution: Applied a shared resolve_sidecar helper across all 15 SGLang, TensorRT-LLM, and vLLM sidecar launch scripts, so each prefers a native sidecar command and falls back to python3 -m dynamo.<backend>.sidecar (#16059).
  • Cancelled Sidecar Request Aborts: Fixed the vLLM native gRPC sidecar to abort cancelled requests, so queued requests are released instead of occupying an engine slot until output is produced (#16201).
  • TensorRT-LLM Sidecar Request Semantics: Fixed the TensorRT-LLM sidecar to forward conversation identity across prefill and decode, disable engine detokenization when Dynamo handles it, advertise the configured default thinking mode, and accept logical rank zero while still rejecting unsupported nonzero rank targeting, so conversation-scoped KV cache reuse works and valid rank-zero requests reach the engine (#16315).
  • Hybrid DP Rank Ownership: Fixed the vLLM sidecar so a frontend owning only part of a data parallel group, such as ranks 4 to 7 of an eight-rank group, is no longer rejected at discovery. Each worker now registers its local rank range (#14697).

SGLang

  • SGLang Cancellation Lifecycle Hardening: Added a bounded cancellation lifecycle for SGLang decode and aggregated workers that aborts requests cancelled before the first output token (#14781), extended it to disaggregated prefill workers and their paired decode-side state (#14782), and returned SGLang validation aborts as request errors before placeholder tokens or a successful finish could be emitted (#15065).
  • DCP Attention Backend Validation: Added a startup check that rejects --dcp-size on attention backends that do not read the DCP parallel state, where the SGLang worker would kill its scheduler (#14800), and extended it to MLA decode context parallel, suggesting --attention-backend flashinfer, trtllm_mla, or cutedsl_mla (#16255).
  • SGLang Guided Decoding Fix: Removed --skip-tokenizer-init from SGLang manifests and docs, because the flag caused response_format: json_schema requests to return content that did not satisfy the schema (#13605).
  • Disaggregated Parallel Sampling Rejection: Added validation that rejects SGLang disaggregated requests with n > 1 using an HTTP 400 error instead of letting them hang indefinitely. Aggregated parallel sampling and disaggregated n=1 requests continue to work (#14112).
  • SGLang API Compatibility Shims: Added compatibility shims for get_model_config and use_mla_backend, which SGLang moved from ServerArgs to its overrides module, fixing an AttributeError that prevented the SGLang worker from starting against newer SGLang versions (#14234).
  • Prefill DP Rank Forwarding: Fixed SGLang disaggregated serving so the Router forwards the prefill data-parallel rank to decode, dropping decode bootstrap_duration from 150 ms to 650 ms down to 0.3 ms to 0.6 ms on GLM-5.2 (#15246).
  • Prefill First-Token Logprobs: Fixed disaggregated prefill in SGLang to forward the generated-token logprobs option through the shared helper, so the first generated token now carries its log-probability data. Prompt-wide logprobs remain excluded because prefill discards that metadata and the Router does not merge it into the decode response (#15512).
  • Text-Only Engine Media Rejection: Added SGLang worker validation that returns a 400 error for image, video, or audio content on a text-only model, instead of silently dropping the media (#15520).
  • Attention-DP Rank Counting Fix: Fixed SGLang DP rank counting to match dp_size * attn_dp_size, so attention-DP workers register every rank instead of only rank 0 (#16598).
  • SGLang Crash Exit Delay: Fixed a 60-second delay in worker exit and pod restart after an SGLang child process died by defaulting SGLANG_CUDA_COREDUMP_BEFORE_CRASH_WAIT_SECS to 0 unless the user sets it, sets CUDA_ENABLE_USER_TRIGGERED_COREDUMP=1, or passes --crash-dump-folder. On one A100, worker exit after a killed detokenizer dropped from 66 seconds to 6 seconds (#16952).

TensorRT-LLM

  • Request-Local Pre-Submit Validation Errors: Fixed TensorRT-LLM handling of pre-submit ValueError, TypeError, and NotImplementedError failures so they return a request-scoped error instead of shutting down a healthy engine, while worker shutdown is preserved for unhealthy engines and errors raised during result iteration (#14006).
  • Decode Cached Token Reporting: Fixed usage.prompt_tokens_details.cached_tokens on the TensorRT-LLM generation-only path, which reported the transferred prompt as a cache hit, so benchmarks, dashboards, and routing decisions that read the usage block get accurate cache-hit data when return_perf_metrics is enabled (#15389).
  • Salted TensorRT-LLM Health Checks: Fixed TensorRT-LLM workers aborting when attention data parallelism and the V2 KV cache manager were enabled by giving the health-check probe request its own cache salt, so it no longer shares a reuse namespace with the engine's dummy requests. A caller-supplied salt still takes precedence (#16688).
  • MPI Child Failure Exit: Fixed TensorRT-LLM workers hanging until Kubernetes killed the container after an MPI engine child died. The worker now exits promptly with code 1 (#16931).
  • Snapshot Restore MPI Transport: Fixed restored TensorRT-LLM engines with TP greater than 1 failing their first collective with ucx send failed: Endpoint timeout. Snapshot mode now defaults OMPI_MCA_pml=ob1 and OMPI_MCA_btl=tcp,self, because Open MPI's UCX transport enables TCP keepalive probes that abort inter-rank connections during the checkpoint dump. Explicit settings still take precedence (#16941).

vLLM

  • ModelExpress RL Version Declaration: Reworked vLLM weight version reporting to report version: null with version_declared: false until declared, adding a set_weight_version route (#13041). Workers loading through the ModelExpress RL loader now declare MX_REFIT_DESIRED_VERSION_UID at startup (#15252).
  • Prefill Worker Lifecycle Hardening: Shut down EngineCore when a prefill worker exits and made FPM self-benchmark inputs reproducible with retained input, timing, execution, preparation, and worker provenance (#16789), with prefill health probes no longer requesting a KV connector handoff, which previously left unclaimed KV transfer state on idle prefill workers and blocked KV cache resets (#16825).
  • Elastic EP Grow Rollback: Fixed vLLM scale_elastic_ep so a failed grow rolls back to the previous data-parallel size and keeps serving instead of wedging the engine (#12991).
  • Engine-Expanded Prompt ID Metadata: Fixed vLLM nvext.engine_data to report the engine-processed prompt tokens, so training consumers can align them with completion tokens and log probabilities (#15470).

Model Loading & LoRA

  • LoRA Card Registration Fix: Fixed LoRA adapter registration in the SGLang and vLLM handlers by passing ignore_weights=True, so card registration no longer downloads base-model weights on a cache miss and fails before publishing the adapter (#11164).

Performance Modeling & Replay

  • Forward-Pass Metrics KV Warm-Up: Settled KV warm-up so decode sweeps no longer abort when attention-DP ranks report READY in different steps, with hybrid KDA/Mamba fixes (#14728, #14614). Snapshot-restored vLLM replicas now stay visible to the Planner (#12545).

Fault Tolerance & Observability

  • Request Trace Record Accuracy: Reworked audit records to derive payload_complete and payload_drop_reason from the actual response (#14143), and wrote request_end for requests cancelled before the first response item (#14799).
  • Histogram Bucket Configuration Fixes: Raised the default inter-token latency bucket ceiling from 2.0s to 80.0s (#14839).
  • Etcd and NATS Connection Resilience: Added etcd watch retry with backoff, enabled HTTP/2 keepalive by default with a 15s ping interval and 10s acknowledgment timeout, and raised the default lease TTL from 10s to 30s (#15266). Startup now retries temporary NATS connection failures for up to 120 seconds (#15269).
  • System Status Listener Rebinding: Fixed the system status server to rebind its TCP listener after accept failures instead of retrying a stale descriptor indefinitely, keeping status endpoints reachable after process restore while retaining the originally bound address, including an assigned ephemeral port (#12059).
  • Prefill Worker KV Gauges: Fixed missing dynamo_component_total_blocks and dynamo_component_gpu_cache_usage_percent samples on disaggregated vLLM prefill workers, which now publish configured total capacity at startup (#12559).
  • Pending Handshake Cancellation: Fixed requests hanging when a worker left discovery before sending the response prologue, so worker removal unblocks the requester with a migratable Disconnected error (#13095).
  • Worker Error Preservation: Fixed finish-reason handling in the backend so a worker's error finish reason is no longer overwritten by local stop detection, token limits, or user-defined stop strings, preventing failed requests from returning HTTP 200 with finish_reason: "stop" and malformed output (#15094).
  • Migration Handshake Cancellation Accounting: Fixed the Frontend to record client cancellation during a migration handshake as outcome="cancelled" instead of outcome="success" in dynamo_frontend_model_migration_duration_seconds, so migration metrics match the HTTP outcome for the same request (#16440).
  • Invalid Request Refusal Handling: Fixed worker routability after a worker refused an invalid request before opening the response stream, returning the worker's own error instead of wrapping it in CannotConnect. Clients now receive HTTP 400 and the worker stays routable rather than being marked down and causing HTTP 503 on the next valid request (#16597).

Runtime & Transport

  • IPv6-Only Networking Support: Reworked local address resolution to prefer non-loopback IPv4, then IPv6, then loopback (#13126), with the ZMQ event plane reusing it (#15324). Narrowed ZMQ_IPV6 to bracketed IPv6 endpoints, fixed remaining IPv6-only gaps in listeners, SGLang KV events, and vLLM multinode (#15383), and changed automatic host selection to prefer the routed source address (#15628).
  • QUIC Response Plane Reliability: Carried typed pre-stream errors through the QUIC response plane, mapping InvalidArgument to HTTP 400 (#14789), and updated Quinn to v0.11.12 to fix too many gaps in stream buffer failures (#14876).
  • Endpoint Identity-Scoped TCP Routing: Scoped TCP endpoint routing and cleanup by full endpoint identity, so endpoints registered under the same name in one process no longer overwrite or tear down each other's handlers (#14723, #14271).
  • Oversized TCP Frame Rejection: Added a preflight size check in the TCP client that rejects requests exceeding DYN_TCP_MAX_MESSAGE_SIZE before connection lookup and admission, returning InvalidArgument which the HTTP layer maps to a 400 response instead of letting the oversized frame reach the worker and close the connection during decode (#14110).
  • ZMQ Socket Exhaustion Guidance: Added error messages when event-plane socket creation fails with EMFILE, so operators hitting file-descriptor limits see advice to reduce direct-ZMQ peers or raise ulimit -n (#14145).
  • Shared Fetch Model Runtime: Fixed fetch_model() creating a second Tokio runtime when called before a Dynamo runtime existed, and the Python Worker mutating the process environment when applying transport overrides (#14534).
  • ZMQ Publisher Port Race: Fixed a race during concurrent ZMQ publisher startup by binding the publisher socket directly to an ephemeral port instead of probing with a temporary TCP listener, and read the assigned address back from the socket for logging and discovery, so concurrent publishers no longer collide on the same port (#14566).
  • Discovery Watch Establishment Ordering: Fixed discovery list-and-watch to return only after the backend watch is established, preventing missed Removed events (#14769). Every watch now opens with one Added event per matching instance and one Resync snapshot, so a reopened watch replaces state without a separate list call (#15143).

Container Images

  • Root Cache Mount Target: Fixed container/run.sh so that user overrides with a group suffix, such as --user root:root or 0:1000, mount the Hugging Face cache at /root/.cache/huggingface instead of the Dynamo user cache directory. Non-root users with group 0 continue to use the Dynamo user cache (#14820).

Documentation

New Content

  • EPP Mode Boundary Documentation: Clarified which EPP mode operator-generated resources apply to and the RBAC boundary with standalone mode (#14756, #14620). Documented the FailClose endpoint-picker failure mode, required a VLLM_IMAGE variable in the on-ramp example, and corrected docs left over from the Go EPP deprecation (#14811, #14481, #14263).
  • Operator Deployment Caveat Docs: Added a warning that compilationCache PVC contents must be cleared after a backend, torch, or CUDA upgrade (#14381), and documented that creating a DGDR carries the same authority as creating a Job or Pod in that namespace, while Pod-level security stays enforced by Pod Security Admission (#14429).
  • Snapshot API Documentation: Replaced the DynamoCheckpoint Auto mode wording with the SnapshotJob to PodSnapshot workflow (#16037), with a Kubernetes capture and restore quickstart including TensorRT-LLM examples for Qwen3-0.6B on one GPU and pinned GLM 5.3 NVFP4 on eight B200s (#16044).
  • KV Cache Transfer Docs Corrections: Corrected the TensorRT-LLM docs to state TRTLLM_USE_UCX_KVCACHE enables direct UCX even with cache_transceiver_config.backend: DEFAULT, and added a warning that mounting host /opt/amazon/efa breaks the NIXL LIBFABRIC backend (#13048).
  • Secure Deployment Guidelines Documentation: Added a Secure Deployment Guidelines page documenting Dynamo's trust model, securing the external inference API, authenticating the discovery, event, and request planes, and running with least privilege (#13505).
  • Nightly Install Selector Accuracy: Fixed the nightly install selector rows to show the shipped backend versions, or "not currently available" instead of the stable tag (#13599).
  • Planner Mode Selection Guide: Added a Choose a Planner Mode guide, split the overview by whether users have TTFT/ITL SLA targets, and documented KV cache reuse and speculative decoding (#14335).
  • EPD Disaggregation Blog Post: Added a blog post on when to use encode-prefill-decode (EPD) disaggregation for multimodal serving, and listed the Pinterest Engineering article on building a VLM serving stack with Dynamo in the external publications page (#14630).
  • LongCat TokenSpeed Disaggregation Example: Added an experimental LongCat-Flash TokenSpeed disaggregated example with KV-aware routing, and its GB300 configuration raises sequence and KV-cache capacity (--max-num-seqs 4 to 32, --max-total-tokens 262144 to 917504) and selects trtllm_mla to address long-context decode slowdown (#15013).
  • Nemotron-3-Ultra Performance Table: Restored the Expected Performance table, covering all 12 aggregated and disaggregated profiles across B200, GB200, and H200 at 256K and 1M context lengths (#15194).
  • SGLang Speculative Decoding Guide: Added a documentation page for EAGLE3 speculative decoding with SGLang, covering aggregated quick start, disaggregated serving with the KV router, and sglang:spec_* metrics, and marked SGLang speculative decoding as supported in the speculative decoding overview, the SGLang feature matrix, and the compatibility tables (#15268).
  • Many-Model GPU Sharing Examples: Added a Many-Model Serving section and examples/gpu-sharing/ directory benchmarking Qwen3-4B on one 8x A100 40GB node. At equal load per GPU, GPU fractions retained 60-88% of dedicated-GPU throughput versus 45-73% for HAMi, about 23% cheaper per output token at light per-model load (#15918).
  • Sidecar Documentation Simplification: Simplified the sidecar READMEs and Fern pages to one example per engine, added topology diagrams for SGLang, TensorRT-LLM, and vLLM, and corrected stale claims: the dynamo-sidecar image is published to NGC starting with v1.6.0 (#16062).
  • Tied-Embedding Policy Update Limitation: Documented that the first policy update fails for Qwen2.5 and other tied-embedding models on vLLM v0.29.0 and later, with links to the upstream verl fixes, which no reviewed recipe snapshot selects yet (#16682).

Version Upgrades

Major Dependencies

  • SGLang: Upgraded from v0.5.18 to v0.5.21 (#15538).
  • TensorRT-LLM: Upgraded from v1.3.0rc25 to v1.3.0rc29 (#15418).
  • vLLM: Upgraded from v0.28.0 to v0.31.0 (#15643).
  • vLLM-Omni: Upgraded from v0.28.0rc1 to v0.31.0rc1 on CUDA images, and from v0.28.0rc1 (CPU) and v0.27.0rc1 (XPU) to v0.30.0rc1.

Other Dependencies

  • NIXL: Updated SGLang from v1.4.0 to v1.5.0, vLLM from v1.3.2 to v1.4.1, and the Frontend from v1.3.2 to v1.5.0. TensorRT-LLM stays on v1.3.1.
  • vLLM (CPU and XPU): Updated from v0.28.0 (CPU) and v0.27.1 (XPU) to v0.30.0.
  • SGLang (XPU): Updated from v0.5.11 to v0.5.19, built against oneAPI 2026.0.
  • libfabric: Updated from v2.4.0 to v2.6.0.
  • etcd: Updated from v3.5.33 to v3.5.34.
  • NATS: Updated from v2.12.14 to v2.14.8 (#16845).
  • LMCache: Raised the floor to v0.5.6 in the vLLM CUDA runtime image (#16845).
  • AIPerf: Updated from v0.12.0 to v0.13.0.
  • AISimulate: Updated from v0.12.0 to v0.13.0.
  • Grove: Updated from v0.1.0-alpha.13 to v0.1.0-alpha.14.
  • ModelExpress: Updated from v0.6.0 to v0.7.0.
  • dynamo-renderer: Updated from v5.1.2 to v8.0.3 (#16686).
  • dynamo-tokenizers: Updated from v1.8.1 to v3.1.1 (#16686).
  • Planner Image Python: Updated from CPython 3.12 to 3.13 for the dynamo-planner container (#14924).
  • Triton Runtime Image: Updated from 26.07-py3 (CUDA 13.2) to 26.09-py3 (CUDA 13.4).

Known Issues

Backend (SGLang)

Out-of-Vocab Stop Token Crash

A request whose stop_token_ids includes an ID at or above the model's vocabulary size can crash the SGLang scheduler when any request in the batch sets min_tokens, failing every request on that worker until it restarts. The issue also affects v1.5.0. No workaround is available yet.

Snapshot and GMS CUDA Graph Startup Failure

SGLang workers in Dynamo Snapshot mode or with GPU Memory Service V1 checkpointing fail at startup with a NotImplementedError from prefill CUDA graph initialization. The issue also affects v1.5.0. Workaround: pass --disable-cuda-graph to these workers.

Redundant Memory Resume Crash

With GPU Memory Service on SGLang, sending resume_memory_occupation twice without a release_memory_occupation in between crashes the scheduler. Workaround: send each resume only after a matching release.

Backend (TensorRT-LLM)

Oversized Anthropic Messages Return 500

On TensorRT-LLM, non-streaming /v1/messages requests whose prompt exceeds the engine's max_num_tokens return HTTP 500 instead of HTTP 400. No workaround is available yet.

Backend (vLLM)

Omitted Max Tokens Generates to the Context Limit

On vLLM, a request without max_tokens can generate up to the full remaining context window when the model sets no default output length. Workaround: set max_tokens on every request.

GMS Persisted KV Standby Wake Failure

With GPU Memory Service persisted KV failover (DYN_GMS_PERSIST_KV), tensor parallelism, and an explicit --kv-cache-memory-bytes, the vLLM standby engine fails to adopt the retained KV cache with StaleMemoryLayoutError. The issue also affects v1.5.0. Workaround: leave DYN_GMS_PERSIST_KV unset.

CLI & Tooling

Replay Module Entrypoint Removal

python -m dynamo.replay fails because the replay CLI moved into the AISimulate wheel in v1.5.0. Workaround: use aisimulate predict --stack dynamo --config <file>. See the replay guide.

Frontend

Logit Bias Ignored

The Frontend accepts logit_bias on /v1/chat/completions and /v1/completions but drops it before the engine, so no bias is applied. No workaround is available yet.

Request Error Details Dropped

When a worker rejects a request, the Frontend returns a generic error without the worker's message. The issue also affects v1.5.0. No workaround is available yet.

Planner

Inflated Capacity Estimate Suppresses Scaling

For aggregated vLLM deployments, a Dynamo Planner regression model seeded from worker self-benchmark data can overestimate per-engine capacity and refuse to scale up under load. Workaround: use a pre-published AISimulate performance model.

Runtime & Router

GlobalRouter Pool Requests Find No Endpoints

On Kubernetes with Grove, requests that the GlobalRouter sends to a pool can fail with no available endpoints even when the pool's worker is healthy. The issue also affects v1.5.0. No workaround is available yet.

Decode-Only Worker Pin Fails in Disaggregated Serving

A request that pins only a decode worker fails instead of letting the Router select a prefill worker. The issue also affects v1.5.0. Workaround: pin both workers with x-dynamo-worker-instance-id and x-dynamo-prefill-instance-id, or pin neither.

Kubernetes & Operator

Brief 404 Errors During Rolling Updates

Rolling updates of disaggregated deployments with a max surge of 1 and max unavailable of 0 can return HTTP 404 for up to about one second while a worker is replaced. The rollout still completes. The issue also affects v1.5.0. No workaround is available yet.

EFA Network Plugin Not Loaded

On AWS EFA clusters, SGLang and TensorRT-LLM workers configured from the EFA on AWS guide silently fall back to socket transport because the guide's LD_LIBRARY_PATH entry is wrong. The issue also affects v1.5.0. Workaround: use /opt/amazon/ofi-nccl/lib instead of /opt/aws-ofi-nccl/lib.

Reinforcement Learning

Slime Weight Synchronization Failure

With the experimental Slime integration, the first weight update from the Slime trainer to SGLang v0.5.21 engines fails with HTTP 400, and restarting the trainer against engines still running from an earlier job also fails. A fix is planned for a future release. No workaround is available yet.

Carried Forward

The other v1.5.0 and v1.5.1 known issues are fixed in v1.6.0 or were closed as intended behavior.

Looking Ahead

Runtime Router Policy Management

A /router_policy API on the Frontend reads and updates live worker-selection parameters and busy thresholds without a restart.

Planner Reconfiguration & Batch Scheduling

Runtime reconfiguration extends beyond GPU budgets to a running Planner's SLA targets and load predictor. An opt-in batch scheduling policy reserves capacity for online traffic, with the Batch API served through the Frontend.

Generation Artifacts for RL

vLLM workers can deliver bounded generation artifacts inline or to object stores for reinforcement learning pipelines.

Kubernetes Deployment Search

An experimental v1beta2 API runs simulation-backed deployment searches, and DisaggregatedSet joins Grove as an opt-in workload provider.

Assets 2
Loading