feat: (experimental) Dynamo aggregated backend - inference (4/5)

[praateekmahajan]

Reviewer tip: the diff in nemo_curator/core/serve/dynamo/backend.py (lifecycle / infra / frontend / health) and the new nemo_curator/core/serve/dynamo/vllm.py (worker launch) are where the real work lives. Everything else is tests, small constants.

Description

• PR 4/5 of the inference-server restack
• replace the feat!: Typed Serve Config + Dyamo config stub - inference (2/5) #1820 DynamoBackend placeholder with a real backend for aggregated serving — single-node TP and multi-node TP — built on the placement-group API landed in feat: Placement Group based subprocess manager - inference (3/5) #1833
• split worker-launch concerns out of dynamo/backend.py into a new dynamo/vllm.py so lifecycle (start/stop, infra, frontend, health) and vLLM-specific worker assembly stay separately reviewable
• promote a few repeated strings and indexes to typed constants in dynamo/constants.py: actor labels (ETCD_ACTOR_LABEL, NATS_ACTOR_LABEL, FRONTEND_ACTOR_LABEL) and infra-bundle layout (INFRA_ETCD_BUNDLE, INFRA_NATS_BUNDLE, INFRA_FRONTEND_BUNDLE, INFRA_NUM_BUNDLES)
• mode=\"disagg\" raises NotImplementedError here; disagg + cross-model validators + full router-flag translation ship in the final PR

Design callouts

• Single frontend per server, not per model. The Dynamo frontend auto-discovers every registered model via etcd, so one --http-port on the infra PG serves all models. The full per-key router-flag merge across models is PR 5's job; this PR wires only --router-mode plus router_kwargs passthrough.
• Explicit --kv-events-config on every worker. Without this, Dynamo's args.py auto-creates a KVEventsConfig bound to tcp://*:20080 when prefix_caching is enabled (vLLM ≥0.16 default), and every worker on the same node fights over the same port. Rank-0 workers publish ZMQ KV events only when the router is mode=\"kv\" with kv_events=True; every other case disables events explicitly so the default-port binding never happens.
• One _launch_vllm_worker for rank-0 and headless ranks. Rank-0 adds endpoint / discovery / planes / optional KV-events publisher; rank-N adds --headless and forces KV events off. Multi-node TP resolves --master-addr post-pg.ready() via get_bundle_node_ip(pg, 0).
• Orphan sweeps in start(). remove_named_pgs_with_prefix reaps stale PGs, but PG removal force-kills scheduled actors and bypasses their atexit hook — which would orphan the subprocess tree. A list_actors(state=\"ALIVE\") sweep filtered by the PG name prefix runs first so graceful_stop_actors can SIGTERM the process groups cleanly (with SIGKILL escalation from PR 3 as the fallback).
• Typed self._models narrowed once in __init__. server.models: list[BaseModelConfig] carries no Dynamo-specific fields; InferenceServer._validate_model_configs already enforces that every entry is a DynamoVLLMModelConfig, so we cast once and every backend method gets .num_replicas / .mode / .engine_kwargs autocomplete.

Usage

from nemo_curator.core.serve import (
    DynamoRouterConfig,
    DynamoServerConfig,
    DynamoVLLMModelConfig,
    InferenceServer,
)

model = DynamoVLLMModelConfig(
    model_identifier=\"Qwen/Qwen3-0.6B\",
    num_replicas=2,
    engine_kwargs={\"tensor_parallel_size\": 1, \"enforce_eager\": True},
    # `dynamo_kwargs` is a dict pass-through for any `python -m dynamo.vllm` flag
    # Curator doesn't type yet (e.g. tool_call_parser, reasoning_parser, ...).
    dynamo_kwargs={\"tool_call_parser\": \"hermes\"},
)
backend_cfg = DynamoServerConfig(
    router=DynamoRouterConfig(mode=\"kv\", kv_events=True),
)
with InferenceServer(models=[model], backend=backend_cfg) as server:
    print(server.endpoint)  # http://<infra-node-ip>:<port>/v1

What's intentionally NOT in this PR

• disaggregated serving (mode=\"disagg\"): raises NotImplementedError with a pointer to PR 5
• cross-model validators: _validate_frontend_config (coherent namespace/planes/router across models), _validate_unique_model_names, and the disagg-TP-fit branch of _validate_gpu_requirements
• full router-flag translation: _resolve_frontend_router_config per-key fallback + --router-kv-events / --router-temperature / --router-ttl-secs / etc. — PR 5 promotes whatever needs typed fields; everything else stays on router_kwargs
• Ray Serve backend changes: this PR touches only the Dynamo subpackage

Verification

• ruff check nemo_curator/core/serve/ tests/core/serve/ — clean
• pytest tests/core/serve/ -m \"not gpu\" — 67 passed (~43s)
• CUDA_VISIBLE_DEVICES=2,3 pytest tests/core/serve/integration/test_dynamo.py -m gpu — 3 passed (~3m49s)
  • TestDynamoAggregatedSingleNode::test_is_active_and_queryable — full Dynamo frontend + etcd + NATS + vLLM worker answers OpenAI chat completions
  • TestDynamoAggregatedSingleNode::test_restart_after_stop — exercises the orphan-PG + orphan-actor sweeps
  • TestDynamoRejectsDisagg::test_disagg_mode_raises_notimplemented — verifies the PR 5 deferral

Checklist

• I am familiar with the Contributing Guide.
• New or Existing tests cover these changes.
• The documentation is up to date with these changes.

[praateekmahajan]

/claude review

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[praateekmahajan]

@greptileai

/claude review

[praateekmahajan]

/ok to test 67e9c47

[praateekmahajan]

@greptileai
/claude review

[praateekmahajan]

/ok to test 3e90016

[claude]

LGTM

[greptile-apps]

Want your agent to iterate on Greptile's feedback? Try greploops.

[oyilmaz-nvidia]

To be honest, it's a long PR and there are too many details. But when I look at the high level API that users will use, I think it's simple and good designed.

So, overall it looks good to me. But we need more examples and tutorials to test it. I guess you'll have them in the last PR?