feat(vllm): auto-prepare trusted dual DGX Stations#7030
Open
ericksoa wants to merge 77 commits into
Open
Conversation
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Senthil Kumar Ravichandran <senthilr@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThis PR adds DGX Station dual-peer qualification, secure SSH and resume state, managed two-node vLLM lifecycle orchestration, onboarding integration, updated documentation, and a guarded fixture-backed simulator with extensive tests. ChangesDual-Station installer and preparation
Estimated code review effort: 5 (Critical) | ~120 minutes Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Contributor
Code Coverage OverviewLanguages: TypeScript TypeScript / code-coverage/pluginThe overall coverage in commit abc34e4 in the TypeScript / code-coverage/cliThe overall coverage in commit abc34e4 in the Show a code coverage summary of the most impacted files.
Updated |
Contributor
PR Review Advisor — InformationalAdvisor assessment: Informational / high confidence Model lanes
Nemotron output stays in workflow artifacts and does not change the assessment above. E2E guidanceAdvisory only. E2E / PR Gate selects and runs jobs independently. Recommended E2E: 1 optional E2E recommendation
This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
ericksoa
added
platform: dgx-station
Signed-off-by: Senthil Kumar Ravichandran <senthilr@nvidia.com>
…st-prereqs Signed-off-by: Senthil Kumar Ravichandran <senthilr@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com> (cherry picked from commit 586a60d)
# Conflicts: # docs/get-started/dgx-station-preparation.mdx # docs/get-started/quickstart.mdx # docs/reference/commands.mdx # docs/resources/starter-prompt.md # test/starter-prompt-docs.test.ts
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Comment on lines
2487
to
2490
| require_command apt-cache | ||
| require_command apt-get | ||
| require_command cmp | ||
| require_command curl | ||
| require_command dpkg |
Comment on lines
568
to
571
| if (vllmRunning) { | ||
| const message = | ||
| `vLLM is already running on localhost:${String(deps.vllmPort)}. ` + | ||
| "vLLM is already running on the managed endpoint. " + | ||
| "Select Local vLLM, or stop the existing server before selecting the managed install path."; |
Comment on lines
+98
to
+106
| const filePath = dualStationVllmApiKeyPath(stateDir); | ||
| const noFollow = fs.constants.O_NOFOLLOW ?? 0; | ||
| let fd: number | undefined; | ||
| try { | ||
| fd = fs.openSync( | ||
| filePath, | ||
| fs.constants.O_WRONLY | fs.constants.O_CREAT | fs.constants.O_EXCL | noFollow, | ||
| 0o600, | ||
| ); |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 67 out of 67 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (2)
src/lib/onboard/setup-nim-flow.ts:571
- The message for
vllmRunningnow claims "managed endpoint", butvllmRunningcomes from generic host-state probing and can represent any already-running vLLM. This can mislead users on non-managed setups; keep the wording topology-neutral.
scripts/prepare-dgx-station-host.sh:2490 verify_apply_statenow callsverify_dual_station_controller_uid_binding, which usescmp, butrun_applyno longer requirescmp. On minimal environments this can fail mid-run with a missing-command error; keepcmpin the--applyprerequisite list.
require_command apt-cache
require_command apt-get
require_command curl
require_command dpkg
Comment on lines
+249
to
+251
| const apiKey = managedBinding?.apiKey ?? null; | ||
| const managedDualEndpoint = managedBinding !== null; | ||
| console.log( |
Comment on lines
1088
to
1093
| const handleVllmSelection = createSetupNimVllmHandler({ | ||
| VLLM_PORT, runCapture, getLocalProviderBaseUrl, getLocalProviderValidationBaseUrl, | ||
| isSafeModelId, requireValue, validateOpenAiLikeSelection, | ||
| getManagedVllmProviderBinding: localInference.getManagedDualStationVllmProviderBinding, queryVllmModels: (baseUrl, apiKey) => { const result = localInference.probeVllmModels(baseUrl, apiKey); return result.ok ? result.body : ""; }, isSafeModelId, requireValue, validateOpenAiLikeSelection, | ||
| applyVllmRuntimeContextWindow: localInference.applyVllmRuntimeContextWindow, isDgxSparkHost: () => nim.detectNvidiaPlatform() === "spark", isNemoClawManagedVllmRunning, | ||
| exitProcess: (code) => process.exit(code), | ||
| }); |
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 67 out of 67 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (1)
src/lib/inference/vllm-api-key.ts:99
- ensureDualStationVllmApiKey() falls back to
0whenfs.constants.O_NOFOLLOWis unavailable. That silently disables the no-symlink guarantee, allowing the key write to follow a symlink on platforms lacking O_NOFOLLOW support. The corresponding read path fails closed when O_NOFOLLOW is missing; the write path should do the same for consistency and to avoid a downgrade in protection.
const noFollow = fs.constants.O_NOFOLLOW ?? 0;
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Comment on lines
118
to
+126
| function probeVllmRunning(runCapture: RunCapture): boolean { | ||
| return !!runCapture( | ||
| ["curl", "-sf", ...LOCAL_PROVIDER_PROBE_CURL_ARGS, `http://127.0.0.1:${VLLM_PORT}/v1/models`], | ||
| { ignoreError: true }, | ||
| ); | ||
| const endpoint = getLocalProviderAvailabilityEndpoint("vllm-local"); | ||
| if (!endpoint) return false; | ||
| const writeOut = endpoint.endsWith("/health") | ||
| ? ["--noproxy", "*", "--write-out", "%{http_code}"] | ||
| : []; | ||
| return !!runCapture(["curl", "-sf", ...LOCAL_PROVIDER_PROBE_CURL_ARGS, ...writeOut, endpoint], { | ||
| ignoreError: true, | ||
| }); |
Comment on lines
959
to
963
| case "vllm-local": | ||
| return { | ||
| ok: false, | ||
| message: `Local vLLM was selected, but nothing is responding on http://127.0.0.1:${VLLM_PORT}.`, | ||
| message: `Local vLLM was selected, but nothing is responding on ${getLocalProviderHealthEndpoint(provider) ?? "the configured endpoint"}.`, | ||
| }; |
Comment on lines
994
to
998
| case "vllm-local": | ||
| return { | ||
| ok: false, | ||
| message: `Local vLLM is responding on 127.0.0.1, but the Docker container reachability check failed for http://host.openshell.internal:${VLLM_PORT}. This may be a Docker networking issue — the sandbox uses a different network path and may still work.`, | ||
| message: `Local vLLM is responding on the host, but the Docker container reachability check failed for ${getContainerCheckUrl(provider)}. This may be a Docker networking issue — the sandbox uses a different network path and may still work.`, | ||
| diagnostic, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR makes the existing dual-DGX Station Nemotron 3 Ultra path automatic after the operator-owned physical/network recipe and SSH trust are already in place. A single NemoClaw installer run on one DGX Station GB300 derives the direct
/30counterparts, selects one unambiguous pretrusted reciprocal peer, prepares both hosts, resumes safely across manual reboots, and hands the qualified pair to the managed vLLM deployment flow.The managed runtime now follows NVIDIA's current dual-Station NemoClaw playbook: vLLM
v0.25.1-aarch64, Ray2.56.0, tensor parallelism 1, pipeline parallelism 2, a 262,144-token maximum model length, FP8 KV cache, prefix caching, and the served aliasnemotron-ultra.DGX Station remains Deferred. This PR does not automate rail configuration, firmware, SSH trust enrollment, or reboots, and it does not claim physical-hardware acceptance.
Changes
/30, 400 Gb/s, MTU-9000 CX-8 rails. There is no subnet scan,ssh-keyscan, mDNS, Avahi, LLDP, ARP sweep, or trust enrollment.NEMOCLAW_DGX_STATION_PEERandNEMOCLAW_VLLM_MODELselections authoritative and fail closed on conflicting selections.scripts/prepare-dgx-station-host.shas the sole host-preparation implementation, with exact-byte hashing and strict noninteractive SSH execution.vllm/vllm-openai@sha256:2cc49b81319f7a66a33dd8bd63a7bfddae079122b33ce51989b6828a1f038c37while leaving the single-Station Ultra runtime unchanged.2.56.0into the container's private writable home, start the head on the selected RoCE address at TCP 6379, wait transactionally for the worker GPU, and launch one authenticated vLLM API with PP=2/TP=1.Blockers before marking ready
Type of Change
Quality Gates
Verification
Exact pushed head:
85371981fc9c5dac7d42ba93587a707141104712Signed-off-by: Aaron Erickson aerickson@nvidia.com