Unblock the DROID policy server setup on CUDA 13 - #187
Merged
Conversation
- Dockerfile: declare TORCH_CUDA_ARCH_LIST as a build arg so apex's CUDA
extensions can be built for specific architectures. Left to torch's
default the list includes compute_70, which CUDA 13 no longer supports,
so the build fails with:
nvcc fatal : Unsupported gpu architecture 'compute_70'
Narrow it at build time with, for example:
docker build --build-arg TORCH_CUDA_ARCH_LIST=12.0 .
ARG rather than ENV so the value reaches apex's os.environ during the
build without persisting into the final image, where it would also
govern runtime JIT compilation. The default is empty, which torch
treats identically to unset, so existing builds are unchanged.
- action_policy_server_robolab: time each infer call and print the
elapsed milliseconds when EVAL_VERBOSE is set, to measure policy
server latency during RoboLab evaluation runs.
- docs: point the policy server setup at cu130-torch213-train.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
apex bf903a2 assembles its own -gencode flags and appends
arch=compute_70,code=sm_70 unconditionally, gated only by lower bounds on
the CUDA version, so it cannot build against CUDA 13:
nvcc fatal : Unsupported gpu architecture 'compute_70'
Those code paths never consult TORCH_CUDA_ARCH_LIST, so the build arg added
in 6bbef0d could not help; drop it again. Bump the pin to 9e3568a, where
upstream has removed the hand-built lists and derives the architectures
from the detected CUDA version instead.
Add INSTALL_APEX to skip the build entirely, which is by far the slowest
layer, and use it in the documented docker build command. apex is optional:
the only import is in callbacks/norm_monitor.py, behind a try/except.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ychao-nvidia
force-pushed
the
ychao/20260812_edge_policy_droid
branch
from
August 12, 2026 19:03
bb61ee1 to
d863918
Compare
lfengad
approved these changes
Aug 13, 2026
lfengad
reviewed
Aug 13, 2026
| ' | ||
| ``` | ||
|
|
||
| The `--group=cu130-train` line targets CUDA 13.x drivers. On CUDA 12.x systems, replace it with `--group=cu128-train` (see the [Cosmos3 Cookbooks: Environment Setup](https://github.com/NVIDIA/cosmos/blob/main/cookbooks/cosmos3/README.md) for details). |
Collaborator
There was a problem hiding this comment.
Could the cu128 also supported?
We may add some explanations such as:
- cu128 not supported?
- if cu130, please use cu130-torch123-train rather than cu130-train
Thanks!
Collaborator
Author
There was a problem hiding this comment.
We have not tested on cu128, so I removed it for now.
Collaborator
|
LGTM |
pengcuo
approved these changes
Aug 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Moves the Cosmos3-Policy-DROID server setup onto torch 2.13, adds the instrumentation needed to measure its inference latency during RoboLab evaluation, and makes the apex build optional — it currently cannot build against the CUDA 13 base image at all, and is the slowest layer by a wide margin.
Changes
docs/action_policy_droid_server.md— Sync the container with--group=cu130-torch213-trainrather thancu130-train, putting the policy server on torch 2.13. The image itself still bakes incu130(torch 2.10), so the container re-syncs over the baked-in wheels at startup. The documenteddocker buildalso passes--build-arg INSTALL_APEX=0, since the policy server never imports apex. Also drops the paragraph directing CUDA 12.x users tocu128-train.scripts/action_policy_server_robolab.py— Time eachRobolabPolicyService.infercall and print the elapsed milliseconds whenEVAL_VERBOSEis set, to measure server latency during RoboLab evaluation runs. Gated behind the env var so evaluation runs opt in without altering default server output.Dockerfile— Bump the apex pinbf903a2→9e3568aand add anINSTALL_APEXbuild arg. Pinbf903a2assembles its own-gencodeflags and appendsarch=compute_70,code=sm_70unconditionally, gated only by lower bounds on the CUDA version, so several extensions fail on the CUDA 13 base image withnvcc fatal : Unsupported gpu architecture 'compute_70'. Those code paths never consultTORCH_CUDA_ARCH_LIST, so no arch override can work around it; upstream has since removed the hand-built lists and derives the architectures from the detected CUDA version instead.INSTALL_APEX=0skips the build entirely, which is worth having regardless: apex is optional, its only import being thetry/except-guarded one incallbacks/norm_monitor.py.