Skip to content

Update Cuda runtime from 12.8 to 13.0 in dstack base Docker images#4007

Merged
r4victor merged 8 commits into
masterfrom
issue_3998_cuda_13
Jul 6, 2026
Merged

Update Cuda runtime from 12.8 to 13.0 in dstack base Docker images#4007
r4victor merged 8 commits into
masterfrom
issue_3998_cuda_13

Conversation

@r4victor

@r4victor r4victor commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Closes #3998

  • Update Cuda runtime from 12.8 to 13.0 in dstack base Docker images by building from nvidia/cuda:13.0.3.
  • Update NCCL_VERSION=2.28.3-1
  • Pins a new dstack base Docker image version 0.15.

Cuda 13.0 requires nvidia host driver >= 580. Many dstack backends where dstack does not control the host drivders (e.g. Lambda, Crusoe, VastAI, Runpod) may provision hosts with older driver versions. But Cuda 13.0 still works since nvidia/cuda:13.0.3 includes cuda-compat-13 providing compatibility for all drivers >= 535. See https://docs.nvidia.com/deploy/cuda-compatibility/forward-compatibility.html. Previous dstack Docker images didn't include cuda-compat-13 so Cuda 13 workloads didn't work out-of-the-box on pre 580 hosts. Now they work.

Backward incompatible changes

  • V100 provisioning is still supported on some backends (e.g. GCP) but requires a custom Docker image or post-installing a compatible (<13) Cuda runtime inside run commands as Cuda 13.0 dropped support for Volta and older generations.
  • Drop P100 provisioning from GCP and OCI.

@r4victor r4victor changed the title Issue 3998 cuda 13 Update Cuda runtime from 12.8 to 13.0 in dstack base Docker images Jul 6, 2026
@r4victor r4victor merged commit 971b866 into master Jul 6, 2026
24 checks passed
@r4victor r4victor deleted the issue_3998_cuda_13 branch July 6, 2026 10:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update cuda in dstack base Docker images to 13.0

1 participant