Skip to content

[Bug]: Kubernetes: all node GPUs allocated when none requested #4010

Description

@un-def

Steps to reproduce

Prerequisites:

  • GPU nodes are not tainted (dstack doesn't add tolerations if no GPU requested).
  • An image with NVIDIA_VISIBLE_DEVICES=all baked into, e.g., NVIDIA images or dstackai/base (which is used when image is not set in the run configuration).
type: dev-environment
name: dev-environment
resources:
  gpu: 0
  1. dstack apply
  2. ssh dev-environment
  3. ls -1 /dev/nvidia[0-9]
  4. nvidia-smi -L

Actual behaviour

All GPUs are passed inside the container via the NVIDIA_VISIBLE_DEVICES env var path.

# ls -1 /dev/nvidia[0-9]
/dev/nvidia0
/dev/nvidia1
/dev/nvidia2
/dev/nvidia3
/dev/nvidia4
/dev/nvidia5
/dev/nvidia6
/dev/nvidia7

# nvidia-smi -L
GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-fe56c782-f8db-cc17-fa82-4286037f988b)
GPU 1: NVIDIA H100 80GB HBM3 (UUID: GPU-cba9dcf6-8c94-fa2b-1bf0-f1476941a458)
GPU 2: NVIDIA H100 80GB HBM3 (UUID: GPU-33ac7880-d9d2-c4b7-8ad1-a1c55d5124d6)
GPU 3: NVIDIA H100 80GB HBM3 (UUID: GPU-74a529a1-2939-79e1-7f54-aec2d35756c6)
GPU 4: NVIDIA H100 80GB HBM3 (UUID: GPU-cbf70ec6-5ef2-eadc-510c-ed98ee0fc6ed)
GPU 5: NVIDIA H100 80GB HBM3 (UUID: GPU-37d22e64-5838-4837-a577-916774e2edf7)
GPU 6: NVIDIA H100 80GB HBM3 (UUID: GPU-43d11db3-eb4d-5e26-0b3a-81ed92f37a86)
GPU 7: NVIDIA H100 80GB HBM3 (UUID: GPU-56aec030-3f6e-09b3-10ce-0c9834e72182)

Expected behaviour

No GPUs inside the container.

dstack version

0.20.26

Server logs

Additional information

No response

Metadata

Metadata

Assignees

Labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions