GPU container breaks with documented+recommended options

[alfredodeza]

Select Topic Area

Bug

Body

When using GPU Codespaces instances and following the DevContainer configuration for NVIDIA to use the following flag:

{
  "runArgs": ["--gpus", "all"]
}

The container fails to start with the following logs:

2022-12-10 20:25:31.078Z: docker: Error response from daemon: failed to create shim: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #1: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
2022-12-10 20:25:31.078Z: nvidia-container-cli: mount error: file creation failed: /var/lib/docker/overlay2/a77bbe6b76ade0966f66dc9df640032701292b1b70546365b6457b9054c67457/merged/run/nvidia-persistenced/socket: no such device or address: unknown.
2022-12-10 20:25:31.078Z: Stop (743 ms): Run: docker run --sig-proxy=false -a STDOUT -a STDERR --mount type=bind,src=/var/lib/docker/codespacemount/workspace,dst=/workspaces --mount type=volume,src=minikube-config,dst=/home/vscode/.minikube --mount source=codespaces-linux-var-lib-docker,target=/var/lib/docker,type=volume --mount source=/root/.codespaces/shared,target=/workspaces/.codespaces/shared,type=bind --mount source=/var/lib/docker/codespacemount/.persistedshare,target=/workspaces/.codespaces/.persistedshare,type=bind --mount source=/.codespaces/agent/mount,target=/.codespaces/bin,type=bind --mount source=/mnt/containerTmp,target=/tmp,type=bind --mount type=bind,src=/.codespaces/agent/mount/cache,dst=/vscode -l Type=codespaces -e CODESPACES=true -e ContainerVersion=12 -e RepositoryName=mlops-template -e NVIDIA_DRIVER_CAPABILITIES=compute,utility -u codespace --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --privileged --init --gpus all --label ContainerVersion=12 --hostname codespaces-40f3d2 --add-host codespaces-40f3d2:127.0.0.1 --cap-add sys_nice --network host --gpus all --init --privileged --cap-add SYS_PTRACE --security-opt seccomp=unconfined --entrypoint /bin/sh vsc-mlops-template-eaeaa897d3f2abe8339514832bae4cfa -c echo Container started
2022-12-10 20:25:31.078Z: $ docker ps -q -a --filter label=Type=codespaces
2022-12-10 20:25:31.078Z: Stop (751 ms): Run: docker events --format {{json .}} --filter event=start
2022-12-10 20:25:31.106Z: Stop (24 ms): Run: docker ps -q -a --filter label=Type=codespaces
2022-12-10 20:25:31.106Z: $ docker inspect --type container e4aca6016151
2022-12-10 20:25:31.133Z: Stop (25 ms): Run: docker inspect --type container e4aca6016151
2022-12-10 20:25:31.133Z: Error: Command failed: docker run --sig-proxy=false -a STDOUT -a STDERR --mount type=bind,src=/var/lib/docker/codespacemount/workspace,dst=/workspaces --mount type=volume,src=minikube-config,dst=/home/vscode/.minikube --mount source=codespaces-linux-var-lib-docker,target=/var/lib/docker,type=volume --mount source=/root/.codespaces/shared,target=/workspaces/.codespaces/shared,type=bind --mount source=/var/lib/docker/codespacemount/.persistedshare,target=/workspaces/.codespaces/.persistedshare,type=bind --mount source=/.codespaces/agent/mount,target=/.codespaces/bin,type=bind --mount source=/mnt/containerTmp,target=/tmp,type=bind --mount type=bind,src=/.codespaces/agent/mount/cache,dst=/vscode -l Type=codespaces -e CODESPACES=true -e ContainerVersion=12 -e RepositoryName=mlops-template -e NVIDIA_DRIVER_CAPABILITIES=compute,utility -u codespace --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --privileged --init --gpus all --label ContainerVersion=12 --hostname codespaces-40f3d2 --add-host codespaces-40f3d2:127.0.0.1 --cap-add sys_nice --network host --gpus all --init --privileged --cap-add SYS_PTRACE --security-opt seccomp=unconfined --entrypoint /bin/sh vsc-mlops-template-eaeaa897d3f2abe8339514832bae4cfa -c echo Container started
2022-12-10 20:25:31.134Z: trap "exit 0" 15
2022-12-10 20:25:31.134Z: /usr/local/share/ssh-init.sh
2022-12-10 20:25:31.134Z: /usr/local/share/docker-init.sh
2022-12-10 20:25:31.134Z: exec "$@"
2022-12-10 20:25:31.134Z: while sleep 1 & wait $!; do :; done - /usr/local/share/docker-init.sh /usr/local/share/ssh-init.sh sleep infinity
2022-12-10 20:25:31.134Z:     at voe (/usr/lib/node_modules/@microsoft/vscode-dev-containers-cli/dist/spec-node/devContainersSpecCLI.js:1893:1355)
2022-12-10 20:25:31.134Z:     at oT (/usr/lib/node_modules/@microsoft/vscode-dev-containers-cli/dist/spec-node/devContainersSpecCLI.js:1893:1291)
2022-12-10 20:25:31.134Z:     at processTicksAndRejections (internal/process/task_queues.js:95:5)
2022-12-10 20:25:31.134Z:     at async Ioe (/usr/lib/node_modules/@microsoft/vscode-dev-containers-cli/dist/spec-node/devContainersSpecCLI.js:1899:2128)
2022-12-10 20:25:31.134Z:     at async Zf (/usr/lib/node_modules/@microsoft/vscode-dev-containers-cli/dist/spec-node/devContainersSpecCLI.js:1899:3278)
2022-12-10 20:25:31.134Z:     at async oue (/usr/lib/node_modules/@microsoft/vscode-dev-containers-cli/dist/spec-node/devContainersSpecCLI.js:2020:15162)
2022-12-10 20:25:31.134Z:     at async sue (/usr/lib/node_modules/@microsoft/vscode-dev-containers-cli/dist/spec-node/devContainersSpecCLI.js:2020:14916)
2022-12-10 20:25:31.134Z: Stop (455726 ms): Run: /usr/bin/node /usr/lib/node_modules/@microsoft/vscode-dev-containers-cli/dist/spec-node/devContainersSpecCLI.js up --user-data-folder /var/lib/docker/codespacemount/.persistedshare --container-data-folder .vscode-remote/data/Machine --container-system-data-folder /var/vscode-remote --workspace-folder /var/lib/docker/codespacemount/workspace/mlops-template --id-label Type=codespaces --log-level info --log-format json --config /var/lib/docker/codespacemount/workspace/mlops-template/.devcontainer/devcontainer.json --override-config /root/.codespaces/shared/merged_devcontainer.json --default-user-env-probe loginInteractiveShell --remove-existing-container --mount type=bind,source=/.codespaces/agent/mount/cache,target=/vscode --skip-post-create --update-remote-user-uid-default never --mount-workspace-git-root false
2022-12-10 20:25:31.134Z: Exit code 1

====================================== ERROR ====================================
2022-12-10 20:25:31.163Z: Failed to create container.
=================================================================================
2022-12-10 20:25:31.164Z: Error: Command failed: /usr/bin/node /usr/lib/node_modules/@microsoft/vscode-dev-containers-cli/dist/spec-node/devContainersSpecCLI.js up --user-data-folder /var/lib/docker/codespacemount/.persistedshare --container-data-folder .vscode-remote/data/Machine --container-system-data-folder /var/vscode-remote --workspace-folder /var/lib/docker/codespacemount/workspace/mlops-template --id-label Type=codespaces --log-level info --log-format json --config /var/lib/docker/codespacemount/workspace/mlops-template/.devcontainer/devcontainer.json --override-config /root/.codespaces/shared/merged_devcontainer.json --default-user-env-probe loginInteractiveShell --remove-existing-container --mount type=bind,source=/.codespaces/agent/mount/cache,target=/vscode --skip-post-create --update-remote-user-uid-default never --mount-workspace-git-root false
2022-12-10 20:25:31.165Z: Error Code: 1302

====================================== ERROR ====================================
2022-12-10 20:25:31.173Z: Container creation failed.
=================================================================================
2022-12-10 20:25:31.211Z: 

The full runArgs flags used are:

"runArgs": [
		"--cap-add=SYS_PTRACE",
		"--security-opt",
		"seccomp=unconfined",
		"--privileged",
		"--init",
		"--gpus", "all"
	],

Example repository where this breaks (note that gpu flag is commented out): https://github.com/nogibjj/mlopstemplate/blob/GPU/.devcontainer/devcontainer.json#L72

[thoughtfuldata]

Can anyone follow up on this? If it cant be made to work, It would be nice to have runargs specific to codespaces( that override the general runargs).