[Docker] Add --gpus all for docker when GPU is available

[Michaelvll]

There seems no harm to add --gpus all when GPUs exist, which is a newer version of the option --runtime nvidia and seems docker's official doc is using this: https://docs.docker.com/engine/containers/resource_constraints/#access-an-nvidia-gpu

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Any manual or new tests for this PR (please specify below)
• All smoke tests: pytest tests/test_smoke.py
• Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
• Backward compatibility tests: conda deactivate; bash -i tests/backward_compatibility_tests.sh

[cblmemo]

Thanks for adding this @Michaelvll ! Will this affect the clouds that are still using sky/skylet/providers/command_runner.py? Those clouds are still using ray's docker configure code path iirc.

[cblmemo]

should we remove other --gpus all in j2 files like {azure,gcp,paperspace}-ray.yml.j2?

[Michaelvll]

Good catch! I just removed them. For those clouds still using node providers, they are not affected by the current PR and will still work as current master, which might be fine?

[cblmemo]

Oh actually, should we only remove --gpus all in clouds that support the new provisioner? For other clouds like paperspace, we should keep this in the ray file so the ray docker command runner can use this option to expose GPUs to the docker.

[Michaelvll]

paperspace is using new provisioner, and it seems none of the clouds using the old provisioned support docker as a runtime? : )

[cblmemo]

Oh good point! Then maybe we could remove the old docker command runner now.

[Michaelvll]

Good point! Let's keep it for reference for now. We can probably remove it in a separate PR.

[cblmemo]

Thanks for updating this @Michaelvll ! LGTM.