[Feature] Autodetect GPU resources to advertise to Ray.

[DmitriGekhtman]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

For Ray pods configured to use GPU using one of the standard device drivers, read the number of GPUs from resource limits and automatically add that number to the --num-gpus argument of ray start, when num-gpus is not already specified.

The implementation should be straightforward and similar to existing CPU detection logic.
See the discussion in ray-project/ray#20265

Use case

This will simplify configuration for GPU workloads.

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[akanso]

We have seen cases where the users trick Ray into thinking there are more CPU --num-cpus than the actual cores allocated to the pod/container.

We also already have the raystartparams section where the users can set the num of cpus and gpus and those values will be translated into start --flags

I am fine with auto-detecting the CPU and GPU from the resource limits. One option is to add an auto-detect field that defaults to true to give the users the flexibility of not using auto-detect.

[DmitriGekhtman]

+1 for flexibility to set the ray start parameters yourself

For common/default use cases, it's better not to require redundant info.

[DmitriGekhtman]

We've covered this one.