[spark] Add heap_memory param for setup_ray_cluster API, and change default value of per ray worker node config, and change default value of ray head node config for global Ray cluster

[WeichenXu123]

• Add heap_memory param for setup_ray_cluster API:
  This is for advanced user to better tune memory setting with co-exist spark executor processes.

• Change default value of per ray worker node config:
  change default value of num_cpus_worker_node to number of CPU cores of spark worker node.
  change default value of num_gpus_worker_node to number of GPUs of spark worker node.
  This is a more optimal setting in most cases, compared with previous default setting.

• Change default value of Ray head node config for Ray global mode cluster:
  change default value of num_cpus_head_node to number of CPU cores of spark driver node.
  change default value of num_gpus_head_node to number of GPUs of spark driver node.
  This is a more optimal setting in most cases, compared with previous default setting.

Why are these changes needed?

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[WeichenXu123]

CC @jjyao

[WeichenXu123]

CC @jjyao

btw, shall I name the new param "heap_memory_worker_node" or "memory_worker_node" ?

[jjyao]

lg

[jjyao]

Why is that?

[WeichenXu123]

The reason is:

in this PR I change num_cpus_worker_node and num_gpus_worker_node default values to "maximum" CPU / GPU cores of spark worker node:

Assuming the spark worker node instance has 4 CPUs and 2 GPU,
then if user only set one param like:

setup_ray_cluster(num_cpus_worker_node=1)

then num_gpus_worker_node default values still use 2, then it causes 3 CPU cores in this node wasted, because once the shape (1 CPUs/ 2 GPUs) ray node launched in this spark worker, no other ray worker node of the same shape can be launched in the same spark worker.

So I only allow 2 cases:

1. num_cpus_worker_node / num_gpus_worker_node are generated by default, to use all CPU /GPU cores in the spark worker.
2. Let user decide both of num_cpus_worker_node and num_gpus_worker_node, then user will set proper value e.g. (num_cpus_worker_node=2, num_gpus_worker_node=1) which can fit 4 CPU cores 2 GPU cores spark worker shape well.

[jjyao]

I feel it's better to allow users to set them separately since they are two independent resources. If you want to avoid wasted resources, we can print some warning message. (What if the waste is on purpose that user wants to leave some resources for spark jobs)?

[WeichenXu123]

CC @jjyao