[Data][Train] Allow manually setting resource limits for training jobs

[raulchen]

Why are these changes needed?

Previously it's not possible to manually set data resource limits for training jobs.
Because we automatically set "exclude_resources" for training jobs, and "exclude_resources" and "resource_limits" cannot be both set.

This PR fixes this issue by not automatically setting "exclude_resources" when "resource_limits" are already set by the user

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[woshiyyya]

Thanks Hao!

Two questions here:

• currently it seems only support excluding num_gpu and num_cpu, how could we ensure that those excluded gpus and cpus are colocated? (instead of excluding x cpu per node)
• Is it a good idea to provide a flag for the train users to exclude all resources of the training nodes?

[raulchen]

currently it seems only support excluding num_gpu and num_cpu, how could we ensure that those excluded gpus and cpus are colocated? (instead of excluding x cpu per node)

Data scheduler doesn't distinguish the resources by nodes right now. It only looks at the global resource usage.

Is it a good idea to provide a flag for the train users to exclude all resources of the training nodes?

This is a good idea, I found it useful as well. But to do that, we also need to config training worker resource to take all resources from the training nodes. Otherwise, data tasks will still be scheduled on the training nodes. we can discuss more later.