[autoscaler] Auto detect memory resource

[ConeyLiu]

Why are these changes needed?

In the current autoscaler, we could auto detect cpu/gpu resources. However, we could not know the memory size when min_workers set to zero. This could lead to those tasks with memory requirements that could not be satisfied forever.

This patch adds memory auto detecting for k8s and aws.

Related issue number

Closes #14553

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[ConeyLiu]

cc @ericl, this problem exists in both ray 1.1.0 and 1.2.0.

[ericl]

Will this also override the object_store_memory settings on the node when it starts? This is a little dangerous since the safe value for object_store_memory depends on factors such as the size of /dev/shm, and is in general un-knowable until the node is actually up. Since it's not allowed to use object_store_memory as a resource request any more, I think we should drop that from this PR at least.

Setting "memory" is relatively safe since it's only a scheduling hint.

[wuisawesome]

Left a few comments. Can we also update AutoscalingConfigTest.testValidateDefaultConfig and AutoscalingConfigTest.testValidateDefaultConfigAWSMultiNodeTypes

[ConeyLiu]

Hi @wuisawesome, thanks for reviewing.

Can we also update AutoscalingConfigTest.testValidateDefaultConfig and AutoscalingConfigTest.testValidateDefaultConfigAWSMultiNodeTypes

You mean to add more tests? testValidateDefaultConfigAWSMultiNodeTypes has updated. Not sure what should do for testValidateDefaultConfig

[DmitriGekhtman]

Thanks for fixing this!

[ConeyLiu]

We only need to autodetect the worker node type memory resources. Because the head node type memory can be updated from runtime. And also the head node type memory is total_memory * (1 - REDIS_PROPORTION - OBJECT_STORE_PROPORTION), however the worker node type is total_memory * (1 - OBJECT_STORE_PROPORTION).

[AmeerHajAli]

@wuisawesome @ericl @DmitriGekhtman , do the new changes look good to you?

[DmitriGekhtman]

@wuisawesome @ericl @DmitriGekhtman , do the new changes look good to you?

K8s part looks good.

[AmeerHajAli]

Thanks @ConeyLiu for this great contribution! We greatly value that. We look forward to more PRs from you!

[ConeyLiu]

thanks all.