Support in tree Autoscaler in ray operator

[Jeffwan]

Controller support to scale in arbitrary pods by following api. This is extremely helpful for user who use out of tree autoscaler.

https://github.com/ray-project/ray-contrib/blob/f4076b4ec5bfae4cea6d9b66a1ec4e63680ca366/ray-operator/api/v1alpha1/raycluster_types.go#L56-L60

In our case, we still like to use in-tree autoscaler. the major differences is

1. we want to enable in-tree autoscaler in head pod via --autoscaling-config and head pod will start monitor process
2. config actually has to come from operator. So operator needs to convert RayCluster custom resource to a config file which can be used by in-tree autoscaler. example here
3. Since head and operator are hosted in different pods, operator needs to create the ConfigMap and mount to head pod transparently.

A new field has been reserved in API support this change.

https://github.com/ray-project/ray-contrib/pull/22/files#diff-edc3be4feb67012c143a57fcaefafb4c95e4cd6e661a67bb2ad1da340255bc00R21-R22

[ericl]

It would be great to see support for in-tree autoscaling! Are there any API changes to the in-tree autoscaler or proto APIs that might make this easier to implement / maintain?

(I'm happy to work together on this issue)

[Jeffwan]

@ericl We did some analysis and notice it's kind of hard to start monitor and keep it exact same pattern as it is in ray/core. I do think we need some changes to provides a smooth and pluggable experience. Let us add more details in the issue and we can have the discussion

[ericl]

Cc @DmitriGekhtman, who maintains the in-tree operator.

[DmitriGekhtman]

@Jeffwan could you say more about why having the autoscaler run in the head pod is preferable for the use-cases you are considering?

If I understand right, you'd also prefer the autoscaler to directly interact with K8s api server, rather than acting on a custom resource and delegating pod management to the operator.

Just curious if there are particular reasons this way of doing things works best for you, besides the fact that the Ray autoscaler is currently set up to favor this deployment strategy.

[DmitriGekhtman]

I guess "in-tree autoscaler" mostly means "monitor.py" from the main Ray project.
One way to make it work is to write a NodeProvider implementation whose "create node" and "terminate node" methods act on the scale fields of the RayCluster CR.

[Jeffwan]

@Jeffwan could you say more about why having the autoscaler run in the head pod is preferable for the use-cases you are considering?

@DmitriGekhtman I missed your last comment. We can scope autoscaler at the cluster level which is under our expectation. Since autoscaler in the future may have different policies etc, this gives us enough flexibility to custom autoscaler for each cluster for different ray versions. (we are not end users and version upgrade takes time, it's common to have multiple versions running at the same time in the cluster)

If I understand right, you'd also prefer the autoscaler to directly interact with K8s api server, rather than acting on a custom resource and delegating pod management to the operator.

I actually prefer to ask autoscaler to update Kubernetes CRD so there's always one owner of the pods and the responsibility is clear.

[Jeffwan]

I guess "in-tree autoscaler" mostly means "monitor.py" from the main Ray project.
One way to make it work is to write a NodeProvider implementation whose "create node" and "terminate node" methods act on the scale fields of the RayCluster CR.

That's correct. We did some POC like below to verify the functionality but feel there're some upstream changes to make. Currently, we are not using autoscaling yet in our envs.

1. CRD -> a config file autoscaler can recongnize
2. operator converts CRD to config and create a ConfigMap and mount to head node
3. head node start monitoring process and reads the config.

[DmitriGekhtman]

All of this makes sense.
I think it might be advantageous to deploy the autoscaler as a separate deployment (scoped to a single Ray cluster). That gives more flexibility. Also, it's better for resource management -- we've observed the autoscaler using up a lot of memory under certain conditions.

Mounting a config map works. Another option is to have the autoscaler read the custom resource and do the translation to a suitable format itself, once per autoscaler iteration. This has the advantage that changes to the CR propagate faster to the autoscaler -- mounted config maps take a while to update.

[pcmoritz]

I wrote a design doc fleshing out the above proposals a bit more:

https://docs.google.com/document/d/1I2CYu2-hTQUJ29wPonMvCZgEiRPs1-KeqT1mzrC6LXY

Please let us know about the direction and any suggestions or improvements you might have :)

[Jeffwan]

ray-project/ray#21086
ray-project/ray#22348

Ray upstream already have the support. Under current implementation, kuberay operator's work become easier, operator should take actions on this field to orchestrate the autoscaler. Entire process should be transparent to users

kuberay/ray-operator/api/raycluster/v1alpha1/raycluster_types.go

Lines 21 to 22 in ffa7e60

	// EnableInTreeAutoscaling indicates whether operator should create in tree autoscaling configs
	EnableInTreeAutoscaling *bool `json:"enableInTreeAutoscaling,omitempty"`

While, version management is still tricky. We should not support autoscaler for earlier Ray versions.

[DmitriGekhtman]

Yep, I agree that we don't need to support the Ray autoscaler with earlier Ray versions.

[Jeffwan]

Major implementation is done. Let's create separate issues to track future improvements.