[REQUEST] Redistribute deployment

[jordan-rash]

Given the following situation:

1. Start a single wasmcloud host
2. Deploy a multi-part application via wadm or wash
3. Start additional hosts in lattice
4. [REQUEST] Normalize workload

In conjunction with #693, I would like to be able to "normalize" the workload when a new host comes online. At the moment, that can be done with wadm and a hack (wash scale 0) but this isn't ideal and there is no way to redistribute wash start based assets

Ref: https://wasmcloud.slack.com/archives/C04RVMJJH17/p1695735058510099

[connorsmith256]

Can you describe more about what it means to normalize a workload when a new host starts? I can imagine several behaviors that could be/are supported by wadm today, but I'm not sure we're thinking of the same thing.

[jordan-rash]

Image you have a host and you deploy pet clinic (5 actors and 2 providers)....
The next day, you notice you're not happy with resource utilization on your host, so you add a second host to your cluster.

How are you going to move loads to the second host (that isnt manual)???

With something managed by wadm with spreadscaler, you can wash scale 0 all the resources and let wadm do its things....this feels like a hack IMO

With items you have started with wash, you have no options but to manually stop things and start them over.

I am thinking of a command that looks at the lattice and ensures hosts are being equally utilized

wash host help-a-neighbor

[connorsmith256]

I think if you're using either the spreadscaler or daemonscaler with a wadm app, wadm will take care of this for you, right @brooksmtownsend?

My understanding:

• With the spreadscaler, wadm will become aware of the new host(s), and if the new host(s) cause the portion of components to become "unbalanced" with respect to labels monitored by the scaler, wadm will stop and start resources as appropriate
• With the daemonscaler, wadm will become aware of the new host(s) and start the appropriate components on the new host(s)

[brooksmtownsend]

With the spreadscaler, wadm will only stop/start new resources if the weights become unbalanced, so it's actually a pretty difficult case to trigger and still have the spreadscaler be happy in the first place.

The daemonscaler would definitely help in this scenario, since it would see the new matching host and then start additional replicas on that host. That sounds like the best thing for redistributing the deployment and this would also let you remove the old host without affecting any newly deployed resources.

I hesitate to add explicit management of workloads to wash or wasmCloud because it would require them to compute requirements just like we do in wadm. I'm hoping the Daemonscaler is the correct answer for your thoughts here @jordan-rash 😄

[jordan-rash]

The Daemonscaler isn't what I would be looking for in the case, but its fine if thats the support the platform will offer. I think it is natural to not want to run (for example) 3 copies of something on every node (daemonscaler) but to want to 3 copies in total (speadscaler) but move them as the operator sees fit. Maybe what I am asking for is a wash shift actor/provider [host-id] [host-id] functionality.

[connorsmith256]

Gotcha. I think that comes down to:

move them as the operator sees fit

If you could specify a manifest that used a spreadscaler with labels, and we had #692, would that work for your use case? A command like shift is higher-level, which could get tricky: does it check for/alter manifests? If we "just" move the workload, wadm might move it back

[connorsmith256]

@jordan-rash bump :)

[connorsmith256]

@jordan-rash closing for now since we haven't had discussion in a while