[Feature]: Fleet in-place update

[jvstme]

Problem

Updating any fleet properties requires stopping all runs that are using the fleet and recreating it, including recreating the underlying VMs in cloud fleets.

Solution

Support in-place update for all properties that could be updated without recreating the fleet.

Note that some properties could be updated without stopping or starting any instances (blocks, placement for SSH fleets, etc), while other properties may require some instances to be started or stopped (nodes, resources, etc).

In-place update for different properties can be implemented in different sub-issues.

Workaround

No response

Would you like to help us implement this feature by sending a PR?

Yes

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[r4victor]

This issue will be a part of #2973. We want to make dstack maintain instances as specified by the fleet configuration for #2921 – there will be a background task for that. The same background task can be used to implement cloud fleets in-place update – it'll consolidate current instances with the updated fleet spec, i.e. terminating instances that don't match the new fleet spec and provisioning new instances using the new fleet spec.

Notes:

• Both fleet, run apply, and run retry can add new instances to fleet – fleet retry maintains nodes.min and run apply/retry can add instances on top of that – they can also run before fleet retry, which adds another layer of non-determinism, but that should be ok since the provisioned instances would still match the fleet spec (Respect fleet spec when adding new instances on run apply #2969).