-
Notifications
You must be signed in to change notification settings - Fork 205
Closed
Description
Part of #2973
It's now possible to specify the nodes.max property for cloud fleets but it does not have an effect. dstack should enforce this limit and do not allow provisioning more than nodes.max instances in the fleet.
Implementation details
It'll be hard to guarantee that fleets never exceed nodes.max with the current processing logic without sacrificing processing throughput: tasks that create instances would need to take fleet-level lock. To avoid the complicated locking logic, I suggest we make nodes.max a soft limit – there can be more instances in the moment due to concurrent provisioning but the fleet consolidation logic will ensure the limit is respected eventually, deleting the extra instances if necessary.