Skip to content

Respect nodes.max fleet property #3056

@r4victor

Description

@r4victor

Part of #2973

It's now possible to specify the nodes.max property for cloud fleets but it does not have an effect. dstack should enforce this limit and do not allow provisioning more than nodes.max instances in the fleet.

Implementation details

It'll be hard to guarantee that fleets never exceed nodes.max with the current processing logic without sacrificing processing throughput: tasks that create instances would need to take fleet-level lock. To avoid the complicated locking logic, I suggest we make nodes.max a soft limit – there can be more instances in the moment due to concurrent provisioning but the fleet consolidation logic will ensure the limit is respected eventually, deleting the extra instances if necessary.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions