A "soft" conflicts attribute

[johanneswuerbach]

Today fleet is scheduling multiple instances of an app evenly distributed in your cluster. The problem arises when an instances is reboot and all units were scheduled away. At this point all new units are scheduled to this specific instance until it is "full" again. When you don't use Conflicts you will loose any redundancy in this case. I can't use Conflicts, because I need more units of the same app then I have machines in the cluster.

A solution for this could be a soft conflicts attribute, where fleet generally tries to schedule units onto different machines, but doesn't fail when all machines are already running one copy of this unit.

[jonboulle]

@johanneswuerbach do you think the proposal outlined in #945 would cover your use case? It essentially would provide for an even weaker Conflicts, operating at a higher level.

[johanneswuerbach]

@jonboulle no, as it has the same issue as the current scheduling logic. When a machine is rebooted, all new units will be scheduled to the just rebooted machine as it's empty and doesn't have any used shares.

I basically need a "spread my units (of the same app) as much as possible over the cluster" flag.

[epipho]

👍
This would be very useful to us as well.

A few possible ideas for new fleet unit options

MinOf=glob - Schedules the unit on the machine that has the least number of units that match the glob.
MinMachines=n - Ensures that jobs specified by MinOf exist on at least n different hosts before reverting to the normal scheduler.

[dbason]

Couldn't you use metadata to do this? Split your available machines into availability zones with appropriate metadata tags and then schedule based on metadata?

[johanneswuerbach]

@dbason wouldn't this be the same as using Conflicts and x different versions of your unit files? This issue is about not having to hardcode the distribution into the unit files, but instead provide a fleet option to schedule a unit like conflicts, just without the not scheduling when it's already running everywhere.

[dbason]

@johanneswuerbach no it's not the same. If for example if you have 2 unit files, a & b and for whatever reason a gets scheduled across the cluster first and ends up putting a unit on each machine in the cluster then b will never be able to be scheduled because it conflicts.

Using the metadata method a will get spread between its availability zone as will b. Yes this requires some more redundancy planning and separate unit files but I think the trade off of simplicity in the scheduling engine is worth it. I see redundancy as a higher level concern so think it's more appropriate for this to be dealt with in an orchestration layer like deis, where as fleet should me a lower level scheduler.

[maccman]

This would be awesome! +1