Build scheduler allocates too many slaves to builds with few atoms

[josephharrington]

If a build has a single atom, it is very likely to get more than one slave assigned.

There is this method in BuildScheduler:

def needs_more_slaves(self):
        """Determine whether or not this build should have more slaves allocated to it."""
        return self._num_executors_allocated < self._max_executors and not self._build._unstarted_subjobs.empty()

This will correctly stop allocating slaves once all subjobs have been started. But because build setup takes at least a few seconds, it's very likely that many slaves will be allocated before any subjobs start. So builds with very few atoms are likely to have more slaves allocated than necessary.

This is not a huge problem, but is inefficient and could be a huge problem for builds with a long setup time.

[tjlee0909]

I see, so this seems to be a problem when:

• max_executors is set to something high, lets say 100.
• The number of subjobs/atoms is small, lets say 1.

I think we should just do a check on the total number of subjobs vs number of executors allocated, and limit it based on that (in addition to what we have). Anything more complicated is probably not worth the effort and additional complexity we'd introduce (e.g.: handling the case where one slave is burning through subjobs, but 10 other slaves are taking forever to clone the repo).

[josephharrington]

Yep, totally agree with the check you suggest. There's no point in allocating more executors than subjobs.

Just for reference, another thing that made this more likely was that the default value of max_executors is sys.maxsize.