Allow slaves to substantiate in parallel #2155
Conversation
BuildRequestDistributor's _maybeStartBuildsOnBuilder() method is decorated with @defer.inlineCallbacks. This means that when it uses "yield bldr.maybeStartBuild", the entire slave substantiation process is sequentialized at that point. slave substantiation can take considerable time, especially with something like EC2 latent slaves. Even without anything substantial happening in the cloud init (user-data) script, it can easily take a couple of minutes. If any substantial configuration of the OS takes place ins the user-data script, maybeStartBuild() can easily take many minutes. To enable parallel slave substantiation, we need to stop waiting on bldr.maybeStartBuild() inline. Instead we move the error handling code into a callback function that can be added to the Deferred that is returned by bldr.maybeStartBuild(). At least for for the EC2LatentBuildSlave class, everything else down the chain is properly event based and/or threaded. This small code change allows slaves to substantiate in parallel. Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov>
|
Something like this makes sense, but the issue with this is that the buildrequest will be claimed without a corresponding build while the worker starts up. I am finishing up a branch that will implement a solution to this that doesn't suffer from that drawback. |
|
OK, thanks. I'm new to the code; can you explain why that is a problem? It seems like the builds are associated during worker startup just as they would be previously, only now multiple will be happening in parallel. I'll look forward to seeing you branch. |
|
The code is here. I plan to submit it (and the dependencies) in several branches. |
|
@tomprince Glad you're doing this, this should be a huge improvement for EC2 latent workers. If I want to test your code, do you suggest I test the commit you linked, or is there a particular branch I should pull down? |
|
@aelsabbahy master...tomprince:latent-with-a-Build is the branch. |
BuildRequestDistributor's _maybeStartBuildsOnBuilder() method is
decorated with @defer.inlineCallbacks. This means that when it
uses "yield bldr.maybeStartBuild", the entire slave substantiation
process is sequentialized at that point.
slave substantiation can take considerable time, especially with
something like EC2 latent slaves. Even without anything substantial
happening in the cloud init (user-data) script, it can easily take
a couple of minutes. If any substantial configuration of the
OS takes place ins the user-data script, maybeStartBuild() can
easily take many minutes.
To enable parallel slave substantiation, we need to stop waiting on
bldr.maybeStartBuild() inline. Instead we move the error
handling code into a callback function that can be added to the
Deferred that is returned by bldr.maybeStartBuild().
At least for for the EC2LatentBuildSlave class, everything else down
the chain is properly event based and/or threaded. This small
code change allows slaves to substantiate in parallel.
Signed-off-by: Christopher J. Morrone morrone2@llnl.gov