Parallel Builds

[deckar01]

In projects that have lots of merges and long build times, a synchronous build queue is not viable.

See https://gitlab.com/gitlab-org/gitlab-ce/issues/4176

Each merge request in the queue would need to be proactively rebased on the previous one before it is finished so that the builds could continue in parallel.

Downsides:

• The implementation would require rebasing the entire queue when a build fails
• The commit log and diff would contain unrelated changes (temporarily until master catches up)

[jcpetruzza]

Would definitely be nice!

There are at least two challenges I can see:

1. Dynamically adjusting to the available build capacity: you don't want to optimistically rebase 100 merge requests if there is only CI runner available for the project.
2. Making it robust

For 1. I imagine that the high-level logic could be something along the lines of: as long as you don't have one merge request with CI pipeline in pending state, try to optimistically rebase another merge request.

With 2. we have to be careful. At the moment, the bot is essentially stateless, so that if it crashes, it can just restart and continue from where it left. The problem with speculative rebasing is that your merge request may have ended up with commits that now need to be removed, but after a crash (having lost the bot's internal state), it is not obvious which commits to drop.

One thing that we could do, though, is to use a fresh branch for running the speculative build, instead of the merge request branch. Then, we only force-push to the merge-request's branch when we know that all the previous requests in the queue passed. Also, this way the commit log would not be polluted with commits that don't make it in the end.

Does this make sense? The only problem I can see is that GitLab will probably rerun the CI pipeline once we pushed to the merge request, even though there is a successful run for the very same head, which is a waste of resources. I don't think we can control this.

[deckar01]

A second branch to maintain statelessness is interesting. You can skip CI by including [skip ci] in a commit message. I don't see any down side to this other than generating an empty commit in the history.

https://docs.gitlab.com/ce/ci/yaml/README.html#skipping-jobs

[Jellby]

In the future it may be possible to skip CI with push options. See https://gitlab.com/gitlab-org/gitlab-ce/issues/18667

[JaimeLennox]

Added in #84.