With more and more CPU cores available on developer desktops and CI servers, it is important that Gradle is able to fully utilise these processing resources. By executing separate project builds within a multiproject build in parallel, we provide a relatively simple mechanism to better utilise available CPU resources.
Parallel execution will involve some additional decoupling of the projects at execution time, with all execution-time inter-project dependencies requiring explicit declaration. Projects will get a fail-fast copy of the multi-project model at execution time, that will prohibit direct, undeclared access to the model of other projects. This additional decoupling of projects through the explicit declaration of inputs and outputs will make project interaction simpler, and should allow us to make multiproject execution more efficient by doing up-to-date checking at the project level and reusing project outputs directly.
- Reduce total build time for a multi-project build where execution is IO bound or otherwise does not consume all available CPU resources
- Provide faster feedback for execution of small projects without awaiting completion of other projects
User visible changes
The primary user-visible change will be a new command-line switch and/or DSL element to allow a multi-project build to take advantage of parallel execution. [Example TBD] However, only projects that do not interact with each-other at execution-time will be able to run in parallel: if any project reads/writes the model of another project during execution the build will fail.
When multiple projects are executed in parallel, it is important that the build output makes it easy to comprehend what happened in each project. Users will see the execution output for each build clearly separated in the console output. The console will also provide an indication of the various project tasks that are presently executing, in a transient format.
In many cases, a ProjectA will depend on artifacts that are generated by another ProjectB where these artifacts are not explicitly published by ProjectB. An example is a version number generated by a subproject and consumed by all other subprojects, or the raw test output from a set of projects which are then aggregated by a separate aggregating project into a coherent test report.
We plan to introduce a number of new features that will make it easy to support these common use cases. (Design TBD)
- Declaring a simple value as a project output, and depending on that value as a project dependency
- Declaring a file or directory as a project output, and depending on that value as a project dependency
Integration test coverage
We will introduce new GradleExecuter implementation that chooses the parallel executer, and run the full coverage build with this executer. In addition to this, we should beef up our coverage for multi-project build scenarios: Happy-day:
- Multiple, unrelated projects
- A depends on published artifact of B
- A->[B,C] where B->C
- A->[B,D] where B->C->D
- A->[B,C] where B->D and C->D Sad-day:
- Project dependency cycle: A->B->C->A
- Project dependency fails: A->B and B fails
- One project dependency fails: A->[B,C] and C->D and B fails
- Execution of a project dependency times out (what does this mean?)
- For separate build processes: lots of separate testing for processes failed to launch, process dies, cannot communicate with process, etc
This will be a new executer implementation, that should be a plug-in replacement for the existing implementation. This will require some refactoring of the GradleTaskExecuter to extract common functionality, but once done there should be little or no further impact on the existing functionality. The new executer will not support all of the existing features permitted in a multi-project build, and should fail-fast whenever a project attempts to directly access the model of another project during execution. Any new features for declaring project inputs/outputs will be available to default and parallel build execution.
This feature only deals with performing the execution phase of independent projects in parallel. Each project build will be executed in a different thread or process on the same machine. Out of scope for this feature:
- Distributing parallel execution across multiple machines
- Distributing execution on different OS/JVM environments
- Parallel execution of tasks within a single project
- Allowing decoupled projects to switch between downloading binary artifacts and rebuilding artifacts from source for dependencies
- Configuration of projects in parallel