This page explains how to reproduce the Gradle vs Bazel performance numbers yourself. For that, you need to install the Gradle profiler, a tool which will automate benchmarking by running several builds in a row, with the appropriate options (it also runs Bazel builds). For non-incremental scenarios, you might want alternatively want to use a tool like hyperfine.
Our performance comparison uses 4 test projects:
- a large, monolithic app
- single project
- 50000 source files and 50000 test source files
- a small sized, multi-project build
- 10 subprojects
- each subproject has 100 source files and 100 test source files
- a medium sized, multi-project build
- 100 subprojects
- each subproject has 100 source files and 100 test source files
- a large, multi-project build
- 500 subprojects
- each subproject has 100 source files and 100 test source files
They can be found in the Gradle repository. To generate a test project, checkout the Gradle sources, then execute:
./gradlew largeMonolithicJavaProject
for the large monolithic build./gradlew mediumJavaMultiProject
for the medium multiproject build./gradlew smallJavaMultiProject
for the small multiproject build./gradlew largeJavaMultiProject
for the large multiproject build
The test projects will then be found in subprojects/performance/build/<test project name>
.
All benchmarks are run with a similar setup. Beware that due to tool differences, some scenarios may not be perfectly comparable. We try to align the scenarios as much as possible to keep them related.
Given all build tools are implemented in Java, runtime performance can vary greatly depending on class caches, JVM load times and JIT compiling. To reduce the noise of that and to have more stable measurements, all benchmarks have been run with
- 10 warmup rounds (not measure)
- 10 measurement rounds
For all benchmarks, we used a machine with 8 cores and configured the tools accordingly
- Gradle: org.gradle.workers.max=8 (e.g. in
gradle.properties
) - Maven:
-T 8
- Bazel:
--jobs 8
For the benchmarks involving remote caches, we choose the following, all residing in the same data center
- Gradle: Gradle Enterprise 2020.2 (Frankfurt)
- Maven: Gradle Enterprise 2020.2 (Frankfurt)
- Bazel: Google Cloud Storage Bucket (
europe-west3
Frankfurt)
Gradle builds should be configured to use the newly available feature in Gradle 6.4 (officially released in 6.5) using
org.gradle.unsafe.instant-execution=true
org.gradle.unsafe.vfs.retention=true
- Full Build
- No prior project state (no output files, no caches, usually the result of runing the tool-specific
clean
) - Uses warm daemons (Gradle/Bazel) due to warmup rounds
- Needs to ensure to have a clean state before each measurement round (e.g.
clean
orclear-project-cache-before
in the scenarios file) - Includes compile (
*.class
), assemble (*.jar
) and running tests - Measure the time using the CLI
- No prior project state (no output files, no caches, usually the result of runing the tool-specific
- Incremental builds
- Makes a single change (ABI or non-ABI)
- Uses up-to-date state to determine what to rerun
- No need to clean between builds
- Measures
assemble
time - Measure the time using the tooling API (used by IDEs)
abiChange {
tasks = ["assemble"]
apply-abi-change-to = "project0/src/main/java/org/gradle/test/performance/smalljavamultiproject/project0/p0/Production0.java"
system-properties {
org.gradle.unsafe.instant-execution = "true"
org.gradle.unsafe.vfs.retention = "true"
}
bazel {
targets = ["build", "//..."]
}
}
Make sure you have installed the Gradle Profiler and that it's available on your PATH.
The generated projects may have different defaults so please ensure to set up the projects as described in "Setup".
Change directory to a test project and run the following commands. This will execute the abiChange
scenario.
gradle-profiler --gradle-version 6.4 --benchmark --scenario-file performance.scenarios abiChange
gradle-profiler --benchmark --scenario-file performance.scenarios abiChange --bazel