Build scripts should be compiled in parallel

[runningcode]

When changing something in the buildSrc all build.gradle.kts must be recompiled. From the command line interface output, it looks like all the build.gradle.kts files are compiled sequentially instead of in parallel. This can slow down builds especially for large projects.

Expected Behavior

Expect build.gradle.kts files to compile in parallel if those modules can compile in parallel.

Steps to Reproduce

Run .gradlew assemble in multi-kotlin-project-with-buildSrc. Notice build.gradle.kts files compiling sequentially.
Change something in buildSrc/src/main/kotlin/utils.kt, notice build.gradle.kts files recompile sequentially.

I would include a build scan, but build scans don't include buildSrc compilation.

[JLLeitschuh]

This is a good thought. As is including the compilation time of the buildSrc and the build.gradle.kts files in the buildscan.

@eriwen Actually, have you considered including the compilation time for the build in the buildscan?

[eriwen]

Compilation time is captured, at least for build scripts. Example: https://scans.gradle.com/s/h2ily574bqb4g/performance/configuration

[runningcode]

Sorry, you're correct, It seems build scans do include compilation of build scripts. I have here a build scan from my production project where we migrated our build scripts to the kotlin dsl. I simply ran ./gradlew after making a small change in the buildSrc. It took nearly 20 seconds to compile the build scripts.

https://scans.gradle.com/s/umqht4zx6m3vk/performance/configuration

Is there a reason the build scripts cannot be compiled in parallel?

[runningcode]

Related to gradle/kotlin-dsl-samples#902

[LouisCAD]

Can this be prioritized?
That'd be a great performance improvement for multi-module projects.

[mathjeff]

Would this also result in parallel compilation of non-Kotlin build.gradle files?

In AndroidX, our build scans indicate that the time to compile our build.gradle's for the first build is also about 18.5s ( https://scans.gradle.com/s/dzeucmd457i2e/performance/configuration?openScriptsAndPlugins=WyJjb2RlLXVuaXQtOmJ1aWxkc3JjIWJ1aWxkLmdyYWRsZSIsImNvZGUtdW5pdC1idWlsZC5ncmFkbGUiXQ#summary-script-compile ) .

[runningcode]

@mathjeff The groovy scripts generally take less than 10 milliseconds to compile. One reason for the long compile time in that build scan you shared is that the build is not using a Gradle Daemon and therefore takes longer to start up.

[mathjeff]

That's interesting. I'm running with the daemon disabled mostly because that's the way our CI is set up for maximum reproducibility, which does give us slower builds

[udalov]

We're suffering from this in the Kotlin project itself. Every time we upgrade the bootstrap Kotlin compiler version, which happens pretty often, or someone changes anything in buildSrc, all build.gradle.kts files are recompiled sequentially, which takes 5-10 minutes for everyone in the team.

[kaushalyap]

@udalov use composite builds instead : https://proandroiddev.com/stop-using-gradle-buildsrc-use-composite-builds-instead-3c38ac7a2ab3

[udalov]

@kaushalyap Thanks for the link! I don't think this will help in the mentioned case of upgrading the bootstrap Kotlin compiler though, since we actually need to recompile everything, including buildscripts, with the new dependencies.

[CXwudi]

I came across this issue and had a workaround for it.

The idea is based on org.gradle.parallel=true. Instead of putting all precompiled script plugins in one mono Gradle project (the one Gradle project is either the buildSrc, or a composite build includeBuild() in pluginsManagement block in settings.gradle.kts), separate each precompiled script plugin in its own module in a multimodule Gradle project (again this project can be either buildSrc or a composite build includeBuild()). Also, try to prefer Composition over Inheritance to achieve maximum parallelism, see my crazy example to see how to use Composition to level up parallelism

Now, the biggest drawback of this approach is that the management overhead is huge. In IntelliJ, you potentially can see lots of modules in your Gradle view and the project view, which can also potential decrease IDE performance, but in my mind, it is worth to accept such high overhead to gain the build speed improvement from 2:30 to 1:42 (tested using gradlew build check --rerun-tasks).

[eskatos]

@CXwudi, this issue is about build scripts, not precompiled scripts

[CXwudi]

@eskatos I know but I think the same idea could be applicable

[NikolayMetchev]

Is this by any chance on the roadmap?

[bamboo]

Hi @NikolayMetchev, yes, we're not ready to commit to having this feature in a specific Gradle release yet but it is part of our active plan for improved IDE sync performance (isolated projects).

[NikolayMetchev]

Thanks @bamboo Good luck and thanks for your hard work.

[abstratt]

Related (albeit with a more general summary): #8538