The Kotlin and Groovy DSLs should have similar performance characteristics

[eskatos]

Expected Behavior

The Kotlin and Groovy DSLs have as much as possible similar performance characteristics.
IOW there's no outlier use case for which the performance characteristics make using one or the other DSL a penalty.

Current Behavior

What follows is the current state of affairs as Gradle 6.8 for four use cases:

• first use (e.g. ephemeral CI agent or clean workstation without build cache)
• buildSrc abi change
• buildSrc non-abi change
• nothing changes

Data for Gradle 6.8

24 Jan 2025 Update (@oleg-nenashev ): As the Gradle's TeamCity instance is no longer public, the performance reports are no longer accessible. The structure has also changed since 6.8. See #32164

The data below comes from performance tests run on the Gradle CI. Those tests use a build with a large number of projects and build scripts, the build scripts being rather simple, and they run on both the Groovy and Kotlin DSL for comparison.

Old Links

Use case	Groovy	Kotlin	Differences
First use	🟢 33.5s	🔴 76.2s	Groovy DSL is 2.2x faster
buildSrc abi change	🟢 13.2s	🔴 42.3s	Groovy DSL is 3.2x faster
buildSrc non-abi change	🔴 13s	🟢 5.2s	Kotlin DSL is 2.5x faster
Nothing changes	🔵 1.7s	🔵 1.8s	Similar performance

The numbers above are, at the time of writing, for current release branch of Gradle that is the upcoming 6.8.1. See:

• https://builds.gradle.org/buildConfiguration/Gradle_Check_PerformanceTestSlowLinux_Trigger/40748694?buildTab=report_project941_Performance
• https://builds.gradle.org/buildConfiguration/Gradle_Check_PerformanceTestTestLinux_Trigger/40757415?buildTab=report_project944_Performance

This issue is a follow up to gradle/kotlin-dsl-samples#902 which resides in an archived repository.

One of the reasons the Kotlin DSL is slower when it is is the time it takes to the Kotlin compiler to compile the scripts.
See https://youtrack.jetbrains.com/issue/KT-24668

[kawpii]

thanks for the benchmark @eskatos

Any issues filed on the youtrack issue tracker against Kotlin compiler related to kotlin dsl performance? if so please link the associated issues.

[LifeIsStrange]

random thoughts from a lambda Kotlin dev:

It could be a useful addition to add a variant for kotlin and groovy on this benchmark:
That could be called e.g Kotlin-optimized and that would enable non-default gradle.properties such as

org.gradle.vfs.watch=true
kapt.use.worker.api=true
kapt.include.compile.classpath=false
org.gradle.parallel=true
org.gradle.caching=true

(btw its unclear whether kapt.include.compile.classpath=false is still useful in 6.8 given compiler avoidance)

Also worth exploring would be to benchmark with the new compiler backend and the new compiler frontend

finally setting a continuous benchmark (in CI) of this would be useful for showing progress and for catching regressions!

note1: I cannot see the linked benchmark results, Login as guest doesn't seems to work (I don't have an account)
note2: reading the guide on how to improve gradle performance:

The Gradle Java plugin allows you to run the compiler as a separate process by using the following configuration for any JavaCompile task:
This process is reused for the duration of a build, so the forking overhead is minimal. The benefit of forking is that the memory-intensive compilation happens in a different process, leading to much less garbage collection in the main Gradle daemon. Less garbage collection in the daemon means that Gradle’s infrastructure can run faster, especially if you are also using --parallel.
It’s unlikely to be useful for small projects, but you should definitely consider it if a single task is compiling close to a thousand or more source files together.

https://docs.gradle.org/nightly/userguide/performance.html#compiler_daemon
This optimization doesn't seems to be provided for Kotlin currently

note3: minor: I wonder whether using the Kotlin BOM incurs a slowdown vs not using it

[gildor]

maybe worth adding one more case, change build.gradle file (not src), I think it even more common comparing to buildSrc

[eskatos]

Guest access to our CI is currently temporarily disabled, it'll be enabled back soon.

@kaushalyap there's already a link to https://youtrack.jetbrains.com/issue/KT-24668. I'm not aware of other Youtrack issues related to this.

@LifeIsStrange this issue is about the Gradle Kotlin DSL performance, not the Kotlin Gradle Plugin performance in general. For instance KAPT is out of the picture. Also, the embedded Kotlin compiler used internally by the Gradle Kotlin DSL isn't ready to be moved to the new compiler infrastructure. The numbers above represent the default Gradle user experience which is a good thing.

@gildor yes this use case is missing from the set above and I agree it's very common. We started with the buildSrc use case because it potentially invalidates a lot of things and shows the worst situation.

[kawpii]

@kaushalyap there's already a link to https://youtrack.jetbrains.com/issue/KT-24668. I'm not aware of other Youtrack issues related to this.

@eskatos I know about that issue. Hope Kotlin team close that issue in the first half of this year with the movement to new compiler as mentioned in the roadmap, so android devs can move to kotlin dsl for android projects without increased build time.

[gildor]

@kaushalyap This is the new backend of the compiler, it has nothing to do with compilation speed. You probably talk about the new frontend of the compiler, which promises up to 2 times faster compilation, but it will not be ready in 6 months, in the best case it would be some experimental version available this year

[cristan]

I wonder if this is still an issue. Gradle 7 has been out for a while and it boosts massive performance improvements for Kotlin DSL script compilation. An updated benchmark would be very appreciated :).

[eskatos]

Those benchmarks are automated and constantly run against Gradle as it evolves. You can browse the results on our CI, e.g. here https://builds.gradle.org/buildConfiguration/Gradle_Master_Check_PerformanceTestSlowLinux_Trigger/46861573?buildTab=report_project55641_Performance

Here I captured the current state:

Use case	Groovy	Kotlin	Differences
First use	🟢 34s	🔴 57.5s	Groovy DSL is 1.7x faster
buildSrc abi change	🟢 20.5s	🔴 32s	Groovy DSL is 1.6x faster
buildSrc non-abi change	🔴 21s	🟢 4.3s	Kotlin DSL is 4.8x faster

Things indeed got better since Gradle 7, especially the non-abi change scenarios. The performance isn't similar though.
The Groovy DSL got slower in general since Groovy 3.
The Kotlin DSL is still almost twice slower in first use and abi-change scenarios.
The Groovy DSL is now much slower than the Kotlin DSL in a non-abi change scenario thanks to compilation avoidance introduced in the Kotlin DSL. This can be added to the Groovy DSL too to make it faster in such scenarios, see #14829.

[kawpii]

@eskatos Thanks for the update, I wonder what would be the build times for composite builds for the same project?

There is an issue reporting converting buildSrc to implicit included.

[rowi1de]

Could you explain what ABI / Nop-ABI means 🙈