The Kotlin and Groovy DSLs should have similar performance characteristics

[eskatos]

Expected Behavior

The Kotlin and Groovy DSLs have as much as possible similar performance characteristics.
IOW there's no outlier use case for which the performance characteristics make using one or the other DSL a penalty.

Current Behavior

What follows is the current state of affairs as Gradle 6.8 for four use cases:

• first use (e.g. ephemeral CI agent or clean workstation without build cache)
• buildSrc abi change
• buildSrc non-abi change
• nothing changes

The data below comes from performance tests run on the Gradle CI. Those tests use a build with a large number of projects and build scripts, the build scripts being rather simple, and they run on both the Groovy and Kotlin DSL for comparison.

Use case	Groovy	Kotlin	Differences
First use	🟢 33.5s	🔴 76.2s	Groovy DSL is 2.2x faster
buildSrc abi change	🟢 13.2s	🔴 42.3s	Groovy DSL is 3.2x faster
buildSrc non-abi change	🔴 13s	🟢 5.2s	Kotlin DSL is 2.5x faster
Nothing changes	🔵 1.7s	🔵 1.8s	Similar performance

Context

The numbers above are, at the time of writing, for current release branch of Gradle that is the upcoming 6.8.1. See:

• https://builds.gradle.org/buildConfiguration/Gradle_Check_PerformanceTestSlowLinux_Trigger/40748694?buildTab=report_project941_Performance
• https://builds.gradle.org/buildConfiguration/Gradle_Check_PerformanceTestTestLinux_Trigger/40757415?buildTab=report_project944_Performance

Use Login as guest and note that these links will eventually get outdated.

This issue is a follow up to gradle/kotlin-dsl-samples#902 which resides in an archived repository.

One of the reasons the Kotlin DSL is slower when it is is the time it takes to the Kotlin compiler to compile the scripts.
See https://youtrack.jetbrains.com/issue/KT-24668

[kaushalyap]

thanks for the benchmark @eskatos

Any issues filed on the youtrack issue tracker against Kotlin compiler related to kotlin dsl performance? if so please link the associated issues.

[LifeIsStrange]

random thoughts from a lambda Kotlin dev:

It could be a useful addition to add a variant for kotlin and groovy on this benchmark:
That could be called e.g Kotlin-optimized and that would enable non-default gradle.properties such as

org.gradle.vfs.watch=true
kapt.use.worker.api=true
kapt.include.compile.classpath=false
org.gradle.parallel=true
org.gradle.caching=true

(btw its unclear whether kapt.include.compile.classpath=false is still useful in 6.8 given compiler avoidance)

Also worth exploring would be to benchmark with the new compiler backend and the new compiler frontend

finally setting a continuous benchmark (in CI) of this would be useful for showing progress and for catching regressions!

note1: I cannot see the linked benchmark results, Login as guest doesn't seems to work (I don't have an account)
note2: reading the guide on how to improve gradle performance:

The Gradle Java plugin allows you to run the compiler as a separate process by using the following configuration for any JavaCompile task:
This process is reused for the duration of a build, so the forking overhead is minimal. The benefit of forking is that the memory-intensive compilation happens in a different process, leading to much less garbage collection in the main Gradle daemon. Less garbage collection in the daemon means that Gradle’s infrastructure can run faster, especially if you are also using --parallel.
It’s unlikely to be useful for small projects, but you should definitely consider it if a single task is compiling close to a thousand or more source files together.

https://docs.gradle.org/nightly/userguide/performance.html#compiler_daemon
This optimization doesn't seems to be provided for Kotlin currently

note3: minor: I wonder whether using the Kotlin BOM incurs a slowdown vs not using it

[gildor]

maybe worth adding one more case, change build.gradle file (not src), I think it even more common comparing to buildSrc

[eskatos]

Guest access to our CI is currently temporarily disabled, it'll be enabled back soon.

@kaushalyap there's already a link to https://youtrack.jetbrains.com/issue/KT-24668. I'm not aware of other Youtrack issues related to this.

@LifeIsStrange this issue is about the Gradle Kotlin DSL performance, not the Kotlin Gradle Plugin performance in general. For instance KAPT is out of the picture. Also, the embedded Kotlin compiler used internally by the Gradle Kotlin DSL isn't ready to be moved to the new compiler infrastructure. The numbers above represent the default Gradle user experience which is a good thing.

@gildor yes this use case is missing from the set above and I agree it's very common. We started with the buildSrc use case because it potentially invalidates a lot of things and shows the worst situation.

[kaushalyap]

@kaushalyap there's already a link to https://youtrack.jetbrains.com/issue/KT-24668. I'm not aware of other Youtrack issues related to this.

@eskatos I know about that issue. Hope Kotlin team close that issue in the first half of this year with the movement to new compiler as mentioned in the roadmap, so android devs can move to kotlin dsl for android projects without increased build time.

[gildor]

@kaushalyap This is the new backend of the compiler, it has nothing to do with compilation speed. You probably talk about the new frontend of the compiler, which promises up to 2 times faster compilation, but it will not be ready in 6 months, in the best case it would be some experimental version available this year

[cristan]

I wonder if this is still an issue. Gradle 7 has been out for a while and it boosts massive performance improvements for Kotlin DSL script compilation. An updated benchmark would be very appreciated :).

[eskatos]

Those benchmarks are automated and constantly run against Gradle as it evolves. You can browse the results on our CI, e.g. here https://builds.gradle.org/buildConfiguration/Gradle_Master_Check_PerformanceTestSlowLinux_Trigger/46861573?buildTab=report_project55641_Performance

Here I captured the current state:

Use case	Groovy	Kotlin	Differences
First use	🟢 34s	🔴 57.5s	Groovy DSL is 1.7x faster
buildSrc abi change	🟢 20.5s	🔴 32s	Groovy DSL is 1.6x faster
buildSrc non-abi change	🔴 21s	🟢 4.3s	Kotlin DSL is 4.8x faster

Things indeed got better since Gradle 7, especially the non-abi change scenarios. The performance isn't similar though.
The Groovy DSL got slower in general since Groovy 3.
The Kotlin DSL is still almost twice slower in first use and abi-change scenarios.
The Groovy DSL is now much slower than the Kotlin DSL in a non-abi change scenario thanks to compilation avoidance introduced in the Kotlin DSL. This can be added to the Groovy DSL too to make it faster in such scenarios, see #14829.

[kaushalyap]

@eskatos Thanks for the update, I wonder what would be the build times for composite builds for the same project?

There is an issue reporting converting buildSrc to implicit included.

[rowi1de]

Could you explain what ABI / Nop-ABI means 🙈

[eskatos]

@rowi1de sure thing!

ABI stands for Application Binary Interface.

An ABI change is a change that modifies the ABI, iow the "public api" that downstream consumers compile against. All consumers need to be recompiled.

A non-ABI change is a change that does not modify the ABI, iow only touches the "private implementation". Consumers compilation can be skipped.

[blundell]

@eskatos clicked your like this one and after signing in a guest I get:

Build with id "46861573" does not exist

( was hoping to check the latest benchmarks :) )

[eskatos]

@blundell that individual run got garbage collected, here you can see the list of runs https://builds.gradle.org/buildConfiguration/Gradle_Master_Check_PerformanceTestSlowLinux_Trigger?branch=master&buildTypeTab=overview&mode=builds

[blundell]

I think I've done this right :D

Gradle 7.4/7.5 numbers
(buildSrc = 7.4, First use = 7.5)

Use Case	Groovy	Kotlin	Difference
First use	🟢 38.855s	🔴 63.54s	Groovy DSL is 1.6x faster
buildSrc abi change	🟢 25.307	🔴 35.014s	Groovy DSL is 1.4x faster
buildSrc non-abi change	🔴 24.526s	🟢 4.732s	Kotlin DSL is 5x faster

Info from 11th Feb 2022 CI run

For example, from the performance tab, I looked up the buildSrc abi change and took the first row, here.
Kotlin:

Groovy:

and the first use numbers from here.
Groovy:

Kotlin:

[eugene-krivobokov]

@blundell , could you please explain the difference calculations?

In the previous results: 21s VS 4.3s - 4.8x faster. I get it.
In the last results: 24.526s VS 4.732s - 135% faster. How have these percents been calculated?

[blundell]

@eugene-krivobokov Sorry about that, wrong equation. Updated to calculate as a percentage.

[mochadwi]

@blundell that individual run got garbage collected, here you can see the list of runs https://builds.gradle.org/buildConfiguration/Gradle_Master_Check_PerformanceTestSlowLinux_Trigger?branch=master&buildTypeTab=overview&mode=builds

does it private? seems unable to open with my team city account

[liutikas]

Is there any sort of break down on what goes in "first use" and why it is so much slower?

[stigkj]

@liutikas I guess "first use" is slower with Kotlin DSL as that needs to be compiled.

[eskatos]

Here are numbers updated for Gradle 8.0

Use case	Groovy	Kotlin	Differences
First use	🟢 30s	🔴 75s	Groovy DSL is 2.5x faster
buildSrc abi change	🟢 9s	🔴 35s	Groovy DSL is 3.8x faster
buildSrc non-abi change	🔴 9s	🟢 5.5s	Kotlin DSL is 1.7x faster
configuration	🟢 1.9s	🔴 2.2s	Groovy DSL is 0.15x faster

This is not looking great. The upgrade to Kotlin 1.8 in Gradle 8.0 caused a regression in Kotlin DSL script compilation. Also those scenarios are not the best at representing reality as they don't even use the plugins {} block for which Gradle 8.0 can provide a 20% performance boost. This would not make the overall build times faster than with the Groovy DSL though.

The gap is not just the Kotlin compiler performance change. Groovy got faster. There are also some changes due to other areas of Gradle at play. But there definitely was a performance regression with 1.8 in our isolated tests. We are discussing this with the compiler team and will try to mitigate.

[Jacks0N23]

@eskatos could you please check gradle 8.1 performance