Proposal: Remove fixed Kotlin runtime version limitation in Gradle

[ZacSweers]

Today Gradle bundles an embedded version of Kotlin. My understanding is that this is due to usage of private compiler APIs in order to support kotlin-dsl. This prevents all plugins on the classpath from using different (let alone newer) Kotlin versions though, even if both they and the consuming build are not using kotlin-dsl.

This is something that I feel is fundamentally holding back the ecosystem, because it negatively affects plugin developers in the following ways:

• They must set arbitrary language settings on their build, like lowering the languageLevel setting to 1.3 today, otherwise risk accidentally writing incompatible code.
• For much of Kotlin 1.4.x's release cycle, they could not use Kotlin 1.4 period. Kotlin 1.4 was not added until Gradle 6.8, and even then it didn't allow kotlin 1.4 language targeting.
• Language level alone only covers their own sources, but then they must also meticulously test any Kotlin-using external libraries that may have updated to 1.4 and use 1.4 APIs.
  • A good example of where this comes up is Okio, OkHttp, or Coroutines. Even something as simple as a member reference becomes a breaking change when compiled in 1.4 and run in a 1.3 environment.

This would be less of an issue if Gradle updated its embedded Kotlin version regularly, but the judgement today is that it cannot target a new minor version of Kotlin (now on a ~6mo cadence) until a new major version of Gradle (~once per year). While this may have lined up somewhat with Kotlin's previous ~annual release strategy, this is unwinding now with Kotlin 1.5's upcoming release. As of now, Gradle is on track to miss the entirety of Kotlin 1.4 due to its reluctance to update or allow users to use a new version. This will only continue now that Kotlin's release cycle is accelerated.

I understand that part of this reluctance to allow newer versions is because they may break. However, I think this is easily remedied by simply making it clear what the currently supported version is. Gradle adopts this approach with Java today, where it often supports running on a newer version but may not officially support it until a later release (major or otherwise). Gradle added Java 15 support in the 6.7 minor release, for instance.

The private API usage does offer a good reason for using a fixed version, but again there is precedent for this too as Gradle uses private javac APIs too (namely from the private-ish tools jar) to support cases like incremental compilation.

I would like to propose adopting one of the following strategies (in order of preference, though they're not all mutually exclusive either):

1. Unbundle Kotlin, remove private API usage or move it behind a flag. "If you want to use these APIs in kotlin-dsl, this must be enabled and comes with these restrictions"
2. Unbundle Kotlin, use a simple minimum/supported version and denote new un-tested versions as unsupported "your mileage may vary"
3. Allow/commit to more frequent updates. I.e. allow minor Kotlin version updates in minor Gradle version updates
4. Unbundle Kotlin, shade the embedded kotlin compiler or its privately used APIs to avoid conflict

[CristianGM]

I would say my preference order is:
3
2
1
4

[bamboo]

This prevents all plugins on the classpath from using different (let alone newer) Kotlin versions though, even if both they and the consuming build are not using kotlin-dsl.

The statement above is not true.

Currently Gradle only exposes (or should only expose) the Kotlin runtime (basically kotlin-stdlib* and kotlin-reflect*, no compiler libs) to the plugin/script classpath. So while plugins that, for one reason or another, are incompatible with the particular version of the Kotlin stdlibs shipped with Gradle are not supported, this is not the same as preventing all plugins on the classpath from using different Kotlin versions.

That being said, I do appreciate how inconvenient and far from ideal the status quo is and agree we should find a better strategy.

[ZacSweers]

Fair! In practice though - restricting both the target language version and stdlib (kotlin's "runtime") is effectively the same as limiting the "language version" for Kotlin in my opinion, just a semantic difference. Adding kotlin-reflect and other core libraries adds to that as well as people expect them to match the core "Kotlin" version (i.e. 1.4.30)

[ZacSweers]

Another example of where this causes friction: in most repos, it ends up requiring maintaining brittle infrastructure to support two kotlin versions. If a repo chooses to move ahead, it faces noisy, obscure warnings from kotlinc like this:

w: Runtime JAR files in the classpath should have the same version. These files were found in the classpath:
    /home/runner/.gradle/wrapper/dists/gradle-7.0-all/9m115ut5nwvtxli7nys8pggfr/gradle-7.0/lib/kotlin-stdlib-1.4.31.jar (version 1.4)
    /home/runner/.gradle/wrapper/dists/gradle-7.0-all/9m115ut5nwvtxli7nys8pggfr/gradle-7.0/lib/kotlin-stdlib-common-1.4.31.jar (version 1.4)
    /home/runner/.gradle/wrapper/dists/gradle-7.0-all/9m115ut5nwvtxli7nys8pggfr/gradle-7.0/lib/kotlin-stdlib-jdk7-1.4.31.jar (version 1.4)
    /home/runner/.gradle/wrapper/dists/gradle-7.0-all/9m115ut5nwvtxli7nys8pggfr/gradle-7.0/lib/kotlin-stdlib-jdk8-1.4.31.jar (version 1.4)
    /home/runner/.gradle/wrapper/dists/gradle-7.0-all/9m115ut5nwvtxli7nys8pggfr/gradle-7.0/lib/kotlin-reflect-1.4.31.jar (version 1.4)
    /home/runner/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib-jdk8/1.4.31/e613be5465ef1e6fd0468707690b7ebf625ea2fe/kotlin-stdlib-jdk8-1.4.31.jar (version 1.4)
    /home/runner/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-reflect/1.4.31/63db9d66c3d20f7b8f66196e7ba86969daae8b8a/kotlin-reflect-1.4.31.jar (version 1.4)
    /home/runner/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib-jdk7/1.4.31/84ce8e85f6e84270b2b501d44e9f0ba6ff64fa71/kotlin-stdlib-jdk7-1.4.31.jar (version 1.4)
    /home/runner/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib/1.5.0-M2/575e1b43819635280e68bd3d3e8441603e97bbc3/kotlin-stdlib-1.5.0-M2.jar (version 1.5)
    /home/runner/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib-common/1.5.0-M2/5e6ad08bde4c95d93ca6b366ca5e3a0bb49c70bc/kotlin-stdlib-common-1.5.0-M2.jar (version 1.5)
w: Consider providing an explicit dependency on kotlin-reflect 1.5 to prevent strange errors
w: Some runtime JAR files in the classpath have an incompatible version. Consider removing them from the classpath
e: warnings found and -Werror specified

> Task :agp-handlers:agp-handler-42:kaptGenerateStubsKotlin

FAILURE: Build failed with an exception.

Even worse - it's per-project and it prevents use of -Werror.

[Tapchicoma]

@ZacSweers this is the problem on KGP side: https://youtrack.jetbrains.com/issue/KT-41142

[ZacSweers]

The Forward Compatibility section of the kotlin-dsl docs mention this:

The biggest issue is the compatibility between the external kotlin-gradle-plugin version and the kotlin-stdlib version shipped with Gradle. More generally, between any plugin that transitively depends on kotlin-stdlib and its version shipped with Gradle. As long as the combination is compatible everything should work. This will become less of an issue as the language matures.

Kotlin as a language is mature today and guarantees backward compatibility. Is Gradle claiming to add an extra layer of compatibility or is this doc out of date?