Cannot run Kotlin DSL tests from IDEA

[wolfs]

When running e.g. NestedInputKotlinImplementationTrackingIntegrationTest from IDEA, the test fails with

Caused by: java.lang.NullPointerException
	at org.gradle.api.internal.classpath.DefaultModuleRegistry.loadExternalModule(DefaultModuleRegistry.java:100)
	at org.gradle.api.internal.classpath.DefaultModuleRegistry.getExternalModule(DefaultModuleRegistry.java:91)
	at org.gradle.kotlin.dsl.provider.KotlinScriptClassPathProvider$gradleApiMetadataJar$2.invoke(KotlinScriptClassPathProvider.kt:199)
	at org.gradle.kotlin.dsl.provider.KotlinScriptClassPathProvider$gradleApiMetadataJar$2.invoke(KotlinScriptClassPathProvider.kt:93)
	at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:74)
	at org.gradle.kotlin.dsl.provider.KotlinScriptClassPathProvider.access$getGradleApiMetadataJar$p(KotlinScriptClassPathProvider.kt:93)
	at org.gradle.kotlin.dsl.provider.KotlinScriptClassPathProvider$gradleKotlinDslExtensions$1.invoke(KotlinScriptClassPathProvider.kt:158)
	at org.gradle.kotlin.dsl.provider.KotlinScriptClassPathProvider$gradleKotlinDslExtensions$1.invoke(KotlinScriptClassPathProvider.kt:93)
	at org.gradle.kotlin.dsl.provider.KotlinScriptClassPathProvider$produceFrom$1$$special$$inlined$use$lambda$1.invoke(KotlinScriptClassPathProvider.kt:165)
	at org.gradle.kotlin.dsl.provider.KotlinScriptClassPathProvider$produceFrom$1$$special$$inlined$use$lambda$1.invoke(KotlinScriptClassPathProvider.kt:93)
	at org.gradle.kotlin.dsl.provider.KotlinScriptClassPathProvider.generateAtomically(KotlinScriptClassPathProvider.kt:172)
	at org.gradle.kotlin.dsl.provider.KotlinScriptClassPathProvider.access$generateAtomically(KotlinScriptClassPathProvider.kt:93)
	at org.gradle.kotlin.dsl.provider.KotlinScriptClassPathProvider$produceFrom$1.invoke(KotlinScriptClassPathProvider.kt:165)
	at org.gradle.kotlin.dsl.provider.KotlinScriptClassPathProvider$produceFrom$1.invoke(KotlinScriptClassPathProvider.kt:93)
	at org.gradle.cache.internal.DefaultGeneratedGradleJarCache$1.run(DefaultGeneratedGradleJarCache.java:61)
	at org.gradle.internal.Factories$1.create(Factories.java:25)
	at org.gradle.cache.internal.DefaultCacheAccess.useCache(DefaultCacheAccess.java:223)
	at org.gradle.cache.internal.DefaultCacheAccess.useCache(DefaultCacheAccess.java:204)
	at org.gradle.cache.internal.DefaultPersistentDirectoryStore.useCache(DefaultPersistentDirectoryStore.java:185)
	at org.gradle.cache.internal.DefaultCacheFactory$ReferenceTrackingCache.useCache(DefaultCacheFactory.java:209)
	at org.gradle.cache.internal.DefaultGeneratedGradleJarCache.get(DefaultGeneratedGradleJarCache.java:58)
	at org.gradle.kotlin.dsl.provider.BuildServices$versionedJarCacheFor$1.invoke(BuildServices.kt:109)
	at org.gradle.kotlin.dsl.provider.BuildServices$versionedJarCacheFor$1.invoke(BuildServices.kt:43)
	at org.gradle.kotlin.dsl.provider.KotlinScriptClassPathProvider.produceFrom(KotlinScriptClassPathProvider.kt:163)
	at org.gradle.kotlin.dsl.provider.KotlinScriptClassPathProvider.gradleKotlinDslExtensions(KotlinScriptClassPathProvider.kt:157)
	at org.gradle.kotlin.dsl.provider.KotlinScriptClassPathProvider.access$gradleKotlinDslExtensions(KotlinScriptClassPathProvider.kt:93)
	at org.gradle.kotlin.dsl.provider.KotlinScriptClassPathProvider$gradleApiExtensions$2.invoke(KotlinScriptClassPathProvider.kt:120)
	at org.gradle.kotlin.dsl.provider.KotlinScriptClassPathProvider$gradleApiExtensions$2.invoke(KotlinScriptClassPathProvider.kt:93)
	at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:74)
	at org.gradle.kotlin.dsl.provider.KotlinScriptClassPathProvider.access$getGradleApiExtensions$p(KotlinScriptClassPathProvider.kt:93)
	at org.gradle.kotlin.dsl.provider.KotlinScriptClassPathProvider$gradleKotlinDsl$2.invoke(KotlinScriptClassPathProvider.kt:107)
	at org.gradle.kotlin.dsl.provider.KotlinScriptClassPathProvider$gradleKotlinDsl$2.invoke(KotlinScriptClassPathProvider.kt:93)
	at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:74)
	at org.gradle.kotlin.dsl.provider.KotlinScriptClassPathProvider.computeCompilationClassPath(KotlinScriptClassPathProvider.kt:145)
	at org.gradle.kotlin.dsl.provider.KotlinScriptClassPathProvider.access$computeCompilationClassPath(KotlinScriptClassPathProvider.kt:93)
	at org.gradle.kotlin.dsl.provider.KotlinScriptClassPathProvider$compilationClassPathOf$1.invoke(KotlinScriptClassPathProvider.kt:141)
	at org.gradle.kotlin.dsl.provider.KotlinScriptClassPathProvider$compilationClassPathOf$1.invoke(KotlinScriptClassPathProvider.kt:93)
	at org.gradle.kotlin.dsl.provider.KotlinScriptClassPathProvider.compilationClassPathOf(KotlinScriptClassPathProvider.kt:141)
	at org.gradle.kotlin.dsl.provider.StandardKotlinScriptEvaluator$InterpreterHost.compilationClassPathOf(KotlinScriptEvaluator.kt:265)
	at org.gradle.kotlin.dsl.execution.Interpreter$emitSpecializedProgramFor$cachedDir$1.invoke(Interpreter.kt:288)
	at org.gradle.kotlin.dsl.execution.Interpreter$emitSpecializedProgramFor$cachedDir$1.invoke(Interpreter.kt:72)
	at org.gradle.kotlin.dsl.cache.ScriptCache.initializeCacheDir(ScriptCache.kt:114)
	at org.gradle.kotlin.dsl.cache.ScriptCache.access$initializeCacheDir(ScriptCache.kt:37)
	at org.gradle.kotlin.dsl.cache.ScriptCache$cacheDirFor$1.execute(ScriptCache.kt:60)
	at org.gradle.kotlin.dsl.cache.ScriptCache$cacheDirFor$1.execute(ScriptCache.kt:37)

Looks like it cannot find the gradle-api-metadata.jar. Any ideas how to fix this?

The test works from the command line, so I guess this is an IDEA configuration problem.

[jjohannes]

I had a look at this hoping there is some easy way to fix it. Unfortunately there is not. I don't think this ever worked.

As far as I can see, Koltlin DSL always needs the gradle-api-metadata.jar. But in the embedded test environment, we do not work with the jars from the modules, but with the classes and resources folders.

So Kotlin DSL calls ModuleRegistry.getExternalModule(), which can not succeed because it always requires a Gradle installation directory to be present (which is not the case).
Instead, under these conditions, getModule() should be used and Kotlin DSL should be able to handle the non-jar variant of gradle-api-metadata.
Or there is some other solution where we treat gradle-api-metadata.jar special and make it available. Note that we then also need to build it, which could be done by calling the jar task after Sync in IDEA.

@gradle/kotlin-dsl any thoughts?

I also wonder if we should wait for the merge of kotlin-dsl with gradle/gradle before working on this.

[jjohannes]

Just tested it again (after the Kotlin DSL migration to the Gradle repository). The problem remains but may be easier to fix now. I think we need to be able to load the gradle-api-declaration.properties file from a folder-based classpath here:

gradle/subprojects/kotlin-dsl/src/main/kotlin/org/gradle/kotlin/dsl/provider/KotlinScriptClassPathProvider.kt

Line 200 in 7a75b24

moduleRegistry.getExternalModule(gradleApiMetadataModuleName).classpath.asFiles.single()

We also need to make sure that the files is actually in the classpath of IDEA. I think there is something not in-line with how the ...classpath.properties are generated. Look at this output:

@gradle/kotlin-dsl ☝️

[eskatos]

Those are stale files. Here's how it looks now

[lptr]

Looks like the error message changed, but the problem still exists:

* What went wrong:
A problem occurred configuring root project 'ebwad'.
> Could not open cache directory 3mxv4cuknlsijwduypok3q39x (/Users/lptr/Workspace/gradle/gradle/intTestHomeDir/worker-1/caches/5.5-20190428220000+0000/gradle-kotlin-dsl/3mxv4cuknlsijwduypok3q39x).
   > Cannot locate JAR for module 'gradle-api-metadata' in in classpath: [...]

[wolfs]

This works now since we delegate to Gradle. I don't think we will fix the tests without using delegation to Gradle.