Unable to process Kotlin code after switching to Gradle 8.11 with `org.gradle.configuration-cache.parallel = true`

[sschuberth]

It seems that after upgrading our Kotlin/JVM project to Gradle 8.11 and enabling the new org.gradle.configuration-cache.parallel = true, we getting

/opt/hostedtoolcache/CodeQL/2.19.3/x64/codeql/codeql database finalize --finalize-dataset --threads=4 --ram=14576 /home/runner/work/_temp/codeql_databases/java
  CodeQL detected code written in Java/Kotlin but could not process any of it. For more information, review our troubleshooting guide at https://gh.io/troubleshooting-code-scanning/no-source-code-seen-during-build .
  Error: Encountered a fatal error while running "/opt/hostedtoolcache/CodeQL/2.19.3/x64/codeql/codeql database finalize --finalize-dataset --threads=4 --ram=14576 /home/runner/work/_temp/codeql_databases/java". Exit code was 32 and last log line was: CodeQL detected code written in Java/Kotlin but could not process any of it. For more information, review our troubleshooting guide at https://gh.io/troubleshooting-code-scanning/no-source-code-seen-during-build . See the logs for more details.

Can it be that parallelizing configuration cache access has an impact on CodeQL analysis?

[smowton]

It looks like the problem is that CodeQL extraction works by observing kotlinc invocations, but in fact your compileKotlin jobs all pull from a cache instead. Try disabling the use of a build cache, using --no-build-cache, and just in case Gradle has been run before in that VM use --no-daemon to disable the Gradle daemon, which similarly defeats attempts to trace compiler invocations.

[sschuberth]

use --no-daemon to disable the Gradle daemon

We're already doing that, but I'll give the --no-build-cache a try on top.

CodeQL extraction works by observing kotlinc invocations

As a side note, isn't that approach a bit fragile? Should maybe the Gradle tooling API be used instead?

[smowton]

I should clarify, we can also see in-process invocations of the Kotlin compiler, not just literal usage of the kotlinc binary, if that's what you mean?

[sschuberth]

not just literal usage of the kotlinc binary

That makes things better, I guess, but I'm still wondering whether CodeQL shouldn't rather hook into Gradle, so that it somehow could also get compiled code from caches instead of always requiring full builds (that's my understanding now how it works).

[smowton]

Our current Kotlin extractor works as a Kotlin compiler plugin, so just getting artifacts from a cache wouldn't help, we'd have to be able to determine the compiler arguments to run our extractor in the right context. Not to say there aren't opportunities to extract Kotlin in a smarter way than we do at present, but specifically getting compiler output isn't useful.

[smowton]

I note the linked commit seems to feature a successful CodeQL analysis, so I'll close this.

[sschuberth]

I note the linked commit seems to feature a successful CodeQL analysis, so I'll close this.

Yes, sorry for forgetting to report back.