Large amount of open files after configuration phase of large build (>5000 projects)

[pablobaxter]

Current Behavior

In a monorepo with > 5000 projects, we are seeing a large number of files left open after the configuration phase (typically after an IDE sync). Mainly we see cp_proj.jar and proj.jar files left open. This has become problematic for us, as many of the devlopers who work on this project are on MacOS, and seem to be hitting the same issue reported here: #26115.

I used the file leak detector to capture all the open file descriptors after an IDE sync, and found ~9.4k proj.jar and ~9.5k cp_proj.jar files left open. Below are the stack traces that is common for the majority of each of these files.

Top 2 stack traces for cp_proj.jar

#20643 /Users/pablobaxter/.gradle/caches/jars-9/1fb912b78ac4bad43be00f9f4156c4df/cp_proj.jar by thread:Daemon worker on Tue Oct 03 08:27:01 PDT 2023
	at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:214)
	at java.base/java.util.zip.ZipFile$Source.<init>(ZipFile.java:1312)
	at java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:1277)
	at java.base/java.util.zip.ZipFile$CleanableResource.<init>(ZipFile.java:709)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:243)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:172)
	at java.base/java.util.jar.JarFile.<init>(JarFile.java:347)
	at java.base/jdk.internal.loader.URLClassPath$JarLoader.getJarFile(URLClassPath.java:841)
	at java.base/jdk.internal.loader.URLClassPath$JarLoader$1.run(URLClassPath.java:785)
	at java.base/jdk.internal.loader.URLClassPath$JarLoader$1.run(URLClassPath.java:778)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
	at java.base/jdk.internal.loader.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:777)
	at java.base/jdk.internal.loader.URLClassPath$JarLoader.<init>(URLClassPath.java:747)
	at java.base/jdk.internal.loader.URLClassPath$3.run(URLClassPath.java:502)
	at java.base/jdk.internal.loader.URLClassPath$3.run(URLClassPath.java:485)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
	at java.base/jdk.internal.loader.URLClassPath.getLoader(URLClassPath.java:484)
	at java.base/jdk.internal.loader.URLClassPath.getLoader(URLClassPath.java:452)
	at java.base/jdk.internal.loader.URLClassPath.getResource(URLClassPath.java:321)
	at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:424)
	at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:421)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:420)
	at org.gradle.groovy.scripts.internal.DefaultScriptCompilationHandler$ScriptClassLoader.loadClass(DefaultScriptCompilationHandler.java:384)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
	at org.gradle.groovy.scripts.internal.DefaultScriptCompilationHandler$ClassesDirCompiledScript.loadClass(DefaultScriptCompilationHandler.java:335)
	at org.gradle.groovy.scripts.internal.DefaultScriptRunnerFactory$ScriptRunnerImpl.getScript(DefaultScriptRunnerFactory.java:55)
	at org.gradle.groovy.scripts.internal.DefaultScriptRunnerFactory$ScriptRunnerImpl.run(DefaultScriptRunnerFactory.java:86)
	at org.gradle.configuration.DefaultScriptPluginFactory$ScriptPluginImpl.apply(DefaultScriptPluginFactory.java:111)
	at org.gradle.configuration.BuildOperationScriptPlugin$1.run(BuildOperationScriptPlugin.java:65)

#11948 /Users/pablobaxter/.gradle/caches/jars-9/9d6ca55066e0de258e10da287d04d3a2/cp_proj.jar by thread:Daemon worker on Tue Oct 03 08:26:09 PDT 2023
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:173)
	at java.base/java.util.jar.JarFile.<init>(JarFile.java:347)
	at java.base/jdk.internal.loader.URLClassPath$JarLoader.getJarFile(URLClassPath.java:841)
	at java.base/jdk.internal.loader.URLClassPath$JarLoader$1.run(URLClassPath.java:785)
	at java.base/jdk.internal.loader.URLClassPath$JarLoader$1.run(URLClassPath.java:778)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
	at java.base/jdk.internal.loader.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:777)
	at java.base/jdk.internal.loader.URLClassPath$JarLoader.<init>(URLClassPath.java:747)
	at java.base/jdk.internal.loader.URLClassPath$3.run(URLClassPath.java:502)
	at java.base/jdk.internal.loader.URLClassPath$3.run(URLClassPath.java:485)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
	at java.base/jdk.internal.loader.URLClassPath.getLoader(URLClassPath.java:484)
	at java.base/jdk.internal.loader.URLClassPath.getLoader(URLClassPath.java:452)
	at java.base/jdk.internal.loader.URLClassPath.getResource(URLClassPath.java:321)
	at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:424)
	at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:421)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:420)
	at org.gradle.groovy.scripts.internal.DefaultScriptCompilationHandler$ScriptClassLoader.loadClass(DefaultScriptCompilationHandler.java:384)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
	at org.gradle.groovy.scripts.internal.DefaultScriptCompilationHandler$ClassesDirCompiledScript.loadClass(DefaultScriptCompilationHandler.java:335)
	at org.gradle.groovy.scripts.internal.DefaultScriptRunnerFactory$ScriptRunnerImpl.getScript(DefaultScriptRunnerFactory.java:55)
	at org.gradle.groovy.scripts.internal.DefaultScriptRunnerFactory$ScriptRunnerImpl.run(DefaultScriptRunnerFactory.java:86)
	at org.gradle.configuration.DefaultScriptPluginFactory$ScriptPluginImpl.apply(DefaultScriptPluginFactory.java:111)
	at org.gradle.configuration.BuildOperationScriptPlugin$1.run(BuildOperationScriptPlugin.java:65)

Top 2 stack traces for proj.jar

#17901 /Users/pablobaxter/.gradle/caches/jars-9/f03c7d2e54128011619f95c0c83aec4a/proj.jar by thread:Daemon worker on Tue Oct 03 08:26:44 PDT 2023
	at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:214)
	at java.base/java.util.zip.ZipFile$Source.<init>(ZipFile.java:1312)
	at java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:1277)
	at java.base/java.util.zip.ZipFile$CleanableResource.<init>(ZipFile.java:709)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:243)
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:172)
	at java.base/java.util.jar.JarFile.<init>(JarFile.java:347)
	at java.base/jdk.internal.loader.URLClassPath$JarLoader.getJarFile(URLClassPath.java:841)
	at java.base/jdk.internal.loader.URLClassPath$JarLoader$1.run(URLClassPath.java:785)
	at java.base/jdk.internal.loader.URLClassPath$JarLoader$1.run(URLClassPath.java:778)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
	at java.base/jdk.internal.loader.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:777)
	at java.base/jdk.internal.loader.URLClassPath$JarLoader.<init>(URLClassPath.java:747)
	at java.base/jdk.internal.loader.URLClassPath$3.run(URLClassPath.java:502)
	at java.base/jdk.internal.loader.URLClassPath$3.run(URLClassPath.java:485)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
	at java.base/jdk.internal.loader.URLClassPath.getLoader(URLClassPath.java:484)
	at java.base/jdk.internal.loader.URLClassPath.getLoader(URLClassPath.java:452)
	at java.base/jdk.internal.loader.URLClassPath.getResource(URLClassPath.java:321)
	at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:424)
	at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:421)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:420)
	at org.gradle.groovy.scripts.internal.DefaultScriptCompilationHandler$ScriptClassLoader.loadClass(DefaultScriptCompilationHandler.java:384)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
	at org.gradle.groovy.scripts.internal.DefaultScriptCompilationHandler$ClassesDirCompiledScript.loadClass(DefaultScriptCompilationHandler.java:335)
	at org.gradle.groovy.scripts.internal.DefaultScriptRunnerFactory$ScriptRunnerImpl.getScript(DefaultScriptRunnerFactory.java:55)
	at org.gradle.groovy.scripts.internal.DefaultScriptRunnerFactory$ScriptRunnerImpl.run(DefaultScriptRunnerFactory.java:86)
	at org.gradle.configuration.DefaultScriptPluginFactory$ScriptPluginImpl.lambda$apply$0(DefaultScriptPluginFactory.java:135)
	at org.gradle.configuration.ProjectScriptTarget.addConfiguration(ProjectScriptTarget.java:79)
	at org.gradle.configuration.DefaultScriptPluginFactory$ScriptPluginImpl.apply(DefaultScriptPluginFactory.java:138)
	at org.gradle.configuration.BuildOperationScriptPlugin$1.run(BuildOperationScriptPlugin.java:65)

#4157 /Users/pablobaxter/.gradle/caches/jars-9/61df8b13771b6445fc7e4906368507cc/proj.jar by thread:Daemon worker on Tue Oct 03 08:24:57 PDT 2023
	at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:173)
	at java.base/java.util.jar.JarFile.<init>(JarFile.java:347)
	at java.base/jdk.internal.loader.URLClassPath$JarLoader.getJarFile(URLClassPath.java:841)
	at java.base/jdk.internal.loader.URLClassPath$JarLoader$1.run(URLClassPath.java:785)
	at java.base/jdk.internal.loader.URLClassPath$JarLoader$1.run(URLClassPath.java:778)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
	at java.base/jdk.internal.loader.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:777)
	at java.base/jdk.internal.loader.URLClassPath$JarLoader.<init>(URLClassPath.java:747)
	at java.base/jdk.internal.loader.URLClassPath$3.run(URLClassPath.java:502)
	at java.base/jdk.internal.loader.URLClassPath$3.run(URLClassPath.java:485)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
	at java.base/jdk.internal.loader.URLClassPath.getLoader(URLClassPath.java:484)
	at java.base/jdk.internal.loader.URLClassPath.getLoader(URLClassPath.java:452)
	at java.base/jdk.internal.loader.URLClassPath.getResource(URLClassPath.java:321)
	at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:424)
	at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:421)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
	at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:420)
	at org.gradle.groovy.scripts.internal.DefaultScriptCompilationHandler$ScriptClassLoader.loadClass(DefaultScriptCompilationHandler.java:384)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
	at org.gradle.groovy.scripts.internal.DefaultScriptCompilationHandler$ClassesDirCompiledScript.loadClass(DefaultScriptCompilationHandler.java:335)
	at org.gradle.groovy.scripts.internal.DefaultScriptRunnerFactory$ScriptRunnerImpl.getScript(DefaultScriptRunnerFactory.java:55)
	at org.gradle.groovy.scripts.internal.DefaultScriptRunnerFactory$ScriptRunnerImpl.run(DefaultScriptRunnerFactory.java:86)
	at org.gradle.configuration.DefaultScriptPluginFactory$ScriptPluginImpl.lambda$apply$0(DefaultScriptPluginFactory.java:135)
	at org.gradle.configuration.ProjectScriptTarget.addConfiguration(ProjectScriptTarget.java:79)
	at org.gradle.configuration.DefaultScriptPluginFactory$ScriptPluginImpl.apply(DefaultScriptPluginFactory.java:138)
	at org.gradle.configuration.BuildOperationScriptPlugin$1.run(BuildOperationScriptPlugin.java:65)

Expected Behavior

File descriptors should be eagerly closed as soon as they are no longer needed. I'm unsure why all these files are required to stay open. If it's for performance, maybe having a flag or method to close them based on a trigger would be something we can work with, as this is causing issues with developers unable to build and/or sync their projects on MacOS.

Context (optional)

No response

Steps to Reproduce

Attempt to run a task on a very large (>5k project) build.

Gradle version

8.3

Build scan URL (optional)

No response

Your Environment (optional)

OS: MacOS 14.0
RAM: 64GB
CPU: Apple M1 Max

[eskatos]

Thank you for providing a valid report.

The issue is in the backlog of the relevant team and is prioritised by them.

---

Thank you very much for the investigation and writeup!
We should be able to reproduce on any build with lots of Groovy DSL scripts, for example using https://github.com/gradle/perf-android-large-2

cp_proj.jar and proj.jar are compiled Groovy DSL .gradle scripts.
We keep the classloaders around between builds for performance reasons, and they hold file descriptors for jars.

We fixed a similar problem in the Kotlin DSL in the past by using classes directories instead of jars.
We should apply the same fix to the Groovy DSL.

[eskatos]

After a bit more investigation, the Groovy DSL itself uses classes directories but the bytecode instrumentation for CC interception and seamless upgrades is putting everything in JAR files. The Kotlin DSL is also impacted by this but uses fewer file descriptors per script thus went unnoticed so far. We're looking into fixing this across the board.