Gradle - Too many open files #26115
Comments
|
Thank you for your interest in Gradle! We are trying to figure out if this is an AGP issue or a problem in the Build Tool. We will keep you posted. |
jbartok
added
👋 team-triage
|
Hello, any update on this? |
|
Thank you for providing a valid report. The issue is in the backlog of the relevant team, but the existence of a workaround makes it non-critical, so it might take a while before a fix is made. Gradle should always pass See also this gist. The workaround is to add |
ov7a
added
in:wrapper
has:reproducer
|
I see, thank you for the update. Can you provide an example on how can I add the And the only way to fix is by running So, i do appreciate any help with this. |
|
@Rodrigolmti Have you tried setting environment variables |
|
Well, for a moment this seemed to fix the problem, however after some time it came back again. I've added:
To my bash file. |
|
We managed to update the project to Gradle 8.3 and Kotlin 1.9.10, the problem happens less than what was happening on Gradle 7.4, however it is still there. I would appreciate it if you can remove the tag has workaround, since it does not have, or at least not these that you recommended. Thank you. |
|
We can't reproduce the problem with this repository https://github.com/schnapster/gradle-darwin-maxfdlimit If you still see something piling up file descriptors, there might be a leak somewhere. You can use I still consider this issue being about the wrapper scripts that should always set |
|
The issue should be easily replicated with this repo. |
|
Looking at Changing the system configuration to raise the maximum limit is out of Gradle's scope. However, the wrapper scripts try to raise the user limit for the Gradle process on all platforms except cygwin and macos. In the same fashion, we could change the wrapper scripts to try to do that too on macos by running |
|
Ok, so we did try to use As described on the issue, we already tested a bunch of different solutions that we found thought the internet. None has worked so far. We have 16 engineers facing this problem basically every day after 3 Gradle syncs or project build. We are going to take a look at the Furthermore, we are at the place where, if we decide to remove the build-logic module where we keep all our convention plugins, it will require a considerable amount of work and time. So I want to explore all the cases before, specially because it is generating some benefits for us today, due to modularization. We tried Gradle enterprise before but seems that we do not meet the minimum criteria and Gradle enterprise do not offer this type of solution or help. In case that any of you have a few minutes where we can show to you what is happening, maybe we can discover something more relevant. Thank you for the help. |
|
Thank you for the context and the link to the Android issue. https://developer.apple.com/forums//thread/735798 This looks like a bug in recent macos versions to me. I hope they'll get a fix released soon. |
|
Adding |
|
This is our current
So, it may not be working, too. |
|
Our fix has been to use a LaunchDaemon to raise the fdlimits at startup:
#!/bin/sh
# https://developer.apple.com/forums//thread/735798
launchctl limit maxfiles 256 unlimited
launchctl limit maxfiles 128000 524288
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>limit.maxfiles</string>
<key>ProgramArguments</key>
<array>
<string>/opt/local/bin/run_max_files</string>
</array>
<key>RunAtLoad</key>
<true/>
</dict>
</plist>Next, restart or run We added the following to if $darwin; then
JAVA_OPTS=-XX:-MaxFDLimit
fi And ensure that |
|
Thank you @joshfriend, this works! |
|
Unfortunately, this does not help with our problem. We tested in 2 different machines. Not sure if it is something particular to the project or some constraint to the managed OS by the company. |
|
@Rodrigolmti what doen't work when you apply Josh's recipe? Are your OS and JVM reporting the right limits? |
|
Well, the same error still happens (too many open files). I did try Josh's recipe a couple of times, and double check if everything was right. Even if I restart the computer, I'll get:
when running:
If I tried to run:
I get the following error:
So I guess it is running. I have also tried to restart the mac, no luck, a second developer also tried, no luck. Not sure if it can be some restriction of the managed OS because this was working a while ago. I was following this article http://blog.mact.me/2014/10/22/yosemite-upgrade-changes-open-file-limit, however it is not working anymore. We also tried to run the |
|
If you still get |
|
One thing to note is that |
|
I managed to get it to work by using
`
I tried to get a build scan to have more data, but not even the scan works:
I'm at lost with this problem, nothing seems to work. Wondering what can be wrong, but how to understand that if the error messages are not even close to be useful. One problem that we also see frequently that I think is related to too many open files is: ` 1: Task failed with an exception.
2: Task failed with an exception.
|
|
it is very important that you have |
|
Like I said before: This is my gradle.properties: This is my Custom VM options from AS: This is my gradlew file: (Added the exactly same code shared here) I'm using:
A selective workaround is not a workaround. Not sure where else to add this flag. |
|
I manage to get the leak jar working. I see a lot of references to /.git folder Could this be related to this plugin? https://github.com/gladed/gradle-android-git-version. We use it to generate version code / version name based on some git params. |
|
@Rodrigolmti, could very well be that this plugin is not properly closing used resources |
|
If you use Okio/Wire, try using the latest snapshots of Wire which fixes a file descriptor leak (square/okio#1359, https://square.github.io/okio/changelog/#version-360) There's also fd leak in Detekt: detekt/detekt#6519 Those fixes might help, but for our very large gradle project we still have issues |
|
If you are seeing the file limit hit while syncing a project on the IDE, you'll also need to add the
|
|
Took a look at @Rodrigolmti project posted on this issue, and made the following findings using the file leak detector. Setup:
Results:
After looking at the diffs of the syncs, we can see we have a large number of files that were opened several times with the following stack traces: This shows that you are dealing with the However, I also saw this other leak occurring in the Kotlin script evaluator code: But investigating this further, if we upgrade to using Gradle 8.3, we get the following results:
As we can see, the increase in file descriptors isn't as much as 7.4.1. Also, comparing these two files, we can see the detekt bug is still there too. However, the Kotlin script evaluator leak is not seen here, instead, there is a minor leak here: This seems to affect the files for As for @Rodrigolmti, I would recommend migrating to Gradle 8.3.0, and fixing your |
|
The TL;DR for the above: Detekt was one reason for the original file leak, but it seems there's some weird file leak happening in Gradle 7.4.1 with the Kotlin script evaluator. That Kotlin evaluator bug goes away(?) with Gradle 8.3, but then we see other smaller leaks with script evaluation too. Finally, there is one small leak in the project code buildsrc. I believe this issue isn't shown right away, but after several builds/syncs. I only tested this for the configuration phase, but it's entirely possible to see file leaks occur in the build phase as well, which will also trigger the |
|
IIRC, the Groovy DSL keeps these files open across builds if they are reused, and should release them if two consecutive builds are not reusing them. If these are coming from temporary init scripts injected by the IDE with changing path/names on each invocation it might trigger a slow fd leak. |
|
I don't have any init scripts in my Gradle User Home, but I did use IntelliJ to trigger the Gradle sync, which is probably where these scripts are coming from then. I made no changes to the project, only hit the "sync" button back to back. Doing a count of the extra jars opened, it's exactly 5 extra file descriptors for each stack trace, which coincides with my testing. Each new file sits in a different directory than the others.
Edit: |
|
Hmn that's fascinating, thanks for the investigation. Right now we are using Gradle 8.3 already, and indeed, we notice some improvements, We'll try to fix the leak for the MPP and wait for the detekt new version and post the update here. Really appreciate all the help. |
Current Behavior
Recently I'm getting a lot of errors on my project, and we do not have any clue what may be causing this. Each time we have a different error, but we can summarize it as (Too many open files).
Closing the Android Studio and killing all java instances solves the problem, but after 2/3 Gradle syncs the problem comes back.
`Caused by: org.gradle.api.resources.MissingResourceException: Could not read build file '/Users/rodrigo.martins/Code/skipthedishes_android_v3/features/order/review/api/build.gradle.kts' as it does not exist.
Caused by: java.lang.NullPointerException: Cannot read the array length because "sessionFiles" is null
`
gradle_error.txt
gradle_sync.txt
I have attached some supporting files that contain more details about the error, stack trace and dumps.
Expected Behavior
I can sync the project without getting error or having to increase the number of open files that my system suppports.
Context (optional)
Too many open files
The issue arises consistently every time we attempt to run a Gradle sync. Neither invalidating the cache nor performing a clean build has resolved the issue. This problem is not confined to a specific type of machine; it occurs on M1, M2, and Intel-based machines alike.
The error message lacks detailed information, but we have attached logs and stack traces to this issue. This issue has persisted across various Android Studio versions over a period of time.
Tested solution:
Add -XX:-MaxFDLimit to studio.vmoptions.
Reference: gradle/gradle#17274
Run the attached script update_max_limit.sh which is supposed to increase the limit of open files. Initially, it helped, but the PC started to slow down, and now the script is no longer effective.
The script is based on this blog post.
Sometimes it helps to run pkill -f "java|studio", but now this command is also not working anymore.
We also have an open issue on the Gradle forum, but there have been no answers so far.
Forum link: discuss.gradle.org/t/caused-by-java-io-ioexception-error-24-too-many-open-files/45884
This issue seems to have started after we added Compose to the project and Gradle conventional plugins. We are not sure which one is the root cause or if this is the cause at all.
I have attached a sample of our project with reduced files, along with logging files of the problem and the IDA as requested.
stracktrace.txt
threadDump-20230814-110815.txt
project.zip
idea.log
local_history.txt
open_files.txt
Steps to Reproduce
This is a project that is supposed to simulate the issue: github.com/napstr/gradle-darwin-maxfdlimit. We also tried to make the changes proposed in this repository, but the issue still occurs.
Gradle version
7.4.2 / 7.5
Build scan URL (optional)
https://scans.gradle.com/s/uieftr67or3qw
Your Environment (optional)
Studio Build:
Android Studio Giraffe | 2022.3.1 Build #AI-223.8836.35.2231.10406996, built on June 29, 2023 Runtime version: 17.0.6+0-17.0.6b829.9-10027231 aarch64 VM: OpenJDK 64-Bit Server VM by JetBrains s.r.o. macOS 13.5 GC: G1 Young Generation, G1 Old Generation Memory: 4096M Cores: 8 Metal Rendering is ON
Version of Gradle Plugin: 7.4.2 Version of Gradle: gradle-7.4.2-bin Version of Java: openjdk 17.0.7 2023-04-18 OS: Mac OS Ventura 13.5
The text was updated successfully, but these errors were encountered: