Repository navigation
[JENKINS-23248] CLI calls are causing file descriptor leaks. #23149
Description
I have a job that dynamically creates jobs using the CLI. I have noticed that since installing the job, which verifies the existing of jobs by calling 'get-job', Jenkins is leaking file descriptors. I am currently making around 40 calls per build, which runs on every CVS commit. I have a job setup to monitor the number of FD's in /proc/$jenkins_pid/fd. Calling garbage collection in the JVM doesn't release the FD's and thus the only cure is to restart Jenkins before the number of files reaches the Open file ulimit. I have set my ulimit to 65356 so I don't have to reboot so frequently. I restarted Jenkins at 7:49 this morning and the file descriptor count is currently at 6147 files; it's now 12:10 in the afternoon, so it's been steadily leaking FD's at approximately 1500 per hour.
Originally reported by iwonbigbro, imported from: CLI calls are causing file descriptor leaks.
- assignee:
kohsuke
- status: Resolved
- priority: Major
- component(s): core
- label(s): 1.565.3-fixed, remoting
- resolution: Fixed
- resolved: 2014-07-29T01:00:39+00:00
- votes: 0
- watchers: 14
- imported: 2025-11-24
Raw content of original issue
I have a job that dynamically creates jobs using the CLI. I have noticed that since installing the job, which verifies the existing of jobs by calling 'get-job', Jenkins is leaking file descriptors. I am currently making around 40 calls per build, which runs on every CVS commit. I have a job setup to monitor the number of FD's in /proc/$jenkins_pid/fd. Calling garbage collection in the JVM doesn't release the FD's and thus the only cure is to restart Jenkins before the number of files reaches the Open file ulimit. I have set my ulimit to 65356 so I don't have to reboot so frequently. I restarted Jenkins at 7:49 this morning and the file descriptor count is currently at 6147 files; it's now 12:10 in the afternoon, so it's been steadily leaking FD's at approximately 1500 per hour.
Activity
jenkins-infra-bot commentedon May 30, 2014
Raw content of original comment:
When you look at ls -la /proc/$PID/fd output, can you tell which files are left open? Can you take a diff between two points of time and give us the delta?
jenkins-infra-bot commentedon Jun 1, 2014
iwonbigbro:
Raw content of original comment:
— /tmp/jenk_fd.1 2014-06-01 21:03:34.006155887 +0100
+++ /tmp/jenk_fd.2 2014-06-01 21:09:27.053382015 +0100
@@ -52342 +52341,0 @@
-l-wx------ 1 kcc users 64 Jun 1 20:45 57103 -> socket:[282820885]
@@ -55657 +55655,0 @@
lr-x----- 1 kcc users 64 Jun 1 20:59 60087 -> /user1/jenkins/jobs/scm-poll-jenkins-branch-monitor/builds/2014-06-01_21-01-01/log@@ -55662 +55660 @@
lr-x----- 1 kcc users 64 Jun 1 21:01 60091 -> pipe:[282533977]+lr-x------ 1 kcc users 64 Jun 1 21:01 60091 -> socket:[283188722]
@@ -55747 +55745 @@
lrwx----- 1 kcc users 64 Jun 1 21:03 60169 -> socket:[282813330]+lrwx------ 1 kcc users 64 Jun 1 21:03 60169 -> socket:[283188811]
@@ -55807 +55804,0 @@
lrwx----- 1 kcc users 64 Jun 1 21:03 60222 -> socket:[282819646]@@ -55824,0 +55822 @@
+lrwx------ 1 kcc users 64 Jun 1 21:04 60239 -> socket:[282821225]
@@ -55825,0 +55824,5 @@
+lrwx------ 1 kcc users 64 Jun 1 21:04 60240 -> socket:[282821552]
+lrwx------ 1 kcc users 64 Jun 1 21:05 60241 -> socket:[282821858]
+l-wx------ 1 kcc users 64 Jun 1 21:05 60242 -> socket:[282947156]
+lrwx------ 1 kcc users 64 Jun 1 21:07 60243 -> socket:[282947065]
+lr-x------ 1 kcc users 64 Jun 1 21:05 60244 -> socket:[283309559]
This is just a small sample within several minutes of one another. I can send a larger delta if needs be. The leak is now at >60000 files, which you can see from the line numbers in the diff.
jenkins-infra-bot commentedon Jun 1, 2014
iwonbigbro:
Raw content of original comment:
Running garbage collection in the JVM doesn't clear them down either. I think I mentioned this already.
jenkins-infra-bot commentedon Jun 1, 2014
Raw content of original comment:
Jenkins version?
jenkins-infra-bot commentedon Jun 2, 2014
iwonbigbro:
Raw content of original comment:
1.564
jenkins-infra-bot commentedon Jun 2, 2014
iwonbigbro:
Raw content of original comment:
Just upgraded to 1.565 and the issue still persists.
jenkins-infra-bot commentedon Jun 2, 2014
iwonbigbro:
Raw content of original comment:
Here is a delta with approximately an hour between each snapshot.
jenkins-infra-bot commentedon Jun 10, 2014
iwonbigbro:
Raw content of original comment:
I know what the cause was. I found this article: https://wiki.jenkins-ci.org/display/JENKINS/Spawning+processes+from+build
I'd say the article needs updating. It assumes that the bug/feature is only prevalent for sub processes that hold pipes open that become detached from the main process. This is not entirely true, since I was able to reproduce the same effect by running the CLI utility as a foreground process from a job spawned on the Jenkins server. The output and input from the CLI was attached directly to the build pipeline. There was no detaching or backgrounding going on, simply execute and return the exit code. However, what I think may be happening (I am a Java noob, so this is complete speculation) is that the same bug/feature found in Java that causes the problem for build processes, was emitted by the Java process spawned from running the CLI utility. I suspect this created some kind of circular file descriptor reference inside the JVM, preventing the EOF from being transmitted by the CLI utility when it exited.
I have cured the problem completely, by modifying my wrapper script to explicitly close all descriptors apart from stdin, stdout and stderr on invocation, before running the Java util. Also, after running the util, stdin, stdout and stderr are then explicitly closed before the shell script exits. Since making this change, the file descriptor leak has been stable at 823 total file descriptors open.
My guess is that the CLI utility should be doing something similar.
jenkins-infra-bot commentedon Jul 1, 2014
damong:
Raw content of original comment:
I just logged jenkins-23572 about a week ago and it looks like this exact same issue. Happy to mark my a duplicate of this or run tests to determine the root cause. We are calling the CLI from within our groovy test code inside of our job while it's executing on the slaves. I don't see how I could work around this by closing file descriptors in my wrapper code that's executing on the slave since it's on the master where there are too many open file descriptors. Maybe your job was executing directly on the master?
jenkins-infra-bot commentedon Jul 1, 2014
Raw content of original comment:
damong Looks like a duplicate. If you could do a "binary search" for the responsible Jenkins release (1.546 good, 1.564 bad is too many possible versions) that would help.
jenkins-infra-bot commentedon Jul 1, 2014
damong:
Raw content of original comment:
Hi Daniel,
It may take me a few days to get back to you but yes i could run this test against the intermediary versions. I've cloned this vm off into an isolated environment for testing. Isn't there something else I could do at the same time to narrow it down? lsof just shows me that the file descriptors are leaking from java. Any sort of jenkins tracing I might want to enable at the same time that might narrow down where it's coming from?
Actually it looks like we'll be able to try this on Thursday.
jenkins-infra-bot commentedon Jul 1, 2014
Raw content of original comment:
damong: If you know how to search a heap dump for these, go for it. Otherwise, it's probably easiest narrowing it down to the first broken release.
If you then have Git and Maven (3.0.5+) available, check out https://github.com/jenkinsci/jenkins and try using git bisect to find the responsible commit. mvn -DskipTests=true clean verify && java -jar war/target/jenkins.war can be used to compile and run Jenkins from source.
jenkins-infra-bot commentedon Jul 1, 2014
Raw content of original comment:
From what I can tell, the problem is with the CLI, and specifically, its use of the Jenkins remoting library. By going back to the 1.546 version of the CLI and adjusting the version of the remoting library used, I determined that the leaks occur on version 2.38 but not on 2.37. Those two version are significantly different, so I'm not sure of the exact change that introduced the problem.
jenkins-infra-bot commentedon Jul 1, 2014
Raw content of original comment:
Adjusted issue metadata based on Matthew Reiter's comment. Damon G: If you could try doing the same to confirm you're seeing the same issue would be helpful.
jenkins-infra-bot commentedon Jul 1, 2014
Raw content of original comment:
Damon and I are working on the same system, so I can confirm he is seeing the same issue.
13 remaining items
jenkins-infra-bot commentedon Jul 29, 2014
Raw content of original comment:
Integrated in
jenkins_main_trunk #3560
[FIXED JENKINS-23248] (Revision 5b7d59a)
Result = SUCCESS
kohsuke : 5b7d59a71aa01d0c430d98d0b6879c3662a6248b
Files :
jenkins-infra-bot commentedon Sep 7, 2014
Raw content of original comment:
Code changed in jenkins
User: Kohsuke Kawaguchi
Path:
pom.xml
http://jenkins-ci.org/commit/jenkins/0ae16264db1b07c011e76aee81442033caf5c72a
Log:
[FIXED JENKINS-23248]
Integrated remoting.jar that fixes the problem
(cherry picked from commit 5b7d59a)
Conflicts:
changelog.html
jenkins-infra-bot commentedon Jun 25, 2017
jayanmn:
Raw content of original comment:
Is this really fixed?
I am on Jenkins ver. 2.32.3. Tested with script to run a dummy job --that seems to have leaked handles - one per invocation.
JAVA_CMD=java
CLI_JAR=/jenkins/jenkins-cli.jar
JOB_NAME=dummyJob
URL=http://testserver:8080/
for i in {1..1000}
do
$JAVA_CMD -jar $CLI_JAR -s $URL build $JOB_NAME
done
jenkins-infra-bot commentedon Jun 25, 2017
Raw content of original comment:
jayanmn There's really no point testing this on an outdated version of Jenkins when 2.46.2 contained a major rewrite of the CLI. If you're affected by an issue that looks like this one, start by upgrading to a recent version of Jenkins.
If the problem persists (given the age of this issue and major changes since), please file a new bug.
jenkins-infra-bot commentedon Jun 26, 2017
jayanmn:
Raw content of original comment:
Thanks, great news.
I posted my case as this was claimed to be addressed long back in 1.563.... I will pick 2.46.2 and test it in next couple of days..
jenkins-infra-bot commentedon Jun 26, 2017
Raw content of original comment:
jayanmn
Right; an indicator you might be seeing a different bug, so please file a new issue if it's still present. It's very confusing to have multiple reports of the "same" issue that have very different causes.
jenkins-infra-bot commentedon Jul 25, 2017
jayanmn:
Raw content of original comment:
tested with Jenkins ver. 2.60.2 - I will create a new bug as you suggested.
jenkins-infra-bot commentedon Nov 24, 2025
[Original
relates_tofrom Jira: JENKINS-23572]