[JENKINS-15331] Workaround Windows unpredictable file locking in Util.deleteContentsRecursive

Originally reported by pjdarton, imported from: Workaround Windows unpredictable file locking in Util.deleteContentsRecursive

status: Resolved
priority: Major
component(s): core
resolution: Fixed
resolved: 2016-05-03T23:32:59+00:00
votes: 28
watchers: 36
imported: 2025-11-24

environment: Microsoft Windows

2 attachments

pjdarton:

Original comment link

Raw content of original comment:

Note: This file locking behavior also causes non-Jenkins issues, e.g. deleting multiple folders using Windows explorer will sometimes leave one (usually empty) folder behind, and even a simple "RD /S /Q MyFolder" will sometimes fail to delete the folder on its first attempt. In these cases, simply retrying the operation will succeed. Personally, I think it's a Windows "feature".

As a workaround, I've wrapped most of my calls to Ant's <delete> task in <retry>, and this has eliminated this problem from any of my builds that manage to start BUT this doesn't help if Jenkins doesn't get as far as running my builds. e.g. I'm using the accurev plugin for my SCM and it cleans the working directory before it grabs the source - I typically get about a 1% failure rate at this stage. Whilst 1% is not a blocking issue, it's not reliable, which is not what one wants from a build system.

Personally, I've found that excluding the build areas from Search & anti-virus helps reduce the problem, but it is insufficient to stop these failures completely (at least on Windows 7) - something, somewhere, will still lock files, sometimes, but any investigation (after the build has failed failed) shows that no process has the file "open".

Note: This file locking behavior also causes non-Jenkins issues, e.g. deleting multiple folders using Windows explorer will sometimes leave one (usually empty) folder behind, and even a simple "RD /S /Q MyFolder" will sometimes fail to delete the folder on its first attempt. In these cases, simply retrying the operation will succeed. Personally, I think it's a Windows "feature".

As a workaround, I've wrapped most of my calls to Ant's task in , and this has eliminated this problem from any of my builds that manage to start BUT this doesn't help if Jenkins doesn't get as far as running my builds.
e.g. I'm using the accurev plugin for my SCM and it cleans the working directory before it grabs the source - I typically get about a 1% failure rate at this stage. Whilst 1% is not a blocking issue, it's not reliable, which is not what one wants from a build system.

Personally, I've found that excluding the build areas from Search & anti-virus helps reduce the problem, but it is insufficient to stop these failures completely (at least on Windows 7) - something, somewhere, will still lock files, sometimes, but any investigation (after the build has failed failed) shows that no process has the file "open".

pjdarton:

Original comment link

Raw content of original comment:

Features:

Added two new system properties that control behavior: "Util.deletionRetries" (an integer, defaults to 3) and "Util.deletionRetryWait" (an integer, defaults to 500ms).
Delete operations that affect directories now try to delete the entire contents of the directory, continuing on to subfolders etc even after encountering files that wouldn't die, before eventually throwing an exception about what wouldn't die. i.e. if a folder has a file "a", "b" and "c", and you can't delete "b", then "a" and "c" would get deleted (and you'll still get the exception about "b").
Delete operations now have multiple attempts at deleting things, so if not everything could be deleted first time around, maybe they'll get deleted 2nd/3rd etc time around. An exception is only thrown if all retry attempts are exhausted and there are still files/directories that won't delete.
Added some unit tests for these methods.
After posting this back in October 2012, I built a version of Jenkins LTS with this patch applied. I've been using it at work for all our development stuff and I've not had file locking problems since. I'm pretty confident that it fixes the problem.

Disclaimers:

I've not tested this on Linux (or the unit-tests). It should be harmless (behaviorial changes are conditional on being on Windows), but it'd be worth running the unit-tests on Linux just to verify that.

Features:

Added two new system properties that control behavior: "Util.deletionRetries" (an integer, defaults to 3) and "Util.deletionRetryWait" (an integer, defaults to 500ms).
Delete operations that affect directories now try to delete the entire contents of the directory, continuing on to subfolders etc even after encountering files that wouldn't die, before eventually throwing an exception about what wouldn't die. i.e. if a folder has a file "a", "b" and "c", and you can't delete "b", then "a" and "c" would get deleted (and you'll still get the exception about "b").
Delete operations now have multiple attempts at deleting things, so if not everything could be deleted first time around, maybe they'll get deleted 2nd/3rd etc time around. An exception is only thrown if all retry attempts are exhausted and there are still files/directories that won't delete.
Added some unit tests for these methods.
After posting this back in October 2012, I built a version of Jenkins LTS with this patch applied. I've been using it at work for all our development stuff and I've not had file locking problems since. I'm pretty confident that it fixes the problem.

Disclaimers:

I've not tested this on Linux (or the unit-tests). It should be harmless (behaviorial changes are conditional on being on Windows), but it'd be worth running the unit-tests on Linux just to verify that.

pjdarton:

Original comment link

Raw content of original comment:

~~JENKINS-15331~~ should fix ~~JENKINS-10905~~.

pjdarton:

Original comment link

Raw content of original comment:

Uploaded git patch file; this was produced using the git command-line and isn't claiming to change the entire file. This will probably be a lot easier to merge.

This is my "New-and-improved" solution. In addition to retrying the deletes, this also calls System.gc() if it's on Windows (a tactic that's also used in Apache Ant's Delete task to workaround the same problem).

Uploaded git patch file; this was produced using the git command-line and isn't claiming to change the entire file. This will probably be a lot easier to merge.

This is my "New-and-improved" solution.
In addition to retrying the deletes, this also calls System.gc() if it's on Windows (a tactic that's also used in Apache Ant's Delete task to workaround the same problem).

pjdarton:

Original comment link

Raw content of original comment:

Have re-done my GitHub pull request to reflect the new changes (and to fix the CRLF issue with the previous pull request). New pull request is [FIXED JENKINS-15331] Workaround Windows unpredictable file locking in Util.deleteContentsRecursive #615

Have re-done my GitHub pull request to reflect the new changes (and to fix the CRLF issue with the previous pull request).
New pull request is #615

pjdarton:

Original comment link

Raw content of original comment:

I've now been running the LTS Jenkins build (1.480.1) with this patch applied at work for a while. I've not seen any builds failing due to "file in use" since. I would therefore recommend that this patch / pull-request be incorporated into the main branch ASAP, and to the next LTS release.

I've now been running the LTS Jenkins build (1.480.1) with this patch applied at work for a while.
I've not seen any builds failing due to "file in use" since.
I would therefore recommend that this patch / pull-request be incorporated into the main branch ASAP, and to the next LTS release.

dankirkd:

Original comment link

Raw content of original comment:

I believe this is also the root cause of ~~JENKINS-15852~~. The Git Plugin has a call in GitAPI to FilePath.deleteRecursive(), which in turn calls Util.deleteRecursive(). It is almost immediately trying to delete a workspace that has just been created. Additionally, we have encryption and McAfee software monitoring files that could be locking them.

pjdarton:

Original comment link

Raw content of original comment:

File-locking is the bane of anyone running any kind of automated system on Windows, so I'd agree that this might well solve the problem (as long as you're sure that the Git code doesn't use the workspace as its current directory, as no amount of retrying will change that).

I also have anti-virus stuff running on my build slaves, and despite that I've not noticed any builds fail due to file-locking issues since I started running a custom build of Jenkins LTS that has this fix in it. I think that this amounts to a fair amount of circumstantial evidence that this fix works.

File-locking is the bane of anyone running any kind of automated system on Windows, so I'd agree that this might well solve the problem (as long as you're sure that the Git code doesn't use the workspace as its current directory, as no amount of retrying will change that).

I also have anti-virus stuff running on my build slaves, and despite that I've not noticed any builds fail due to file-locking issues since I started running a custom build of Jenkins LTS that has this fix in it.
I think that this amounts to a fair amount of circumstantial evidence that this fix works.

per_westling:

Original comment link

Raw content of original comment:

This is a very interesting patch, as we encounter a similar bug several times a week.

Will this be added to the Jenkins releases in the near future?

This is a very interesting patch, as we encounter a similar bug several times a week.

Will this be added to the Jenkins releases in the near future?

brian3791:

Original comment link

Raw content of original comment:

We are encountering a similar problem that I originally attributed to some kind of weird conflict between

Use private Maven repository

and

SCM / Subversion / Check-out Strategy / Always checkout a fresh copy

Not sure why a Maven repo entry local to the workspace would be locked before the code is even checked out. Maven shouldn't even be running yet and no process other than the Jenkins job which uses this workspace should be referencing a workspace private maven repo entry.

Environment:

Jenkins 1.517
Maven 3.4 (-Xmx1536m -XX:MaxPermSize=256m)
Java 1.7.0_15-b03 Oracle JVM 64-bit
Windows 2008 Server 64-bit
Clean server with no virus scanner, indexing, etc.
Dell PowerEdge 2950
PERC 5i Serial Attached SCSI controller This machine has 2 CPUs with 4 cores each (a total of 8 cores). This server is configured with a single C: partition formed from two physical drives in RAID 1.

Build Console Output

Started by timer
Building in workspace C:\Jenkins\jobs\Maxview-Daily-Build-6.2-WINDOWS-Trunk\workspace
Cleaning local Directory .
java.nio.file.FileSystemException: C:\Jenkins\jobs\Maxview-Daily-Build-6.2-WINDOWS-Trunk\workspace\.\.repository\ant\ant-antlr\1.6.5\ant-antlr-1.6.5.jar: The process cannot access the file because it is being used by another process.

	at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
	at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
	at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
	at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
	at sun.nio.fs.AbstractFileSystemProvider.delete(Unknown Source)
	at java.nio.file.Files.delete(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at hudson.Util.deleteFile(Util.java:237)
	at hudson.Util.deleteRecursive(Util.java:305)
	at hudson.Util.deleteContentsRecursive(Util.java:202)
	at hudson.Util.deleteRecursive(Util.java:296)
	at hudson.Util.deleteContentsRecursive(Util.java:202)
	at hudson.Util.deleteRecursive(Util.java:296)
	at hudson.Util.deleteContentsRecursive(Util.java:202)
	at hudson.Util.deleteRecursive(Util.java:296)
	at hudson.Util.deleteContentsRecursive(Util.java:202)
	at hudson.Util.deleteRecursive(Util.java:296)
	at hudson.Util.deleteContentsRecursive(Util.java:202)
	at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:75)
	at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:153)
	at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:903)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:884)
	at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:867)
	at hudson.FilePath.act(FilePath.java:905)
	at hudson.FilePath.act(FilePath.java:878)
	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:843)
	at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:781)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1369)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:676)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:581)
	at hudson.model.Run.execute(Run.java:1576)
	at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:486)
	at hudson.model.ResourceController.execute(ResourceController.java:88)
	at hudson.model.Executor.run(Executor.java:241)

We are encountering a similar problem that I originally attributed to some kind of weird conflict between

Use private Maven repository

and

SCM / Subversion / Check-out Strategy / Always checkout a fresh copy

Not sure why a Maven repo entry local to the workspace would be locked before the code is even checked out. Maven shouldn't even be running yet and no process other than the Jenkins job which uses this workspace should be referencing a workspace private maven repo entry.

Environment:

Jenkins 1.517
Maven 3.4 (-Xmx1536m -XX:MaxPermSize=256m)
Java 1.7.0_15-b03 Oracle JVM 64-bit
Windows 2008 Server 64-bit
Clean server with no virus scanner, indexing, etc.
Dell PowerEdge 2950
PERC 5i Serial Attached SCSI controller
This machine has 2 CPUs with 4 cores each (a total of 8 cores).
This server is configured with a single C: partition formed from two physical drives in RAID 1.

Build Console Output

Started by timer
Building in workspace C:\Jenkins\jobs\Maxview-Daily-Build-6.2-WINDOWS-Trunk\workspace
Cleaning local Directory .
java.nio.file.FileSystemException: C:\Jenkins\jobs\Maxview-Daily-Build-6.2-WINDOWS-Trunk\workspace\.\.repository\ant\ant-antlr\1.6.5\ant-antlr-1.6.5.jar: The process cannot access the file because it is being used by another process.
at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
at sun.nio.fs.AbstractFileSystemProvider.delete(Unknown Source)
at java.nio.file.Files.delete(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at hudson.Util.deleteFile(Util.java:237)
at hudson.Util.deleteRecursive(Util.java:305)
at hudson.Util.deleteContentsRecursive(Util.java:202)
at hudson.Util.deleteRecursive(Util.java:296)
at hudson.Util.deleteContentsRecursive(Util.java:202)
at hudson.Util.deleteRecursive(Util.java:296)
at hudson.Util.deleteContentsRecursive(Util.java:202)
at hudson.Util.deleteRecursive(Util.java:296)
at hudson.Util.deleteContentsRecursive(Util.java:202)
at hudson.Util.deleteRecursive(Util.java:296)
at hudson.Util.deleteContentsRecursive(Util.java:202)
at hudson.scm.subversion.CheckoutUpdater$1.perform(CheckoutUpdater.java:75)
at hudson.scm.subversion.WorkspaceUpdater$UpdateTask.delegateTo(WorkspaceUpdater.java:153)
at hudson.scm.SubversionSCM$CheckOutTask.perform(SubversionSCM.java:903)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:884)
at hudson.scm.SubversionSCM$CheckOutTask.invoke(SubversionSCM.java:867)
at hudson.FilePath.act(FilePath.java:905)
at hudson.FilePath.act(FilePath.java:878)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:843)
at hudson.scm.SubversionSCM.checkout(SubversionSCM.java:781)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1369)
at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:676)
at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:88)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:581)
at hudson.model.Run.execute(Run.java:1576)
at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:486)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:241)

pjdarton:

Original comment link

Raw content of original comment:

Yes, that's the kind of error you can get when doing any filesystem access on Windows (whether from Java or anything else) - basically, if you're on Windows, ANY file operation can fail (at any point) with a "file locked by another process" error and you need to catch these and retry (as, if you retry after a small delay, whatever process was sabotaging your operation will have moved on). It's also the kind of error that I kept getting that prompted me to create this patch, and I can state (with some confidence now) that this fixed it for me.

Note: under Java, the process sabotaging your file operation might well be your own - if you don't manually close file handles but just rely on the garbage collector to do so, attempts to delete those files will fail until the GC has run. This is why I run the GC as well, just in case (not sure if that was a deciding factor, but it's what Ant does and it worked for me).

Yes, that's the kind of error you can get when doing any filesystem access on Windows (whether from Java or anything else) - basically, if you're on Windows, ANY file operation can fail (at any point) with a "file locked by another process" error and you need to catch these and retry (as, if you retry after a small delay, whatever process was sabotaging your operation will have moved on).
It's also the kind of error that I kept getting that prompted me to create this patch, and I can state (with some confidence now) that this fixed it for me.

Note: under Java, the process sabotaging your file operation might well be your own - if you don't manually close file handles but just rely on the garbage collector to do so, attempts to delete those files will fail until the GC has run. This is why I run the GC as well, just in case (not sure if that was a deciding factor, but it's what Ant does and it worked for me).

dhs:

Original comment link

Raw content of original comment:

To make things even worse, the message "file locked by another process" doesn't necessarily mean that the file IS locked by another process. Notepad++, for example, prints this message even if the real error is "permission denied". Took me quite some time to find out...

pjdarton:

Original comment link

Raw content of original comment:

I've just attached a new patch file "0001-Proposed-solution-to-~~JENKINS-15331~~.patch" This one is based on the current Jenkins master trunk (at the time of writing, that's aimed at 1.560-SNAPSHOT).

This is slightly different than the earlier patch: 1) The configuration for garbage-collection when deletes fail now defaults to "false" on all platforms. 2) The garbage-collection should now get called if it's enabled (the previous version had a bug).

I've just attached a new patch file "0001-Proposed-solution-to-~~JENKINS-15331~~.patch"
This one is based on the current Jenkins master trunk (at the time of writing, that's aimed at 1.560-SNAPSHOT).

This is slightly different than the earlier patch:
1) The configuration for garbage-collection when deletes fail now defaults to "false" on all platforms.
2) The garbage-collection should now get called if it's enabled (the previous version had a bug).

pjdarton:

Original comment link

Raw content of original comment:

There's an open pull request that fixes this on github, [FIX JENKINS-15331] Retry failed file delete operations #1209 That superceeds the patches etc here.

There's an open pull request that fixes this on github, #1209
That superceeds the patches etc here.

laurent_malvert:

Original comment link

Raw content of original comment:

Also see this issue (and have for a while) up until at least the current 1.5776 release.

As mentioned by others above, it's a common gripe with NTFS, and sadly your chances of hitting that issue increase considerably with large checkouts/workspaces (it struggles to delete efficiently a large number of files).

It would be great if this patch could finally be merged into the head.

Also see this issue (and have for a while) up until at least the current 1.5776 release.

As mentioned by others above, it's a common gripe with NTFS, and sadly your chances of hitting that issue increase considerably with large checkouts/workspaces (it struggles to delete efficiently a large number of files).

It would be great if this patch could finally be merged into the head.

oleg_nenashev:

Original comment link

Raw content of original comment:

Actually we still can merge it to .3 if olivergondza agrees. But I'm not so happy about it since RC is under testing now. Regarding .4, it will unlikely happen according to the current release model. Needs a wide discussion in the developer list.

BR, Oleg

Actually we still can merge it to .3 if olivergondza agrees. But I'm not so happy about it since RC is under testing now.
Regarding .4, it will unlikely happen according to the current release model. Needs a wide discussion in the developer list.

BR, Oleg

danielbeck:

Original comment link

Raw content of original comment:

We don't do .4's, except when we mess up so badly there's no way around it, but this doesn't qualify.

olivergondza:

Original comment link

Raw content of original comment:

I decided not to squeeze this into .3 (last in its line) for stability's sake. We need to be extra careful as we do not do much testing on windows, unfortunately.

olivergondza:

Original comment link

Raw content of original comment:

Consumed by 2.7.X line so need to backport.

[Original is_duplicated_by from Jira: JENKINS-10905]

[Original is_duplicated_by from Jira: JENKINS-24481]

[Original is_duplicated_by from Jira: JENKINS-14808]

[Original is_duplicated_by from Jira: JENKINS-3053]

[Original relates_to from Jira: JENKINS-3053]

[Original relates_to from Jira: JENKINS-17995]

Navigation Menu

Repository navigation

[JENKINS-15331] Workaround Windows unpredictable file locking in Util.deleteContentsRecursive #12981

20 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[JENKINS-15331] Workaround Windows unpredictable file locking in Util.deleteContentsRecursive #12981

Description

Activity

jenkins-infra-bot commented on Sep 27, 2012

jenkins-infra-bot commented on Oct 1, 2012

jenkins-infra-bot commented on Oct 16, 2012

jenkins-infra-bot commented on Nov 9, 2012

jenkins-infra-bot commented on Nov 12, 2012

jenkins-infra-bot commented on Dec 7, 2012

jenkins-infra-bot commented on Jan 29, 2013

jenkins-infra-bot commented on Jan 29, 2013

jenkins-infra-bot commented on Jun 18, 2013

jenkins-infra-bot commented on Jul 10, 2013

jenkins-infra-bot commented on Jul 10, 2013

jenkins-infra-bot commented on Jul 12, 2013

jenkins-infra-bot commented on Apr 11, 2014

jenkins-infra-bot commented on Jun 1, 2014

jenkins-infra-bot commented on Aug 27, 2014

20 remaining items

jenkins-infra-bot commented on Jun 1, 2016

jenkins-infra-bot commented on Jun 1, 2016

jenkins-infra-bot commented on Jun 8, 2016

jenkins-infra-bot commented on Jun 20, 2016

jenkins-infra-bot commented on Nov 24, 2025

jenkins-infra-bot commented on Nov 24, 2025

jenkins-infra-bot commented on Nov 24, 2025

jenkins-infra-bot commented on Nov 24, 2025

jenkins-infra-bot commented on Nov 24, 2025

jenkins-infra-bot commented on Nov 24, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions