[JENKINS-16319] Failure to delete old config files during rekeying on Windows

[jenkins-infra-bot]

Started re-keying Wed Jan 09 12:59:32 EST 2013
Scanning C:\JenkinsService\com.michelin.cio.hudson.plugins.copytoslave.CopyToSlaveBuildWrapper.xml
ERROR: Failed to rewrite C:\JenkinsService\hudson.scm.CVSSCM.xml
java.io.IOException: Unable to delete C:\JenkinsService\hudson.scm.CVSSCM.xml
	at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:112)
	at hudson.util.SecretRewriter.rewrite(SecretRewriter.java:121)
	at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:170)
	at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:143)
	at jenkins.security.RekeySecretAdminMonitor$RekeyThread.run(RekeySecretAdminMonitor.java:182)
ERROR: Failed to rewrite C:\JenkinsService\users\andrewg\config.xml
java.io.IOException: Unable to delete C:\JenkinsService\users\andrewg\config.xml
	at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:112)
	at hudson.util.SecretRewriter.rewrite(SecretRewriter.java:121)
	at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:170)
	at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:182)
	at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:182)
	at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:143)
	at jenkins.security.RekeySecretAdminMonitor$RekeyThread.run(RekeySecretAdminMonitor.java:182)
Completed re-keying 0 files on Wed Jan 09 12:59:33 EST 2013

Jenkins always fails to edit and delete these files as the Service will have a lock on them. If I stop the service I won't be able to get to Jenkins to rerun the rekey job.

---

Originally reported by jglick, imported from: Failure to delete old config files during rekeying on Windows

• status: Resolved
• priority: Critical
• component(s): core
• label(s): 1.480.3-fixed
• resolution: Fixed
• resolved: 2013-01-30T20:32:19+00:00
• votes: 15
• watchers: 20
• imported: 2025-11-24

Raw content of original issue

Started re-keying Wed Jan 09 12:59:32 EST 2013
Scanning C:\JenkinsService\com.michelin.cio.hudson.plugins.copytoslave.CopyToSlaveBuildWrapper.xml
ERROR: Failed to rewrite C:\JenkinsService\hudson.scm.CVSSCM.xml
java.io.IOException: Unable to delete C:\JenkinsService\hudson.scm.CVSSCM.xml
	at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:112)
	at hudson.util.SecretRewriter.rewrite(SecretRewriter.java:121)
	at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:170)
	at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:143)
	at jenkins.security.RekeySecretAdminMonitor$RekeyThread.run(RekeySecretAdminMonitor.java:182)
ERROR: Failed to rewrite C:\JenkinsService\users\andrewg\config.xml
java.io.IOException: Unable to delete C:\JenkinsService\users\andrewg\config.xml
	at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:112)
	at hudson.util.SecretRewriter.rewrite(SecretRewriter.java:121)
	at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:170)
	at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:182)
	at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:182)
	at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:143)
	at jenkins.security.RekeySecretAdminMonitor$RekeyThread.run(RekeySecretAdminMonitor.java:182)
Completed re-keying 0 files on Wed Jan 09 12:59:33 EST 2013


Jenkins always fails to edit and delete these files as the Service will have a lock on them.  If I stop the service I won't be able to get to Jenkins to rerun the rekey job.

• environment: Jenkins as a Windows service

[jenkins-infra-bot]

beta:

• Original comment link
• Raw content of original comment:

  I disable Jenkins service, and then start Jenkins from command line using java -jar, the rewriting issue remains.
  
  Have any workaround for this problem?
  

I disable Jenkins service, and then start Jenkins from command line using java -jar, the rewriting issue remains.

Have any workaround for this problem?

[jenkins-infra-bot]

wolfwolfswinkel:

• Original comment link
• Raw content of original comment:

  This seems to be similar to the issues reported in https://groups.google.com/d/topic/jenkinsci-users/hBRb8XqNQyM/discussion
  

This seems to be similar to the issues reported in https://groups.google.com/d/topic/jenkinsci-users/hBRb8XqNQyM/discussion

[jenkins-infra-bot]

alexlombardi:

• Original comment link
• Raw content of original comment:

  I am seeing an even bigger problem with this issue. I have Jenkins running on several 2k3 servers as window services. After my original upgrade to 1.499 (from 1.489), I was unable to get jenkins to recognize/start any of the slaves which all where running as windows services. There is a bug report already for this issue: https://issues.jenkins-ci.org/browse/JENKINS-16346. 
  
  I had unfortunately started the re-keying process already when it became necessary to revert to the older version of Jenkins to get things back up and running. The revert worked fine, and I was able to trouble shoot the issue on a different machine. The answer seemed to be to give Anonymous users conenction capability in the Global Security settings in the Project-based Matrix Authorization Strategy for a system using AD.
  
  I re-upgraded my main system to 1.499, applied the security setting, got my slaves running, triggered the re-keying process, and was on my marry way. However, once that completed, any attempt to view the log created results in Jenkins crashing and the windows service it runs on being terminated. I manually looked at the log files and see many of the failed conversion steps as mentioned in this bug report.
  

I am seeing an even bigger problem with this issue. I have Jenkins running on several 2k3 servers as window services. After my original upgrade to 1.499 (from 1.489), I was unable to get jenkins to recognize/start any of the slaves which all where running as windows services. There is a bug report already for this issue: https://issues.jenkins-ci.org/browse/JENKINS-16346.

I had unfortunately started the re-keying process already when it became necessary to revert to the older version of Jenkins to get things back up and running. The revert worked fine, and I was able to trouble shoot the issue on a different machine. The answer seemed to be to give Anonymous users conenction capability in the Global Security settings in the Project-based Matrix Authorization Strategy for a system using AD.

I re-upgraded my main system to 1.499, applied the security setting, got my slaves running, triggered the re-keying process, and was on my marry way. However, once that completed, any attempt to view the log created results in Jenkins crashing and the windows service it runs on being terminated. I manually looked at the log files and see many of the failed conversion steps as mentioned in this bug report.

[jenkins-infra-bot]

romanhar:

• Original comment link
• Raw content of original comment:

  It repeats same error all the time, even after granting RW rights for every standard user to whole folder Jenkins in "Program Files":
  ERROR: Failed to rewrite C:\Program Files\Jenkins\jenkins.plugins.publish_over_ssh.BapSshPublisherPlugin.xml
  java.io.IOException: Unable to delete C:\Program Files\Jenkins\jenkins.plugins.publish_over_ssh.BapSshPublisherPlugin.xml
  	at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:112)
  	at hudson.util.SecretRewriter.rewrite(SecretRewriter.java:121)
  	at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:170)
  	at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:143)
  	at jenkins.security.RekeySecretAdminMonitor$RekeyThread.run(RekeySecretAdminMonitor.java:182)
  
  For me it seems, that this rekey is made to search for any XML file in jenkins subfolders. Is it problem if it tries to rekey this xml? I am thinking that it is maybe not necessary for some XMLs... And in this case I can probably assume, that rekey is successful. Can somebody confirm my assumption?
  

It repeats same error all the time, even after granting RW rights for every standard user to whole folder Jenkins in "Program Files":
ERROR: Failed to rewrite C:\Program Files\Jenkins\jenkins.plugins.publish_over_ssh.BapSshPublisherPlugin.xml
java.io.IOException: Unable to delete C:\Program Files\Jenkins\jenkins.plugins.publish_over_ssh.BapSshPublisherPlugin.xml
at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:112)
at hudson.util.SecretRewriter.rewrite(SecretRewriter.java:121)
at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:170)
at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:143)
at jenkins.security.RekeySecretAdminMonitor$RekeyThread.run(RekeySecretAdminMonitor.java:182)

For me it seems, that this rekey is made to search for any XML file in jenkins subfolders. Is it problem if it tries to rekey this xml? I am thinking that it is maybe not necessary for some XMLs... And in this case I can probably assume, that rekey is successful. Can somebody confirm my assumption?

[jenkins-infra-bot]

jglick:

• Original comment link
• Raw content of original comment:

  @alexlombardi keep discussion about authentication in JENKINS-16273. This is about rekeying only. And BTW giving anonymous users slave connect permissions is insecure (unless your network or servlet container adds independent security layers).
  
  @romanhar Jenkins will only attempt to rewrite a file if it found some changes to make, meaning that the existing file contained insecurely encrypted passwords, so yes I think this is a problem.
  
  I suspect that the root cause here is code in core or plugins which opens input streams on config files and fails to promptly close them in a finally block. On Windows, an open InputStream holds a mandatory file lock until it is garbage collected.
  

@​alexlombardi keep discussion about authentication in JENKINS-16273. This is about rekeying only. And BTW giving anonymous users slave connect permissions is insecure (unless your network or servlet container adds independent security layers).

@​romanhar Jenkins will only attempt to rewrite a file if it found some changes to make, meaning that the existing file contained insecurely encrypted passwords, so yes I think this is a problem.

I suspect that the root cause here is code in core or plugins which opens input streams on config files and fails to promptly close them in a finally block. On Windows, an open InputStream holds a mandatory file lock until it is garbage collected.

[jenkins-infra-bot]

damienfinck67:

• Original comment link
• Raw content of original comment:

  Hello,
  
  I have the same problem on my Server and on my personal computer.
  
    [...]
    ERROR: Failed to rewrite C:\PIC\Jenkins\hudson.scm.CVSSCM.xml
    [...]
  
  How I can fix this problem ?
  
  
  Instance Jenkins of my Server
  OS : Windows 2008r2 64 bits
  Jenkins : 1.498
  
  Instance Jenkins of my Personnal Computer
  OS : Windows 7 64 bits
  Jenkins : 1.498 and after update 1.499
  

Hello,

I have the same problem on my Server and on my personal computer.

[...]
ERROR: Failed to rewrite C:\PIC\Jenkins\hudson.scm.CVSSCM.xml
[...]

How I can fix this problem ?

Instance Jenkins of my Server
OS : Windows 2008r2 64 bits
Jenkins : 1.498

Instance Jenkins of my Personnal Computer
OS : Windows 7 64 bits
Jenkins : 1.498 and after update 1.499

[jenkins-infra-bot]

beta:

• Original comment link
• Raw content of original comment:

  @jglick I have following plugin installed and enabled:
  
  
  
  
  Name
  Version
  
  
  AnsiColor
  0.3.1
  
  
  Audit Trail
  1.7
  
  
  Build Flow Plugin
  0.6
  
  
  Checkstyle Plug-in
  3.32
  
  
  Claim Plugin
  1.7
  
  
  Compact Columns
  1.9
  
  
  Configuration Slicing plugin
  1.36
  
  
  Copy Artifact Plugin
  1.25
  
  
  Dashboard View
  2.4
  
  
  Dependency Graph Viewer Plugin
  0.10
  
  
  Duplicate Code Scanner Plug-in
  2.33
  
  
  External Monitor Job Type Plugin
  1.1
  
  
  FindBugs Plug-in
  4.45
  
  
  Fortify 360 Plugin
  3.6
  
  
  Git server plugin
  1.1
  
  
  javadoc
  1.0
  
  
  Jenkins Artifact Deployer Plug-in
  0.26
  
  
  Jenkins Cobertura Plugin
  1.8
  
  
  Jenkins Continuous Integration game
  1.19
  
  
  Jenkins disk-usage plugin
  0.18
  
  
  Jenkins Email Extension Plugin
  2.25
  
  
  Jenkins GIT plugin
  1.1.26
  
  
  Jenkins Gravatar plugin
  1.1
  
  
  Jenkins Job Configuration History Plugin
  2.0
  
  
  Jenkins jQuery plugin
  1.7.2-1
  
  
  Jenkins jQuery UI plugin
  1.0.2
  
  
  Jenkins Mailer Plugin
  1.4
  
  
  Jenkins promoted builds plugin
  2.8
  
  
  Jenkins Slave SetupPlugin
  1.6
  
  
  Jenkins SLOCCount Plug-in
  1.8
  
  
  Jenkins SSH Slaves plugin
  0.22
  
  
  Jenkins Subversion Plug-in
  1.44
  
  
  Jenkins Translation Assistance plugin
  1.10
  
  
  LDAP Plugin
  1.2
  
  
  Maven 2 Project Plugin
  1.498
  
  
  pam-auth
  1.0
  
  
  PMD Plug-in
  3.33
  
  
  Priority Sorter
  1.3
  
  
  Radiator View Plugin
  1.13
  
  
  Redmine Plugin
  0.11-SNAPSHOT (private-12/14/2012 15:48-Tang)
  
  
  SCM Sync Configuration Plugin
  0.0.6.1
  
  
  Static Analysis Collector Plug-in
  1.34
  
  
  Static Analysis Utilities
  1.48
  
  
  Task Scanner Plug-in
  4.35
  
  
  thinBackup
  1.6.2
  
  
  Timestamper
  1.5
  
  
  Token Macro Plugin
  1.5.1
  
  
  View Job Filters
  1.22
  
  
  Warnings Plugin
  4.18
  
  
  WAS Builder Plugin
  1.6.1
  
  
  
  
  
  
  I will try to disable some of them to investigate this issue while our Jenkins is not in a rush.
  

@​jglick I have following plugin installed and enabled:

Name	Version
AnsiColor	0.3.1
Audit Trail	1.7
Build Flow Plugin	0.6
Checkstyle Plug-in	3.32
Claim Plugin	1.7
Compact Columns	1.9
Configuration Slicing plugin	1.36
Copy Artifact Plugin	1.25
Dashboard View	2.4
Dependency Graph Viewer Plugin	0.10
Duplicate Code Scanner Plug-in	2.33
External Monitor Job Type Plugin	1.1
FindBugs Plug-in	4.45
Fortify 360 Plugin	3.6
Git server plugin	1.1
javadoc	1.0
Jenkins Artifact Deployer Plug-in	0.26
Jenkins Cobertura Plugin	1.8
Jenkins Continuous Integration game	1.19
Jenkins disk-usage plugin	0.18
Jenkins Email Extension Plugin	2.25
Jenkins GIT plugin	1.1.26
Jenkins Gravatar plugin	1.1
Jenkins Job Configuration History Plugin	2.0
Jenkins jQuery plugin	1.7.2-1
Jenkins jQuery UI plugin	1.0.2
Jenkins Mailer Plugin	1.4
Jenkins promoted builds plugin	2.8
Jenkins Slave SetupPlugin	1.6
Jenkins SLOCCount Plug-in	1.8
Jenkins SSH Slaves plugin	0.22
Jenkins Subversion Plug-in	1.44
Jenkins Translation Assistance plugin	1.10
LDAP Plugin	1.2
Maven 2 Project Plugin	1.498
pam-auth	1.0
PMD Plug-in	3.33
Priority Sorter	1.3
Radiator View Plugin	1.13
Redmine Plugin	0.11-SNAPSHOT (private-12/14/2012 15:48-Tang)
SCM Sync Configuration Plugin	0.0.6.1
Static Analysis Collector Plug-in	1.34
Static Analysis Utilities	1.48
Task Scanner Plug-in	4.35
thinBackup	1.6.2
Timestamper	1.5
Token Macro Plugin	1.5.1
View Job Filters	1.22
Warnings Plugin	4.18
WAS Builder Plugin	1.6.1

I will try to disable some of them to investigate this issue while our Jenkins is not in a rush.

[jenkins-infra-bot]

beta:

• Original comment link
• Raw content of original comment:

  Disabled "SCM Sync Configuration" and "Jenkins Job Configuration History" Plugins, no luck 
  

Disabled "SCM Sync Configuration" and "Jenkins Job Configuration History" Plugins, no luck

[jenkins-infra-bot]

damienfinck67:

• Original comment link
• Raw content of original comment:

  Hello,
  
  I have try to create a new proper instance of Jenkins (1.496) on a Windows Server 2008r2 with no plugins.
  
  I have juste configure the proxy in Jenkins, and so I have do the update to 1.499.
  
  When I have start the re-key, I have :
  Re-keying log
  
  Started re-keying Fri Jan 18 09:00:02 CET 2013
  Scanning C:\Users\Administrateur\.jenkins\hudson.model.UpdateCenter.xml
  ERROR: Failed to rewrite C:\Users\Administrateur\.jenkins\proxy.xml
  java.io.IOException: Unable to delete C:\Users\Administrateur\.jenkins\proxy.xml
  	at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:112)
  	at hudson.util.SecretRewriter.rewrite(SecretRewriter.java:121)
  	at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:170)
  	at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:143)
  	at jenkins.security.RekeySecretAdminMonitor$RekeyThread.run(RekeySecretAdminMonitor.java:182)
  Completed re-keying 0 files on Fri Jan 18 09:00:02 CET 2013
  

Hello,

I have try to create a new proper instance of Jenkins (1.496) on a Windows Server 2008r2 with no plugins.

I have juste configure the proxy in Jenkins, and so I have do the update to 1.499.

When I have start the re-key, I have :
Re-keying log

Started re-keying Fri Jan 18 09:00:02 CET 2013
Scanning C:\Users\Administrateur\.jenkins\hudson.model.UpdateCenter.xml
ERROR: Failed to rewrite C:\Users\Administrateur\.jenkins\proxy.xml
java.io.IOException: Unable to delete C:\Users\Administrateur\.jenkins\proxy.xml
at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:112)
at hudson.util.SecretRewriter.rewrite(SecretRewriter.java:121)
at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:170)
at hudson.util.SecretRewriter.rewriteRecursive(SecretRewriter.java:143)
at jenkins.security.RekeySecretAdminMonitor$RekeyThread.run(RekeySecretAdminMonitor.java:182)
Completed re-keying 0 files on Fri Jan 18 09:00:02 CET 2013

[jenkins-infra-bot]

cforce:

• Original comment link
• Raw content of original comment:

  Here to on windows serever 2008 with jenkins running on tomcat as windows service with own user in winndows tomcat runs from.
  This user has write/read bright on the complete partition jenkins_home is saved. Till today this alls worked without problems.
  
  Please provide a fix!
  

Here to on windows serever 2008 with jenkins running on tomcat as windows service with own user in winndows tomcat runs from.
This user has write/read bright on the complete partition jenkins_home is saved. Till today this alls worked without problems.

Please provide a fix!

[jenkins-infra-bot]

scm_issue_link:

• Original comment link
• Raw content of original comment:

  Code changed in jenkins
  User: Jesse Glick
  Path:
   changelog.html
   core/src/main/java/hudson/util/SecretRewriter.java
  http://jenkins-ci.org/commit/jenkins/8b8231108fb5930bab5c7e2f20685c9a9d237749
  Log:
    [FIXED JENKINS-16319] Stream ordering problem prevented SecretRewriter from working on Windows.
  
  
  
  
  

Code changed in jenkins
User: Jesse Glick
Path:
changelog.html
core/src/main/java/hudson/util/SecretRewriter.java
http://jenkins-ci.org/commit/jenkins/8b8231108fb5930bab5c7e2f20685c9a9d237749
Log:
[FIXED JENKINS-16319] Stream ordering problem prevented SecretRewriter from working on Windows.

[jenkins-infra-bot]

dogfood:

• Original comment link
• Raw content of original comment:

  Integrated in  jenkins_main_trunk #2205
       [FIXED JENKINS-16319] Stream ordering problem prevented SecretRewriter from working on Windows. (Revision 8b8231108fb5930bab5c7e2f20685c9a9d237749)
  
       Result = SUCCESS
  Jesse Glick : 8b8231108fb5930bab5c7e2f20685c9a9d237749
  Files : 
  
  	
  changelog.html
  	
  core/src/main/java/hudson/util/SecretRewriter.java
  
  
  

Integrated in jenkins_main_trunk #2205
[FIXED JENKINS-16319] Stream ordering problem prevented SecretRewriter from working on Windows. (Revision 8b82311)

Result = SUCCESS
Jesse Glick : 8b8231108fb5930bab5c7e2f20685c9a9d237749
Files :

• changelog.html
• core/src/main/java/hudson/util/SecretRewriter.java

[jenkins-infra-bot]

cforce:

• Original comment link
• Raw content of original comment:

  Still not released, is it?
  

Still not released, is it?

[jenkins-infra-bot]

damienfinck67:

• Original comment link
• Raw content of original comment:

  @cforce : You can see the changelog : http://jenkins-ci.org/changelog > "Upcomming changes" > "Rekeying operation (from SECURITY-49 fix in 1.498) failed on Windows. (issue 16319)"
  
  The fix will be in 1.501
  

@​cforce : You can see the changelog : http://jenkins-ci.org/changelog > "Upcomming changes" > "Rekeying operation (from SECURITY-49 fix in 1.498) failed on Windows. (issue 16319)"

The fix will be in 1.501

[jenkins-infra-bot]

hc_starzinger:

• Original comment link
• Raw content of original comment:

  Hello, 
  
  same problem in LTS Version Jenkins 1.480.2
  please merge fix to LTS Branch.
  
  kr.
  hc
  

Hello,

same problem in LTS Version Jenkins 1.480.2
please merge fix to LTS Branch.

kr.
hc