Skip to content

[JENKINS-12753] PluginManager.dynamicLoad on installed plugin fails on NFS with IOException not RestartRequiredException #12461

Closed
Bug
Closed
[JENKINS-12753] PluginManager.dynamicLoad on installed plugin fails on NFS with IOException not RestartRequiredException#12461
Bug
Labels
component:coreimported-jira-issuepriority:minorresolution:fixed

Description

@jenkins-infra-bot
jenkins-infra-bot
opened on Feb 14, 2012

Got the question from the irc channel, this was initially discussed on the fosdem jenkins booth.

Updating plugins doesn't work when the jenkins home directory is located on nfs, the failure looks like this

Caused by: java.io.IOException: Unable to delete /var/lib/jenkins/plugins/maven-plugin/WEB-INF/lib/.nfs00000000000e897d00000096

The .nfs files appear when deleting a file on nfs that is still open, in this case the jar files in the lib directory are still open and cannot be deleted.

Originally reported by alexlehm, imported from: PluginManager.dynamicLoad on installed plugin fails on NFS with IOException not RestartRequiredException
  • assignee: jglick
  • status: Resolved
  • priority: Minor
  • component(s): core
  • label(s): lock, plugin
  • resolution: Fixed
  • resolved: 2014-05-20T17:56:31+00:00
  • votes: 0
  • watchers: 4
  • imported: 2025-11-24
Raw content of original issue 
Got the question from the irc channel, this was initially discussed on the fosdem jenkins booth.



Updating plugins doesn't work when the jenkins home directory is located on nfs, the failure looks like this



Caused by: java.io.IOException: Unable to delete /var/lib/jenkins/plugins/maven-plugin/WEB-INF/lib/.nfs00000000000e897d00000096



The .nfs files appear when deleting a file on nfs that is still open, in this case the jar files in the lib directory are still open and cannot be deleted.
  • environment: linux, nfs, jenkins 1.450, any plugin

Activity

  1. jenkins-infra-bot commented on Feb 14, 2012

    @jenkins-infra-bot
    jenkins-infra-bot
    on Feb 14, 2012
    Author
    kohsuke:
    • Original comment link
    • Raw content of original comment: 
      This appears to indicate that we are trying to overwrite a plugin currently in use, and that won't work regardless, NFS or not.

    This appears to indicate that we are trying to overwrite a plugin currently in use, and that won't work regardless, NFS or not.

  2. jenkins-infra-bot commented on Feb 19, 2012

    @jenkins-infra-bot
    jenkins-infra-bot
    on Feb 19, 2012
    Author
    mika:
    • Original comment link
    • Raw content of original comment: 
      Being the one who raised this issue on IRC (thanks to Alex for reporting it here): this seems to be a locking issue caused by NFS. I can't reproduce the very same issue without NFS and the open .nfs000000... file handle indicates that there's something going wrong with file deletion.

    Being the one who raised this issue on IRC (thanks to Alex for reporting it here): this seems to be a locking issue caused by NFS. I can't reproduce the very same issue without NFS and the open .nfs000000... file handle indicates that there's something going wrong with file deletion.

  3. jenkins-infra-bot commented on Feb 19, 2012

    @jenkins-infra-bot
    jenkins-infra-bot
    on Feb 19, 2012
    Author
    alexlehm:
    • Original comment link
    • Raw content of original comment: 
      the way open files are handled is different on local filesystems and nfs, the issue may be nfs specific, however if Kohsuke is right, it doesn't work on local storage either but it doesn't give an obvious error message in that case.
      


      if the plugin is in use, the different jar files for the plugin are all loaded into the jvm and are open (this is visible with lsof), if the plugin is updated, the files are still open even though they are already delete (this is perfectly ok on local filesystems), e.g. like this (this is the output of lsof|grep java.*xunit.*guice as one example file):
      


      (removed some columns from the lsof output to fit into the page width)
      


      before:
      

      

      mem       REG     8,1   807021 4067923 .../plugins/xunit/WEB-INF/lib/guice-2.0.1.jar
237r      REG     8,1   807021 4067923 .../plugins/xunit/WEB-INF/lib/guice-2.0.1.jar

      

      

      after:
      

      

      DEL       REG     8,1          4067923 .../plugins/xunit/WEB-INF/lib/guice-2.0.1.jar
237r      REG     8,1   807021 4067923 .../plugins/xunit/WEB-INF/lib/guice-2.0.1.jar (deleted)

      

      

      the jars will not be reloaded even if they were written to the directory in a new version.
      

      on nfs, open deleted files are not directly possible, so instead each file is changed to a .nfs file which causes the exception when trying to delete the directory, but reloading wouldn't work either if the files could be deleted (or e.g. are moved to another dir and delete later).
      

      When updating a plugin when the directory is on a local filesystem, the update page still says "xunit plugin is already installed. Jenkins needs to be restarted for the update to take effect" with a yellow ball, so the update process notices that it cannot update the files and forces suggests to restart. With nfs he never gets to the stage where the message appears, but the problem is the same in both cases.

    the way open files are handled is different on local filesystems and nfs, the issue may be nfs specific, however if Kohsuke is right, it doesn't work on local storage either but it doesn't give an obvious error message in that case.

    if the plugin is in use, the different jar files for the plugin are all loaded into the jvm and are open (this is visible with lsof), if the plugin is updated, the files are still open even though they are already delete (this is perfectly ok on local filesystems), e.g. like this (this is the output of lsof|grep java.*xunit.*guice as one example file):

    (removed some columns from the lsof output to fit into the page width)

    before:

    mem       REG     8,1   807021 4067923 .../plugins/xunit/WEB-INF/lib/guice-2.0.1.jar
237r      REG     8,1   807021 4067923 .../plugins/xunit/WEB-INF/lib/guice-2.0.1.jar

    after:

    DEL       REG     8,1  4067923 .../plugins/xunit/WEB-INF/lib/guice-2.0.1.jar
237r      REG     8,1   807021 4067923 .../plugins/xunit/WEB-INF/lib/guice-2.0.1.jar (deleted)

    the jars will not be reloaded even if they were written to the directory in a new version.

    on nfs, open deleted files are not directly possible, so instead each file is changed to a .nfs file which causes the exception when trying to delete the directory, but reloading wouldn't work either if the files could be deleted (or e.g. are moved to another dir and delete later).

    When updating a plugin when the directory is on a local filesystem, the update page still says "xunit plugin is already installed. Jenkins needs to be restarted for the update to take effect" with a yellow ball, so the update process notices that it cannot update the files and forces suggests to restart. With nfs he never gets to the stage where the message appears, but the problem is the same in both cases.

  4. jenkins-infra-bot commented on May 20, 2014

    @jenkins-infra-bot
    jenkins-infra-bot
    on May 20, 2014
    Author
    jglick:
    • Original comment link
    • Raw content of original comment: 
      It seems that the problem has been misstated because the bug itself masks the problem. When you already have a plugin installed, you cannot dynamically load its update, ever. Normally Jenkins stops you from trying to do so by throwing a RestartRequiredException. Unfortunately when PluginManager.dynamicLoad is checking whether the plugin is already installed, it calls ClassicPluginStrategy.createPluginWrapper which actually unpacks the *.hpi right away, and when on a filesystem with locks, that fails—and the IOException then is reported as if this were the real error, when the error is simply a user misunderstanding.

    It seems that the problem has been misstated because the bug itself masks the problem. When you already have a plugin installed, you cannot dynamically load its update, ever. Normally Jenkins stops you from trying to do so by throwing a RestartRequiredException. Unfortunately when PluginManager.dynamicLoad is checking whether the plugin is already installed, it calls ClassicPluginStrategy.createPluginWrapper which actually unpacks the *.hpi right away, and when on a filesystem with locks, that fails—and the IOException then is reported as if this were the real error, when the error is simply a user misunderstanding.

  5. jenkins-infra-bot commented on May 20, 2014

    @jenkins-infra-bot
    jenkins-infra-bot
    on May 20, 2014
    Author
    jglick:
    • Original comment link
    • Raw content of original comment: 
      Here is a typical stack trace:
      


      

      java.nio.file.FileSystemException: /…/plugins/credentials/WEB-INF/lib/.nfs000000000000…: Device or resource busy 
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91) 
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) 
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) 
	at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244) 
	at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103) 
	at java.nio.file.Files.delete(Files.java:1077) 
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
	at java.lang.reflect.Method.invoke(Method.java:606) 
	at hudson.Util.deleteFile(Util.java:238) 
	at hudson.Util.deleteRecursive(Util.java:301) 
	at hudson.Util.deleteContentsRecursive(Util.java:203) 
	at hudson.Util.deleteRecursive(Util.java:300) 
	at hudson.Util.deleteContentsRecursive(Util.java:203) 
	at hudson.Util.deleteRecursive(Util.java:292) 
	at hudson.Util.deleteContentsRecursive(Util.java:203) 
	at hudson.Util.deleteRecursive(Util.java:292) 
	at hudson.ClassicPluginStrategy.explode(ClassicPluginStrategy.java:423) 
	at hudson.ClassicPluginStrategy.createPluginWrapper(ClassicPluginStrategy.java:128) 
	at hudson.PluginManager.dynamicLoad(PluginManager.java:412) 
	at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:1300)

    Here is a typical stack trace:

    java.nio.file.FileSystemException: /…/plugins/credentials/WEB-INF/lib/.nfs000000000000…: Device or resource busy 
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91) 
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) 
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) 
	at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244) 
	at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103) 
	at java.nio.file.Files.delete(Files.java:1077) 
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
	at java.lang.reflect.Method.invoke(Method.java:606) 
	at hudson.Util.deleteFile(Util.java:238) 
	at hudson.Util.deleteRecursive(Util.java:301) 
	at hudson.Util.deleteContentsRecursive(Util.java:203) 
	at hudson.Util.deleteRecursive(Util.java:300) 
	at hudson.Util.deleteContentsRecursive(Util.java:203) 
	at hudson.Util.deleteRecursive(Util.java:292) 
	at hudson.Util.deleteContentsRecursive(Util.java:203) 
	at hudson.Util.deleteRecursive(Util.java:292) 
	at hudson.ClassicPluginStrategy.explode(ClassicPluginStrategy.java:423) 
	at hudson.ClassicPluginStrategy.createPluginWrapper(ClassicPluginStrategy.java:128) 
	at hudson.PluginManager.dynamicLoad(PluginManager.java:412) 
	at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:1300)

  6. jenkins-infra-bot commented on May 20, 2014

    @jenkins-infra-bot
    jenkins-infra-bot
    on May 20, 2014
    Author
    jglick:
    • Original comment link
    • Raw content of original comment: 
      Tried to reproduce on Windows, where I would expect this to also be a problem, but did not get that far: only Download now and install after restart is offered in /pluginManager/ to begin with.

    Tried to reproduce on Windows, where I would expect this to also be a problem, but did not get that far: only Download now and install after restart is offered in /pluginManager/ to begin with.

  7. jenkins-infra-bot commented on May 20, 2014

    @jenkins-infra-bot
    jenkins-infra-bot
    on May 20, 2014
    Author
    scm_issue_link:
    • Original comment link
    • Raw content of original comment: 
      Code changed in jenkins
User: Jesse Glick
Path:
 changelog.html
 core/src/main/java/hudson/ClassicPluginStrategy.java
 core/src/main/java/hudson/PluginManager.java
 core/src/main/java/hudson/PluginStrategy.java
 core/src/main/java/hudson/PluginWrapper.java
 test/src/test/java/hudson/PluginManagerTest.java
http://jenkins-ci.org/commit/jenkins/7f856b56082b0b164c94bafc108408dfc4b4b138
Log:
  [FIXED JENKINS-12753] Added PluginStrategy.getShortName to avoid actually trying to unpack a plugin before throwing RestartRequiredException.

    Code changed in jenkins
    User: Jesse Glick
    Path:
    changelog.html
    core/src/main/java/hudson/ClassicPluginStrategy.java
    core/src/main/java/hudson/PluginManager.java
    core/src/main/java/hudson/PluginStrategy.java
    core/src/main/java/hudson/PluginWrapper.java
    test/src/test/java/hudson/PluginManagerTest.java
    http://jenkins-ci.org/commit/jenkins/7f856b56082b0b164c94bafc108408dfc4b4b138
    Log:
    [FIXED JENKINS-12753] Added PluginStrategy.getShortName to avoid actually trying to unpack a plugin before throwing RestartRequiredException.

  8. jenkins-infra-bot
    closed this as completedon May 20, 2014

  9. jenkins-infra-bot commented on May 20, 2014

    @jenkins-infra-bot
    jenkins-infra-bot
    on May 20, 2014
    Author
    danielbeck:

    Jesse: Since JENKINS-20732

  10. jenkins-infra-bot commented on May 20, 2014

    @jenkins-infra-bot
    jenkins-infra-bot
    on May 20, 2014
    Author
    dogfood:
    • Original comment link
    • Raw content of original comment: 
      Integrated in  jenkins_main_trunk #3390
     [FIXED JENKINS-12753] Added PluginStrategy.getShortName to avoid actually trying to unpack a plugin before throwing RestartRequiredException. (Revision 7f856b56082b0b164c94bafc108408dfc4b4b138)
      


           Result = SUCCESS
Jesse Glick : 7f856b56082b0b164c94bafc108408dfc4b4b138
Files : 
      

        
	
      • core/src/main/java/hudson/PluginStrategy.java
        • 
	
      • test/src/test/java/hudson/PluginManagerTest.java
        • 
	
      • core/src/main/java/hudson/PluginWrapper.java
        • 
	
      • core/src/main/java/hudson/PluginManager.java
        • 
	
      • core/src/main/java/hudson/ClassicPluginStrategy.java
        • 
	
      • changelog.html
        •

    Integrated in jenkins_main_trunk #3390
    [FIXED JENKINS-12753] Added PluginStrategy.getShortName to avoid actually trying to unpack a plugin before throwing RestartRequiredException. (Revision 7f856b5)

    Result = SUCCESS
    Jesse Glick : 7f856b56082b0b164c94bafc108408dfc4b4b138
    Files :

    • core/src/main/java/hudson/PluginStrategy.java
    • test/src/test/java/hudson/PluginManagerTest.java
    • core/src/main/java/hudson/PluginWrapper.java
    • core/src/main/java/hudson/PluginManager.java
    • core/src/main/java/hudson/ClassicPluginStrategy.java
    • changelog.html

  11. jenkins-infra-bot commented on May 20, 2014

    @jenkins-infra-bot
    jenkins-infra-bot
    on May 20, 2014
    Author
    jglick:
    • Original comment link
    • Raw content of original comment: 
      Finally did manage to reproduce the corresponding bug on Windows in 1.554.1:
      


      

      May 20, 2014 3:26:21 PM INFO hudson.model.UpdateCenter$DownloadJob run
Starting the installation of Support Core Plugin on behalf of anonymous
May 20, 2014 3:26:21 PM INFO hudson.model.UpdateCenter$UpdateCenterConfiguration download
Downloading Support Core Plugin
May 20, 2014 3:26:22 PM INFO hudson.PluginManager dynamicLoad
Attempting to dynamic load C:\Documents and Settings\Jesse Glick\.jenkins\plugins\support-core.jpi
May 20, 2014 3:26:22 PM SEVERE hudson.model.UpdateCenter$DownloadJob run
Failed to install Support Core Plugin
java.io.IOException: Failed to dynamically deploy this plugin
	at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:1304)
	at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:1103)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:104)
	at java.lang.Thread.run(Unknown Source)
Caused by: java.nio.file.FileSystemException: C:\Documents and Settings\Jesse Glick\.jenkins\plugins\support-core\WEB-INF\lib\animal-sniffer-annotations-1.9.jar: The process cannot access the file because it is being used by another process.

	at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
	at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
	at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
	at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
	at sun.nio.fs.AbstractFileSystemProvider.delete(Unknown Source)
	at java.nio.file.Files.delete(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at hudson.Util.deleteFile(Util.java:238)
	at hudson.Util.deleteRecursive(Util.java:301)
	at hudson.Util.deleteContentsRecursive(Util.java:203)
	at hudson.Util.deleteRecursive(Util.java:292)
	at hudson.Util.deleteContentsRecursive(Util.java:203)
	at hudson.Util.deleteRecursive(Util.java:292)
	at hudson.Util.deleteContentsRecursive(Util.java:203)
	at hudson.Util.deleteRecursive(Util.java:292)
	at hudson.ClassicPluginStrategy.explode(ClassicPluginStrategy.java:422)
	at hudson.ClassicPluginStrategy.createPluginWrapper(ClassicPluginStrategy.java:126)
	at hudson.PluginManager.dynamicLoad(PluginManager.java:415)
	at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:1300)
	... 5 more

    Finally did manage to reproduce the corresponding bug on Windows in 1.554.1:

    May 20, 2014 3:26:21 PM INFO hudson.model.UpdateCenter$DownloadJob run
Starting the installation of Support Core Plugin on behalf of anonymous
May 20, 2014 3:26:21 PM INFO hudson.model.UpdateCenter$UpdateCenterConfiguration download
Downloading Support Core Plugin
May 20, 2014 3:26:22 PM INFO hudson.PluginManager dynamicLoad
Attempting to dynamic load C:\Documents and Settings\Jesse Glick\.jenkins\plugins\support-core.jpi
May 20, 2014 3:26:22 PM SEVERE hudson.model.UpdateCenter$DownloadJob run
Failed to install Support Core Plugin
java.io.IOException: Failed to dynamically deploy this plugin
	at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:1304)
	at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:1103)
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(Unknown Source)
	at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:104)
	at java.lang.Thread.run(Unknown Source)
Caused by: java.nio.file.FileSystemException: C:\Documents and Settings\Jesse Glick\.jenkins\plugins\support-core\WEB-INF\lib\animal-sniffer-annotations-1.9.jar: The process cannot access the file because it is being used by another process.

	at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)
	at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
	at sun.nio.fs.WindowsException.rethrowAsIOException(Unknown Source)
	at sun.nio.fs.WindowsFileSystemProvider.implDelete(Unknown Source)
	at sun.nio.fs.AbstractFileSystemProvider.delete(Unknown Source)
	at java.nio.file.Files.delete(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at hudson.Util.deleteFile(Util.java:238)
	at hudson.Util.deleteRecursive(Util.java:301)
	at hudson.Util.deleteContentsRecursive(Util.java:203)
	at hudson.Util.deleteRecursive(Util.java:292)
	at hudson.Util.deleteContentsRecursive(Util.java:203)
	at hudson.Util.deleteRecursive(Util.java:292)
	at hudson.Util.deleteContentsRecursive(Util.java:203)
	at hudson.Util.deleteRecursive(Util.java:292)
	at hudson.ClassicPluginStrategy.explode(ClassicPluginStrategy.java:422)
	at hudson.ClassicPluginStrategy.createPluginWrapper(ClassicPluginStrategy.java:126)
	at hudson.PluginManager.dynamicLoad(PluginManager.java:415)
	at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:1300)
	... 5 more
  12. jenkins-infra-bot
    added
    bugFor changelog: Minor bug. Will be listed after features
    imported-jira-issue
    component:core
    on Nov 24, 2025

  13. jenkins-infra-bot commented on Nov 24, 2025

    @jenkins-infra-bot
    jenkins-infra-bot
    on Nov 24, 2025
    Author

    [Original relates_to from Jira: JENKINS-8550]

  14. jenkins-infra-bot commented on Nov 24, 2025

    @jenkins-infra-bot
    jenkins-infra-bot
    on Nov 24, 2025
    Author

    [Original relates_to from Jira: JENKINS-20732]

  15. jenkins-infra-bot commented on Nov 24, 2025

    @jenkins-infra-bot
    jenkins-infra-bot
    on Nov 24, 2025
    Author

    [Original relates_to from Jira: JENKINS-22205]

  16. jenkins-infra-bot
    added
    priority:minor
    resolution:fixed
    on Nov 24, 2025
  17. jenkins-infra-bot
    added the
    Bug
    issue type on Nov 24, 2025
  18. jenkins-infra-bot
    removed
    bugFor changelog: Minor bug. Will be listed after features
    on Nov 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    component:coreimported-jira-issuepriority:minorresolution:fixed

    Type

    Bug

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions