[JENKINS-2111] removing a job (including multibranch/org folder branches/repos) does not remove the workspace

[jenkins-infra-bot]

Removal of a job leaves the workspace intact.

---

Originally reported by bll6969, imported from: removing a job (including multibranch/org folder branches/repos) does not remove the workspace

• assignee: jglick
• status: Resolved
• priority: Blocker
• component(s): core
• label(s): cloudbees-internal-pipeline, multibranch, pipeline, workspace
• resolution: Fixed
• resolved: 2018-11-09T18:54:44+00:00
• votes: 90
• watchers: 101
• imported: 2025-11-24

Raw content of original issue

Removal of a job leaves the workspace intact.

• environment: Platform: All, OS: All

10 attachments

• image-2018-11-08-21-26-39-503.png

  

• image-2018-11-08-21-27-53-821.png

  

• image-2018-11-08-21-29-26-758.png

  

• image-2018-11-08-21-31-08-504.png

  

• image-2018-11-09-14-25-37-159.png

  

• image-2018-11-09-14-40-45-316.png

  

• image-2018-11-09-19-45-07-763.png

  

• image-2018-11-12-13-46-17-318.png

  

• image-2018-11-12-13-47-19-049.png

  

• image-2018-11-12-13-50-15-039.png

  

[jenkins-infra-bot]

• Remote links associated with this issue:

  • branch-api PR 50
  • CloudBees Internal CD-180
  • branch-api PR 129

[jenkins-infra-bot]

olenz:

• Original comment link
• Raw content of original comment:

  This is still true. But is this really a bug?
  

This is still true. But is this really a bug?

[jenkins-infra-bot]

pjdarton:

• Original comment link
• Raw content of original comment:

  It's certainly messy.
  It means that if you've got various slaves, and folks create/rename/delete jobs, you end up with all sorts of junk data left on your slaves.
  I would have expected that any data that isn't reachable from the Jenkins UI to be removed.
  
  At present, we've got a resource leak that will ultimately kill the slave.
  (it's a bug, but I wouldn't call it a "Major" bug)
  

It's certainly messy.
It means that if you've got various slaves, and folks create/rename/delete jobs, you end up with all sorts of junk data left on your slaves.
I would have expected that any data that isn't reachable from the Jenkins UI to be removed.

At present, we've got a resource leak that will ultimately kill the slave.
(it's a bug, but I wouldn't call it a "Major" bug)

[jenkins-infra-bot]

spots107:

• Original comment link
• Raw content of original comment:

  I agree that this is a bug. When a build gets deleted, so should all the data be deleted.
  When a build gets renamed, the files or directories on the file system (with the previous name) should be renamed as well.
  At present Jenkins will use space on the file system unnecessarily. Over a longer period of time, this will cause an issue, particularly for jobs that used a fair bit of space prior to being deleted or renamed.
  

I agree that this is a bug. When a build gets deleted, so should all the data be deleted.
When a build gets renamed, the files or directories on the file system (with the previous name) should be renamed as well.
At present Jenkins will use space on the file system unnecessarily. Over a longer period of time, this will cause an issue, particularly for jobs that used a fair bit of space prior to being deleted or renamed.

[jenkins-infra-bot]

mgraciano:

• Original comment link
• Raw content of original comment:

  I have used Hudson for a long time and now Jenkins, and as far as I remember, the default behaviour always was delete the workspace when deleting jobs. IMHO, it is a big regression.
  

I have used Hudson for a long time and now Jenkins, and as far as I remember, the default behaviour always was delete the workspace when deleting jobs. IMHO, it is a big regression.

[jenkins-infra-bot]

jollychang:

• Original comment link
• Raw content of original comment:

  you can use script and https://wiki.jenkins-ci.org/display/JENKINS/NodeLabel+Parameter+Plugin to do the same thing,such as my script https://gist.github.com/jollychang/5260975
  

you can use script and https://wiki.jenkins-ci.org/display/JENKINS/NodeLabel+Parameter+Plugin to do the same thing,such as my script https://gist.github.com/jollychang/5260975

[jenkins-infra-bot]

danielbeck:

• Original comment link
• Raw content of original comment:

  I agree that this is a bug. When a build gets deleted, so should all the data be deleted.
  
  This may not be possible in the case of distributed builds (nodes may be offline). It may not be desirable in the case of custom workspaces shared with other projects or not meant to be deleted at all (it happens).
  
  I have used Hudson for a long time and now Jenkins, and as far as I remember, the default behaviour always was delete the workspace when deleting jobs. 
  
  Possibly since new installs use a different top level directory for workspaces than for jobs. It used to be jobs/foo/workspace, now it is workspace/foo, like slave FS layout always was. However, for the master node, this is configurable in global config.
  

I agree that this is a bug. When a build gets deleted, so should all the data be deleted.

This may not be possible in the case of distributed builds (nodes may be offline). It may not be desirable in the case of custom workspaces shared with other projects or not meant to be deleted at all (it happens).

I have used Hudson for a long time and now Jenkins, and as far as I remember, the default behaviour always was delete the workspace when deleting jobs.

Possibly since new installs use a different top level directory for workspaces than for jobs. It used to be jobs/foo/workspace, now it is workspace/foo, like slave FS layout always was. However, for the master node, this is configurable in global config.

[jenkins-infra-bot]

lautou:

• Original comment link
• Raw content of original comment:

  Hi
  this issue is very annoying when dealing with hundred jobs you manage dynamically. In long term, this causes a lot of space disk issues.
  I agree this not desirable when having a workspace shared with others jobs. In this case it should be the responsibility of the job author to know if the job deletion (along with its workspace) should impact others jobs.
  To avoid such case I suggest to add a check box "Don't delete the workspace if this job is deleted" in job configuration and find a way to mark a directory not eraseable.
  
  Concerning the slaves I suggest to check once a the node is reconnected, the list of orphan job workspaces and remove them automaticaly.
  Is it conceivable?
  

Hi
this issue is very annoying when dealing with hundred jobs you manage dynamically. In long term, this causes a lot of space disk issues.
I agree this not desirable when having a workspace shared with others jobs. In this case it should be the responsibility of the job author to know if the job deletion (along with its workspace) should impact others jobs.
To avoid such case I suggest to add a check box "Don't delete the workspace if this job is deleted" in job configuration and find a way to mark a directory not eraseable.

Concerning the slaves I suggest to check once a the node is reconnected, the list of orphan job workspaces and remove them automaticaly.
Is it conceivable?

[jenkins-infra-bot]

nick_volynkin:

• Original comment link
• Raw content of original comment:

  This happens when you rename a job - its workspace is not renamed and not deleted either. The job page shows the size of the old workspace, but after next build the new workspace will be created and the used space will equal to the sum of two workspaces. When the job is deleted or when you wipe out the workspace, only the new workspace gets deleted.
  
  Just found that my Jenkins (v 1.635) "stashed" 35GB this way.
  

This happens when you rename a job - its workspace is not renamed and not deleted either. The job page shows the size of the old workspace, but after next build the new workspace will be created and the used space will equal to the sum of two workspaces. When the job is deleted or when you wipe out the workspace, only the new workspace gets deleted.

Just found that my Jenkins (v 1.635) "stashed" 35GB this way.

[jenkins-infra-bot]

pjdarton:

• Original comment link
• Raw content of original comment:

  I think that it would be very nice if, when a job does not have a custom workspace directory, renaming or deleting the job would rename or delete the workspace too (on ALL slaves).
  If a job has a custom workspace directory then things get more complicated, and that's where the "Don't delete the workspace if this job is deleted" check-box would be necessary.
  So, I'd suggest that this "Don't delete the workspace if this job is deleted" flag default to true where there is a custom workspace, and default to false (i.e. "do delete it") where a job just uses the workspace that Jenkins gives it.
  
  Or one could solve this another way:
  If the main Jenkins server were to iterate over all the jobs, it could build up a Map<Slave,List<expectedWorkspace>> listing, for each slave, what workspaces that slave might be asked about.
  Each slave could then be instructed to remove all the workspaces not in that list.
  This would ensure that each slave only retains the workspaces that the Jenkins server might ask it about - workspaces belonging to deleted/renamed builds would be removed, and workspaces belonging to valid builds that were last built on a different slave (and hence if the Jenkins server needed to access the workspace, it'd ask that slave not this one) would also be removed.
  
  The Distributed Workspace Clean plugin helps, but only works on valid builds - it doesn't remove data belonging to renamed/deleted builds.
  

I think that it would be very nice if, when a job does not have a custom workspace directory, renaming or deleting the job would rename or delete the workspace too (on ALL slaves).
If a job has a custom workspace directory then things get more complicated, and that's where the "Don't delete the workspace if this job is deleted" check-box would be necessary.
So, I'd suggest that this "Don't delete the workspace if this job is deleted" flag default to true where there is a custom workspace, and default to false (i.e. "do delete it") where a job just uses the workspace that Jenkins gives it.

Or one could solve this another way:
If the main Jenkins server were to iterate over all the jobs, it could build up a Map> listing, for each slave, what workspaces that slave might be asked about.
Each slave could then be instructed to remove all the workspaces not in that list.
This would ensure that each slave only retains the workspaces that the Jenkins server might ask it about - workspaces belonging to deleted/renamed builds would be removed, and workspaces belonging to valid builds that were last built on a different slave (and hence if the Jenkins server needed to access the workspace, it'd ask that slave not this one) would also be removed.

The Distributed Workspace Clean plugin helps, but only works on valid builds - it doesn't remove data belonging to renamed/deleted builds.

[jenkins-infra-bot]

danielbeck:

• Original comment link
• Raw content of original comment:

  Or one could solve this another way:
  If the main Jenkins server were to iterate over all the jobs, it could build up a Map<Slave,List<expectedWorkspace>> listing, for each slave, what workspaces that slave might be asked about.
  Each slave could then be instructed to remove all the workspaces not in that list.
  This would ensure that each slave only retains the workspaces that the Jenkins server might ask it about - workspaces belonging to deleted/renamed builds would be removed, and workspaces belonging to valid builds that were last built on a different slave (and hence if the Jenkins server needed to access the workspace, it'd ask that slave not this one) would also be removed.
  
  I think it's possible to reference environment variables in the 'Custom Workspace' option, and if those are only defined during a build or occasionally change values, this will result in the deletion of legitimate workspaces.
  
  Jenkins is a complex system, so any magic here will break things horribly for quite a few users – see the frequent complaints about the existing WorkspceCleanupThread.
  
  What would likely be feasible: If there are workspaces, ask whether they should be deleted as well when the project gets deleted. If a slave is offline, it's your responsibility to clean up.
  

Or one could solve this another way:
If the main Jenkins server were to iterate over all the jobs, it could build up a Map> listing, for each slave, what workspaces that slave might be asked about.
Each slave could then be instructed to remove all the workspaces not in that list.
This would ensure that each slave only retains the workspaces that the Jenkins server might ask it about - workspaces belonging to deleted/renamed builds would be removed, and workspaces belonging to valid builds that were last built on a different slave (and hence if the Jenkins server needed to access the workspace, it'd ask that slave not this one) would also be removed.

I think it's possible to reference environment variables in the 'Custom Workspace' option, and if those are only defined during a build or occasionally change values, this will result in the deletion of legitimate workspaces.

Jenkins is a complex system, so any magic here will break things horribly for quite a few users – see the frequent complaints about the existing WorkspceCleanupThread.

What would likely be feasible: If there are workspaces, ask whether they should be deleted as well when the project gets deleted. If a slave is offline, it's your responsibility to clean up.

[jenkins-infra-bot]

pjdarton:

• Original comment link
• Raw content of original comment:

  Re: "feasible"
  Would it also be feasible to have all (online) slaves (attempt to) rename their workspace when a job is renamed?
  
  I think that having all (online) slaves (attempt to) delete/rename the workspace belonging to a job that is being deleted/renamed would be a good idea.
  Jenkins already has a web-ui page asking "are you sure" when you ask it to delete/rename a job, so this page could also be the place where the question is asked about whether or not Jenkins should attempt to delete/rename the matching workspaces.
  
  Note: Any code that's iterating over multiple slaves, asking them to delete/rename workspace folders, MUST NOT STOP if one slave declares a failure.  We have (far too many) Windows-based slaves, and filesystem operations on Windows aren't reliable as the OS can decide (at any point, albeit briefly) that a file is locked and hence cause a deletion/rename to fail because a "file is in use", so we've leaned the hard way that if you want something to "just work" then you need to cope with filesystem "failures" (because they're often not reporting a fatal problem).
  i.e. the loop going over all slaves should ask them to try, but shouldn't stop asking the 3rd slave just because the 2nd failed - this "tidy up" operation should be on a "best effort" basis, not a "succeed or die trying" basis.
  
  I would also suggest that any operation that's being done "on all slaves" should run in multiple threads, rather than asking each slave in turn.
  

Re: "feasible"
Would it also be feasible to have all (online) slaves (attempt to) rename their workspace when a job is renamed?

I think that having all (online) slaves (attempt to) delete/rename the workspace belonging to a job that is being deleted/renamed would be a good idea.
Jenkins already has a web-ui page asking "are you sure" when you ask it to delete/rename a job, so this page could also be the place where the question is asked about whether or not Jenkins should attempt to delete/rename the matching workspaces.

Note: Any code that's iterating over multiple slaves, asking them to delete/rename workspace folders, MUST NOT STOP if one slave declares a failure. We have (far too many) Windows-based slaves, and filesystem operations on Windows aren't reliable as the OS can decide (at any point, albeit briefly) that a file is locked and hence cause a deletion/rename to fail because a "file is in use", so we've leaned the hard way that if you want something to "just work" then you need to cope with filesystem "failures" (because they're often not reporting a fatal problem).
i.e. the loop going over all slaves should ask them to try, but shouldn't stop asking the 3rd slave just because the 2nd failed - this "tidy up" operation should be on a "best effort" basis, not a "succeed or die trying" basis.

I would also suggest that any operation that's being done "on all slaves" should run in multiple threads, rather than asking each slave in turn.

[jenkins-infra-bot]

nick_volynkin:

• Original comment link
• Raw content of original comment:

  From the user's side being asked to delete/rename workspace when deleting/renaming a job looks quite fine. The options on job deletion could be:
  
  
  	
  delete altogether
  	
  leave intact
  
  
  
  And on renaming:
  
  
  	
  rename to the job's new name
  	
  use the old workspace (the "Use custom workspace" option)
  	
  delete and create a new workspace with job's name
  	
  leave intact and create a new workspace with job's name
  
  
  
  Not every user is able to operate directly in the filesystem of a particular slave (either by ssh or directly). Maybe such users should not be able to leave a workspace intact, because otherwise they would waste storage space, which then will have to be cleaned by somebody else, like a sysadmin. (But there's a workaround with creating a new job with the old name, "catching" the workspace and wiping it.) 
  
  There is a major exclusion with workspaces used by several jobs. They should be detected and only options 2 and 4 should be available. Maybe the whole feature/bug of not-auto-renaming exists to support reused workspaces?
  
  If a node is offline, it may be useful to allow only "use custom" and "leave intact". The user will be able to explicitly rename, delete or set up the new workspace when the node is online again.
  

From the user's side being asked to delete/rename workspace when deleting/renaming a job looks quite fine. The options on job deletion could be:

1. delete altogether
2. leave intact

And on renaming:

1. rename to the job's new name
2. use the old workspace (the "Use custom workspace" option)
3. delete and create a new workspace with job's name
4. leave intact and create a new workspace with job's name

Not every user is able to operate directly in the filesystem of a particular slave (either by ssh or directly). Maybe such users should not be able to leave a workspace intact, because otherwise they would waste storage space, which then will have to be cleaned by somebody else, like a sysadmin. (But there's a workaround with creating a new job with the old name, "catching" the workspace and wiping it.)

There is a major exclusion with workspaces used by several jobs. They should be detected and only options 2 and 4 should be available. Maybe the whole feature/bug of not-auto-renaming exists to support reused workspaces?

If a node is offline, it may be useful to allow only "use custom" and "leave intact". The user will be able to explicitly rename, delete or set up the new workspace when the node is online again.

[jenkins-infra-bot]

kriegaex:

• Original comment link
• Raw content of original comment:

  For me the workdirs on the Jenkins master (no slaves) are never deleted, no matter whether I delete them via REST or from the GUI. Our Jenkins version is 1.596.2. What am I doing wrong? All workdirs are stored unter /opt/.jenkins/workspace. Is that a non-default directory?
  

For me the workdirs on the Jenkins master (no slaves) are never deleted, no matter whether I delete them via REST or from the GUI. Our Jenkins version is 1.596.2. What am I doing wrong? All workdirs are stored unter /opt/.jenkins/workspace. Is that a non-default directory?

[jenkins-infra-bot]

grevenx:

• Original comment link
• Raw content of original comment:

  With the multi-branch/pipeline setup that is getting popular now, this really is a MUST in order to handle disk usage well
  

With the multi-branch/pipeline setup that is getting popular now, this really is a MUST in order to handle disk usage well