[JENKINS-73835] Builds can be deleted while they are still running

[jenkins-infra-bot]

Builds can be deleted via the /doDelete REST API, and the delete-builds CLI command while they are still running. The “Delete build” link in the sidepanel of builds is hidden unless !it.building and !it.keepLog, but this is a UI-only restriction. Run.delete has the same issue, but in practice all of the callers that I found in plugins already check whether the build is done building (most calls are in LogRotator implementations where LogRotator.shouldKeepRun handles the issue), but there are some subtle race conditions regarding checking Run.isBuilding instead of Run.isLogUpdated that can lead to LogRotator deleting Pipeline builds while they are in the middle of finishing.

Deleting a build while it is running can cause various kinds of anomalous behavior as the build continues to run despite its configuration directory being deleted, and it is likely that the build will return after a Jenkins restart since build.xml may be rewritten as the build continues running.

---

Originally reported by dnusbaum, imported from: Builds can be deleted while they are still running

• assignee: dnusbaum
• status: Closed
• priority: Minor
• component(s): core
• label(s): 2.479.1-fixed
• resolution: Fixed
• resolved: 2024-10-14T18:50:51+00:00
• votes: 0
• watchers: 2
• imported: 2025-11-24

Raw content of original issue

Builds can be deleted via the /doDelete REST API, and the delete-builds CLI command while they are still running. The “Delete build” link in the sidepanel of builds is hidden unless !it.building and !it.keepLog, but this is a UI-only restriction. Run.delete has the same issue, but in practice all of the callers that I found in plugins already check whether the build is done building (most calls are in LogRotator implementations where LogRotator.shouldKeepRun handles the issue), but there are some subtle race conditions regarding checking Run.isBuilding instead of Run.isLogUpdated that can lead to LogRotator deleting Pipeline builds while they are in the middle of finishing.

Deleting a build while it is running can cause various kinds of anomalous behavior as the build continues to run despite its configuration directory being deleted, and it is likely that the build will return after a Jenkins restart since build.xml may be rewritten as the build continues running.

[jenkins-infra-bot]

• Remote links associated with this issue:

  • jenkinsci/jenkins#​9810
  • jenkinsci/workflow-job-plugin#​470
  • jenkinsci/jenkins#​9876
  • jenkinsci/lockable-resources-plugin#​716
  • jenkinsci/artifact-manager-s3-plugin#​551

[jenkins-infra-bot]

dnusbaum:

• Original comment link
• Raw content of original comment:

  It is awkward to handle this in Run.delete because of how and where LogRotator is called today. When LogRotator runs for a Freestyle build, this call is guaranteed to observe !Run.isBuilding() and Run.isLogUpdated(). For Pipelines this timer task will always observe both !Run.isBuilding and !Run.isLogUpdated. Further, for Pipelines, using !Run.isBuilding() to detect when a build has completed is inaccurate. You must check Run.isLogUpdated() instead to avoid doing things while Pipeline cleanup is still ongoing.
  
  This means we can't make Run.delete just check Run.isBuilding() || Run.isLogUpdated() or else log rotation may throw spurious exceptions for Freestyle builds, and we ideally don't want to only check Run.isBuilding() to avoid race conditions when attempting to delete Pipelines while they are being cleaned up leading to leftover directories with a single build.xml file due to this call running after LogRotator.
  
  So we either need to slightly change when Freestyle jobs trigger log rotation so that it happens while !Run.isLogUpdated, adjust WorkflowRun.isBuilding to match WorkflowRun.isLogUpdated, or fix the issue only for the API and CLI commands directly and ignore the existing race condition with Pipelines and log rotation.
  

It is awkward to handle this in Run.delete because of how and where LogRotator is called today. When LogRotator runs for a Freestyle build, this call is guaranteed to observe !Run.isBuilding() and Run.isLogUpdated(). For Pipelines this timer task will always observe both !Run.isBuilding and !Run.isLogUpdated. Further, for Pipelines, using !Run.isBuilding() to detect when a build has completed is inaccurate. You must check Run.isLogUpdated() instead to avoid doing things while Pipeline cleanup is still ongoing.

This means we can't make Run.delete just check Run.isBuilding() || Run.isLogUpdated() or else log rotation may throw spurious exceptions for Freestyle builds, and we ideally don't want to only check Run.isBuilding() to avoid race conditions when attempting to delete Pipelines while they are being cleaned up leading to leftover directories with a single build.xml file due to this call running after LogRotator.

So we either need to slightly change when Freestyle jobs trigger log rotation so that it happens while !Run.isLogUpdated, adjust WorkflowRun.isBuilding to match WorkflowRun.isLogUpdated, or fix the issue only for the API and CLI commands directly and ignore the existing race condition with Pipelines and log rotation.

[jenkins-infra-bot]

dnusbaum:

• Original comment link
• Raw content of original comment:

  > So we either need to slightly change when Freestyle jobs trigger log rotation so that it happens while !Run.isLogUpdated
  
  Turns out that GlobalBuildDiscarderListener has redundantly been calling Job.logRotate from Run.onFinalized for a while now, so getting rid of the problematic call in Run.run in favor of GlobalBuildDiscarderListener seems like the best option.
  

> So we either need to slightly change when Freestyle jobs trigger log rotation so that it happens while !Run.isLogUpdated

Turns out that GlobalBuildDiscarderListener has redundantly been calling Job.logRotate from Run.onFinalized for a while now, so getting rid of the problematic call in Run.run in favor of GlobalBuildDiscarderListener seems like the best option.