Server Tasks not canceling when a the task is canceled #7903
Description
Team
- I've assigned a team label to this issue
Severity
Impacting several clients
Version
Versions from 2023.1.744 are impacted by this issue
Latest Version
This issue impacts the latest version of Octopus Server and the current version of Octopus Cloud
What happened?
When a user cancels a server task, the task status is changed to canceling and shows as cancelling in the UI but the server task continues to run.
The server task log will show a status of Canceled but it will have run to completion.
An issue has been identified where a race condition in Octopus Server can result in the Thread running the Server Task (Deployment / Runbook Run) not being canceled correctly when the server task status changes.
This issue is possible in either a multi-node cluster or single-node configuration.
An Octopus Server cluster with more than one node has a higher chance of causing the issue when the node receiving the request to cancel the server task is not the node running the server task.
Reproduction
- Start a server task (runbook run/deployment/health check)
- Cancel the server task through the UI or API
- Intermittently the server task will change its status to Cancelling but continue to run to completion
Error and Stacktrace
n/a
More Information
n/a
Workaround
The only workaround to the issue is to restart the Octopus Server instance running the Server Task to ensure it is stopped.
In a multi-node configuration canceling the server task on the node running the server task by directly accessing the nodes UI or API rather than via a load balancer may reduce the possibility of this issue occurring.