Skip to content
This repository has been archived by the owner on Dec 7, 2023. It is now read-only.

Support cancelling running tasks #18

Open
JeanMertz opened this issue Jun 15, 2019 · 2 comments
Open

Support cancelling running tasks #18

JeanMertz opened this issue Jun 15, 2019 · 2 comments
Milestone

Comments

@JeanMertz
Copy link
Contributor

JeanMertz commented Jun 15, 2019

When starting a long-running task, and then closing the pipeline details view in the web client, the task will continue to run on the server, and the client will continue polling for results in the background.

While some kind of "background tasks with notifications" system might be nice to have in the future, the web client currently isn't build to support this use-case, and the above will result in weird errors.

For now, three things need to happen:

  1. when a task is started, and the pipeline details view is closed, the web client should stop polling for the results of this task Support cancelling running tasks #18 (comment).
  2. Additionally, when the pipeline details view is closed, a query should be sent to the server, letting it know that the client no longer cares about the results.
  3. On the server, all running tasks should include a cancellation context, allowing tasks to be cancelled mid-execution.

Steps 2 and 3 are more involved, and less of a priority for immutable tasks, but when running a task that performs some destructive action, you might want to be able to quickly cancel a task when you accidentally started one with the wrong variables.

For 2 and 3, the most straightforward solution would be to:

  • When a cancellation request comes in, set the Task to status Cancelling.
  • Before running each task step, check if the task status was changed to cancelling.
  • If it was, stop execution of any steps still waiting to run, and update both the pending task steps and the task itself to Cancelled (tasks and task steps have their own statuses in the database).
  • By fetching task details, the client can precisely show which steps did run, and which got cancelled, so the user knows if their accidental run caused any harm.

For supporting the "cancel accidental run" case, some additional changes need to happen:

  1. When starting a task, the Run button needs to change to Cancel.
  2. When clicking that button, a cancellation request needs to happen.
  3. The response should be a Task object, after it was cancelled, so the client can show which task steps ran, and which got cancelled.

In the case of step 3, we might actually want to make the cancellation asynchronous, since a long-running step still in progress might block the response for a significant time. So instead, we should immediately return an acknowledgement that the task will be cancelled, and let the client poll until cancellation happened.

@JeanMertz JeanMertz added the bug Something isn't working label Jun 15, 2019
@JeanMertz
Copy link
Contributor Author

JeanMertz commented Jun 22, 2019

To make this a bit more clear to the person running a task, I think we should also do the following:

  • Remove the feature to hide a pipeline details view by clicking outside the modal window.
  • Rename the white Cancel button in the pipeline view to Back.
  • The only way to hide a the pipeline view is by clicking the white Back button.
  • If a pipeline is running, the white Back button turns to a red Cancel button, to signal it will actually stop a running pipeline.
  • After cancelling a pipeline, there will be some UI element showing "cancelling...".
  • After cancellation succeeded, show which steps did run, and which didn't due to cancellation.
  • Change the red Cancel button back to a white Back, allowing you to exit the details pipeline view.
  • Optionally, after a period of time, if the cancellation did not yet succeed (or an error is returned), show a message mentioning something went wrong, add the task ID as output, and tell the person running the pipeline to ask for support from someone technical.

JeanMertz added a commit that referenced this issue Jul 8, 2019
The goal of this rewrite is to make the code more composable to make it
easier to add new functionality in the future.

Here's a list of the most relevant changes:

* Use a declarative component-based UI with a virtual DOM to separate
  rendering logic from the business logic.

* Add a router to support deep-linking to specific tasks.

* Reduce Wasm binary from ~2 MB to ~450 kB (uncompressed).

* Add local session caching to reduce the number of network requests.

* Remove atomic-based CSS classes in favor of semantic classes.

* Rename all occurrences of "task" with "job" and "pipeline" with
  "task" (this change will be applied to the other crates as well).

* Minor visual changes.

This commit (partially) resolves the following issues:

* #14
* #15
* #18
* #27
@JeanMertz JeanMertz removed the bug Something isn't working label Jul 8, 2019
@JeanMertz
Copy link
Contributor Author

JeanMertz commented Jul 8, 2019

I've changed the above list into a checklist, and removed the bug label, as this is now no longer a bug since 2d9bc81.

You can no longer exit a task while its running (unless you reload your browser or hit the browser's back button, but that won't break anything).

There isn't a way to cancel a running task yet, but the infrastructure to support this is there, and this is now a regular feature request, instead of a bug.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant