Cancellation issue in V2 workflows running Input generation.

[sambles]

Issue Description

Had a report of an analyses cancellation call failing when running generate inputs.

The cancellation call returns HTTP 200, and sub-tasks are marked as cancelled, but tasks appear to keep running in the background.

Cancellation issued - 13:12:25

INFO 2024-07-23 13:12:25,828 middleware 66 140474528149952 ^[[36mPOST /api/v2/analyses/1813/cancel_generate_inputs/ - 200^[[0m

Worker monitor - 13:29:19

[2024-07-23 13:29:19,709: INFO/ForkPoolWorker-9] set_task_status_v2[1efe6c6d-f504-44ea-917f-8bd72a92a0d9]: Task Status Update: analysis_pk: 1813, status: INPUTS_GENERATION_STARTED

[sambles]

This Issue also feels linked to #1057

The attached parent task id used to identify the celery canvas workflow is not correct. analysis.generate_inputs_task_id

1. run generate_inputs is posted.
2. No workers are available to accept initial task, all sub-tasks don't yet have task id's as they are all PENDING.
3. cancel_generate_inputs is posted.
4. the sub-task cancellation does nothing, since there are no sub-task id's to revoke yet. The parent task revoke call also fails (because the main task id is not correct)
5. Analysis is marked as cancelled in DB, while on the backend nothing has been cancelled.

--- Worker becomes available ---
6. initial task (still on queue) is picked up and the sub-tasks start to execute.
7. follow-up post to cancel_generate_inputs do noting, becuase the analysis is tagged as already cancelled.

[sambles]

Part of the issue is described here: celery/celery#8888

Issue: The challenge arises when the worker associated with the task's queue is offline. In such cases, the broadcast signal sent by .revoke() is lost. Consequently, even if tasks have been revoked, upon restarting the workers, these tasks are still executed.