workflow_run_manager: workflow stop race condition

[diegodelemos]

• Happening when user stops a workflow before its first job is created.
  RWC stops only the existing jobs. However, because there is a grace
  period for stopping pods, RJC sidecar still runs, submitting a new
  job, and reporting its status, causing the workflow to “revive”
  (closes cli: stop command misbehaviour reana-client#395).

[diegodelemos]

• Quick solution: Force delete with grace period 0
  • Didn't work, still is not 0 seconds to delete -> ❌
• Better solution: Ignore status updates for workflows stopped in DB
  • stop_workflow performs an atomic operation, therefore, if the stop of a workflow and its alive jobs was done, the workflow has successfully been marked as stopped in the database.
  • Use that field to ignore, in job-status-consumer, status updates for stopped workflows (something similar is being done with GitLab, see here)
• Best solution: Create the state machine for workflow status manipulation and embed logic there, see previous musings.

[diegodelemos]

I haven't fully addressed the state machine but I've recorded the knowledge summarised in reanahub/reana-client#192 (comment) (referenced in #149) inside a function in a separate module prepared for a possible future refactor for status transition management.

[mvidalgarcia]

Tested successfully on my machine 👍
Something to take into account for future improvements, several trials to switch to a forbidden status are produced. See the job-status-consumer logs:

2020-04-10 08:55:57,554 | kombu.mixins | MainThread | INFO | Connected to amqp://test:**@reana-message-broker:5672//
2020-04-10 08:57:44,143 | root | MainThread | ERROR | Cannot transition workflow d4e80dbd-80b4-4725-a415-60708a8aa896 from status WorkflowStatus.stopped to WorkflowStatus.running.
2020-04-10 08:57:44,291 | root | MainThread | ERROR | Cannot transition workflow d4e80dbd-80b4-4725-a415-60708a8aa896 from status WorkflowStatus.stopped to WorkflowStatus.running.
2020-04-10 08:57:50,325 | root | MainThread | ERROR | Cannot transition workflow d4e80dbd-80b4-4725-a415-60708a8aa896 from status WorkflowStatus.stopped to WorkflowStatus.running.
2020-04-10 08:57:50,431 | root | MainThread | ERROR | Cannot transition workflow d4e80dbd-80b4-4725-a415-60708a8aa896 from status WorkflowStatus.stopped to WorkflowStatus.running.
2020-04-10 08:57:59,471 | root | MainThread | ERROR | Cannot transition workflow d4e80dbd-80b4-4725-a415-60708a8aa896 from status WorkflowStatus.stopped to WorkflowStatus.finished.