Workflow can be remove from the system in RUNNING status

[mhorbatenko]

Describe the bug
Workflow with RUNNING status can be removed from the system

Details
Conductor version: 3.13.8
Persistence implementation: MySQL
Queue implementation: MySQL
Lock: Redis
Workflow definition:

{
  "createTime": 1703154343929,
  "accessPolicy": {},
  "name": "mhTestWf",
  "description": "tltr",
  "version": 1,
  "tasks": [
    {
      "name": "dummy",
      "taskReferenceName": "dummy",
      "inputParameters": {},
      "type": "HUMAN",
      "startDelay": 0,
      "optional": false,
      "asyncComplete": false
    }
  ],
  "inputParameters": [],
  "outputParameters": {},
  "schemaVersion": 2,
  "restartable": true,
  "workflowStatusListenerEnabled": false,
  "ownerEmail": "noreply@email.com",
  "timeoutPolicy": "ALERT_ONLY",
  "timeoutSeconds": 0,
  "variables": {},
  "inputTemplate": {}
}

To Reproduce
Steps to reproduce the behaviour:

1. Create Workflow Definition
2. Run workflow from this definition

curl -X 'POST' \
  'http://conductor-server:8080/api/workflow/mhTestWf?priority=0' \
  -H 'accept: text/plain' \
  -H 'Content-Type: application/json' \
  -d '{}'

3. Check workflow status:
   3.1 in conductor UI

3.2 in db workflow table

MariaDB [conductor_db]> select * from workflow where workflow_id = 'a38e1ebc-a6ec-4a48-8b26-0864e278242d'\G
*************************** 1. row ***************************
          created_on: 2023-12-21 10:29:14
         modified_on: 2023-12-21 10:29:14
         workflow_id: a38e1ebc-a6ec-4a48-8b26-0864e278242d
      correlation_id: NULL
           json_data: {"status":"RUNNING","endTime":0,"workflowId":"a38e1ebc-a6ec-4a48-8b26-0864e278242d","tasks":[],"workflowDefinition":{"createTime":1703154343929,"accessPolicy":{},"name":"mhTestWf","description":"tltr","version":1,"tasks":[{"name":"dummy","taskReferenceName":"dummy","inputParameters":{},"type":"HUMAN","startDelay":0,"optional":false,"asyncComplete":false}],"inputParameters":[],"outputParameters":{},"schemaVersion":2,"restartable":true,"workflowStatusListenerEnabled":false,"ownerEmail":"noreply@email.com","timeoutPolicy":"ALERT_ONLY","timeoutSeconds":0,"variables":{},"inputTemplate":{}},"priority":0,"variables":{},"lastRetriedTime":0,"ownerApp":"","createTime":1703154554647,"workflowName":"mhTestWf","workflowVersion":1,"input":{},"output":{}}
    parentWorkflowId: NULL
parentWorkflowTaskId: NULL
1 row in set (0.000 sec)

3.3 in db task table:

MariaDB [conductor_db]> select * from task where task_id='dde40ce9-3134-45bf-bf7d-90e060a1f5b9' \G
*************************** 1. row ***************************
   created_on: 2023-12-21 10:29:14
  modified_on: 2023-12-21 10:29:14
      task_id: dde40ce9-3134-45bf-bf7d-90e060a1f5b9
    json_data: {"taskType":"HUMAN","status":"IN_PROGRESS","referenceTaskName":"dummy","retryCount":0,"seq":1,"pollCount":0,"taskDefName":"dummy","scheduledTime":1703154554676,"startTime":1703154554675,"endTime":0,"updateTime":1703154554682,"startDelayInSeconds":0,"retried":false,"executed":false,"callbackFromWorker":true,"responseTimeoutSeconds":0,"workflowInstanceId":"a38e1ebc-a6ec-4a48-8b26-0864e278242d","workflowType":"mhTestWf","taskId":"dde40ce9-3134-45bf-bf7d-90e060a1f5b9","callbackAfterSeconds":0,"workflowTask":{"name":"dummy","taskReferenceName":"dummy","inputParameters":{},"type":"HUMAN","startDelay":0,"optional":false,"asyncComplete":false},"rateLimitPerFrequency":0,"rateLimitFrequencyInSeconds":0,"workflowPriority":0,"iteration":0,"waitTimeout":0,"subworkflowChanged":false,"taskDefinition":null,"loopOverTask":false,"queueWaitTime":-1,"inputData":{},"outputData":{}}
     taskType: HUMAN
       status: IN_PROGRESS
subWorkflowId: NULL
1 row in set (0.000 sec)

As you can see, there is a workflow with status running, execution of the workflow is in db, a task also exists in db.

4. Use REST API to remove workflow from the system

curl -X 'DELETE' \
  'http://up-be-conductor-server:8080/api/workflow/a38e1ebc-a6ec-4a48-8b26-0864e278242d/remove?archiveWorkflow=true' \
  -H 'accept: */*'

Response body:

{
  "status": 400,
  "message": "Cannot archive workflow: a38e1ebc-a6ec-4a48-8b26-0864e278242d with status: RUNNING",
  "instance": "conductor-server",
  "retryable": false
}

According to the response from server, an operation is not permitted, since the workflow is in running status, and it's good, but actually all related data from database is deleted.

5. checkout database

workflow table:

MariaDB [conductor_db]> select * from workflow where workflow_id = 'a38e1ebc-a6ec-4a48-8b26-0864e278242d'\G
Empty set (0.000 sec)

task table:

MariaDB [conductor_db]> select * from task where task_id='dde40ce9-3134-45bf-bf7d-90e060a1f5b9' \G
Empty set (0.000 sec)

Expected behavior
Conductor server shouldn`t delete workflow and all related tasks from storage while workflow in running status.

Thx.