Metadata Store: WorkFlow Manger ( Parent #50 )

[JJFinnerty]

Parent Task #50

To support the Workflow manager's ability to have conditional workflows we need the tasks to be able pass meta data about themselfs back the workflow manger after been completed.

The Task update status event to the workflow manager will contain a metadata attribute containing all the metadata dictionary for that task if said task generats metadata.

Part 1

This task meta data will be required to be stored in a JSON block in such a way as to allow simple access of the data for use in further workflow conditonal statments, new tasks, and stored in MongoDb.

Structure of the JSON object for the storage of the task metadata:-

{
    "workflowexcutionid":"{workflowexcutionid}"
    "tasks":[
        {
            "taskname": "{taskname}",
            "taskexcutionid": "{taskexcutionid}",
            "datadictionary":{
                "key":"value",
                "key": "value",
                ...
            }
        },
        {
            "taskname": "{taskname}",
            "taskexcutionid": "{taskexcutionid}",
            "datadictionary":{
                "key":"value",
                "key": "value",
                ...
            }
        },
        ...
    ]
}

Part 2

The tasks metadata needs to be accessible for the Conditional system. This will tie in to the Conditial system's string replacement ability when parsing the user conditional Clause(S).

1. When then Contional Parser encounters a string in this format {{context.executions.task['TaskID'].'Key'}}
2. This will trigger a search of task metadata for the current workflow execution. Looking for a match for the 'TaskID'
3. If no match can be found then a null is returned.
4. if a match if found for the 'TaskID' then it will search the Tasks Meta data dictionary for the requested 'Key'
5. If no Key match is found then return a null.
6. If a key match if found then return the value from the Selected Tasks Meta data dictionary.

Part 3

When you lanuch a new task the task can have named paramaters ( Input paramaters ) which are extracted for the current meta data store.

Acceptance Criteria

• GIVEN a valid task update status event WHEN the metadata is availibe Then the workflow manager will update Stored metadata JSON object with the new metadata.
• GIVEN a conditional statement WHEN the statement is Parsed and the tasks metadata key is requested Then the metadata value for the key is resolved.
• GIVEN a conditional statement when the statement is Parsed and the tasks metadata key is requested but doesn't exeist Then the returned Value is Null.

Tests

• Publish Task Update Event with status Success adds metadata to the Workflow Instance
• Publish Task Update Event with status Accept adds metadata to the Workflow Instance
• Publish Task Update Event with status Failed does not add metadata to the Workflow Instance
• Publish Task Update Event with status Success with metadata key which is used in conditional statement. Task dispatch message is published
• Publish Task Update Event with status Success with metadata key missing which is used in conditional statement. Task dispatch message is not published

[joshliberty]

In addition to the above, we need to support sending meta data of one task to subsequent tasks.
Because tasks are developed independently and may use different naming conventions for the same underlying data, we need to allow the workflow author to control which meta data is sent to a task, and where it comes from.

To do that, we need to introduce the concept of input parameters. Each task within the clinical workflow will optionally define a list of parameters to be provided to the task plug-in upon execution of the task. The value of each parameter can either be hardcoded in the workflow, or it can be a variable strings referencing the meta data that was generated by previous task.

Examples
In the below example, two input parameters are provided. One is a hardcoded value and can be used by workflow authors to alter the behaviour of an application, and the other passes a metadata variable that was generated by previous task to this task.

tasks: [
  {
    …
    "input_parameters": {
      "parameter_1": "some hardcoded value",
      "parameter_2": "{{ context.executions.TASK_ID.result.METADATA_VARIABLE }}"
    },
    ...
  }
]

[mocsharp]

@joshliberty Are the input metadata provided to the tasks as a JSON file?

[joshliberty]

I think as parameters: https://argoproj.github.io/argo-workflows/workflow-inputs/#parameter-inputs
This should be a lot easier than artefacts, considering that parameters are designed to be provided at run time

[mocsharp]

Does it apply only if they are defined in the Argo workflow? Or, do we always inject parameters to the entrypoint template?

I would assume they need to be predefined in the Kind:WorkflowTemplate. Otherwise, that means the app doesn't know their existence.

[joshliberty]

We should only send them if they're defined in the Argo workflow - otherwise there's little point in providing them.