Metadata store: Plugin - Argo ( Parent #50 )

[JJFinnerty]

Parent #50
When a Argo Model completes, Their is the possiblty that one of the output files is the metadata.json file. This file is the metadata dictionary for this task. The JSON needs to be incorprated into the task update status response from the TaskExecutor for the WorkFlow manger to consume.

The Model Developers will be informed that to return sucg data this file will need to be in the following format:-

{
    "key":"value",
    "key2": "value2",
}

Acceptance Criteria

• GIVEN that a task has created a metadata.json file WHEN this JSON is valid THEN when the task update status is been instantiate the metadata attribute is populated.
• GIVEN that a task has not created the metadata.json file THEN when the task update status is been instantiate the metadata attribute is left blank.
• GIVEN that a task has created a metadata.json file WHEN this JSON is invalid THEN when the task update status is been instantiate the status should be returned as Error.

[mocsharp]

What defines invalid for the JSON file? Invalid format? Corrupted file? Or something else?

Given that input metadata are suggested to be provided as Argo input parameters, do we want to take output metadata from output parameters instead of a JSON file for Argo specifically?

For output parameters, the Argo plug-in can retrieve the metadata when it receives the md.tasks.callback event.

For JSON file, I see 2 options:

1. Have the TaskPluginBase provide a method to read the metadata.json from the output artifact store.
2. Have the end hook in the Argo workflow include the metadata in the md.tasks.callback event.

For the 2nd option, we may want to first look into optimizing it by either using:

1. HTTP callback
2. Reduce to a single step from 2
3. Use an Argo plugin

[mocsharp]

@jackschofield23 can we close this as complete?

[jackschofield23]

Closed