Task state change callback mechanism

[uniqueg]

Is your feature request related to a problem? Please describe.
Currently, tracking of task states is (or will be) implemented by parsing cwl-tes logs in conjunction with polling of the relevant TES endpoint by cwl-tes. The latter imposes a high burden on the TES instance when handling many tasks simultaneously and may further require unnecessary token refreshing.

Describe the solution you'd like
Current WES/TES specs do not provide a callback solution. An outline for such a solution could be:

• The POST /tasks request to TES should contain a field callback_url to let the TES instance know to which WES instance it should report back a state change
• A POST /task-state (or similar) endpoint should be added to the WES specs, requiring a TES task_id and the new state of the task as parameters
• A directive should be added to the TES specs, instructing TES to report task state changes back to WES (if callback_url is non-empty); if callback remains an optional TES feature, a Boolean field such as callback_supported could further be added to the TES service-info endpoint to allow WES to verify whether a given TES instance supports callback

In this way, WES should be able to handle incoming task state changes gracefully, without having to resort to polling. In order to prevent TES having to have access to a refresh token (in case a task lasts longer than the auth token remains to be valid), access to the new WES endpoint should be granted either by a valid token or by the token that was attached when the corresponding task was originally posted by WES.

Describe alternatives you've considered
Given that distributed task execution requires that WES and TES not be coupled, even for a given workflow run, and that, moreover, statelessness of WES and TES should be maintained, the number of possible alternatives is limited. As WES cannot know what happens inside TES, it either has to ask or rely on a signal sent by TES to WES.

Additional context
Polling can be retained as a fallback mechanism. Make sure to consider integrated WES-ELIXIR/TESK solution and other solutions without proTES and with other TES implementations, e.g., Funnel.

[uniqueg]

See also:

• Proposal: Support for callback_url  ga4gh/task-execution-schemas#121
• Proposal: Support for Workflow Run status update ga4gh/workflow-execution-service-schemas#133
• Planning for FASP Hackathon ga4gh/cloud-interop-testing#98 (comment)