Don't wait for Cromwell label endpoint for subworkflows

[jmthibault79]

A user reported that they were unable to view a Workflow Details page.

Digging into the cause:

• Repeated attempts at the Orchestration API call usually timed out.
• The underlying Rawls API call usually took longer than one minute - causing Orchestration to time out.
• Cromwell calls responded quickly - so the problem is in Rawls or between Rawls and Cromwell

Rawls parses the returned metadata from Cromwell in order to determine whether there are subworkflows. If so, it calls the Cromwell PATCH labels endpoint to associate each of these with the Submission. Rawls waits for all of these to return before returning itself.

The user's workflow has 388 Subworkflows. Observing our attempts in Kibana, we saw that it can take about 2 minutes for Cromwell to respond to all of these label requests, causing the above chain of problems.

IMHO this is a reasonable response time for Cromwell. We should change Rawls so that it doesn't wait for these labels in order to respond. This will result in much faster response times. There are several ways we could do this, so let's make a design doc. Be sure to consider the discussions in the original design docs for this feature:

https://docs.google.com/document/d/1v8HB9aGCLEpAi_IPs10hFJ-DRrPBX_2Gg4KhtH-Evo4/edit#heading=h.a4fxgxydqoek

https://docs.google.com/document/d/1N5MQZhFH2i5BiEeshvUR56ukdWiHf2Z_zi4gQM6l5yM/edit

[jmthibault79]

Design Doc