Optimize Memory Consumption when Evaluating Large Workflows

[jmchilton]

I think we are close to solving #3883 with #5013 which should buy us new flexibility in terms of dealing with memory issues while scheduling workflows and recovering from memory problems. I want to continue to hit on this though - I think it is probably a limiting issue in terms of collection scaling at the moment. I have some ideas about how to limit the memory but I'm not sure how to test them effectively exactly.

• Make separate schedulers for workflow scheduling a best practice for scaling.
  • Prepare documentation on separating out workflow schedulers.
  • Write extensive tests to verify it works and close out Workflow handlers are not working as expected #4976.
  • Add to docker-galaxy-stable out of the box.

With the above in place - admins of instances bumping into this issue could separate things out and isolate if the memory problems are in the scheduling processes or the job handler processes. When I started writing this issue I had some ideas for optimizing memory usage in job handlers but I did a detailed review and it turns out they have already been implemented thankfully. So I think if there are memory problems they will pop up in workflow scheduling process.

Here are some ideas:

• I thought for sure the workflow scheduler would expunge objects between workflow scheduling iterations - it creates lists of IDs to allow doing this very effectively and get isolation between each invocation - but it doesn't after I did a search for that today. I think this means workflow scheduling has a serious memory leak - and so I will work on fixing that today. (These other interventions may not be needed at all in light of this).
• Track forward connections and do not keep unneeded step outputs in memory in the WorkflowProgress object. Do not ever even load unneeded outputs on subsequent scheduling iterations (this is a lot easier after Improved Collection and Workflow State with Applications #5013 since we are persisting everything very clearly).
• I don't think the workflow scheduler right now would ever need a lot of HDA attributes that are probably loaded into memory - e.g. metadata. The job handler would need those to turn things into jobs but not the scheduler I think? Verify this and do not load unneeded attributes into memory ever in that process.