Multi-threaded job scheduling in workflows.

[jmchilton]

Somewhere around 16.01 - I introduced the concept of tool bursting to allow multiple threads to schedule individual jobs from a collection of jobs during tool execution. This seemed to speed up tool submission in some cases - but could be much more powerful when applied to workflows - since each workflow thread could potentially have dedicated scheduling threads - thereby reducing the overhead of actually spinning up and tearing down these threads.

I think the speed up in scheduling large collection workflows would be fairly dramatic as you throw more threads at the problem.

• Fix and implement test cases for Tool burst submission fails on 16.01 release #2259.
• Refactor workflow scheduling manager to allow configuration of specific job creation threads (building on Allow explicit configuration of handlers for workflow scheduling. #3844) - use these threads when bursting.

[mvdbeek]

We've added and removed this in the meantime ... 😆 . I think we'll want to break down expensive steps and schedule them as celery tasks going forward.