Investigate Reducing Choose Worker Calls

[rajadain]

One of the main sources of latency is the choose_worker() calls in many of the Celery jobs. The solution addresses the following issues:

• Tasks from a Blue stack should not be picked up by the Green stack in production
• Geoprocessing start and finish must execute on the same worker
• When choosing a worker for geoprocessing, the chosen worker should be alive (i.e. responding to pings)

The current implementation of choose_worker is either too heavy, or too frequently used, or both.

Investigate the source of the latency described in #1894, and develop a strategy to reduce it.

[mmcfarland]

This may be moot if we go down the path of implementing the Collection API, as it will remove SJS and the need to colocate the geoprocessing start/finish jobs. We could then use build in Celery routing techniques to get around the Blue/Green issue.