Remove custom celery routing and exchange

[mmcfarland]

Overview

Due to the implementation of async SJS requests, we established a mechanism for directly establishing routes that would stay on a single worker. This led to complex code and also high latency when we pinged each worker to determine if they were available. With recent changes removing SJS in favor of a synchronous geoprocessing call, we can now cull these custom routes and exchanges, both simplifying the code and increasing the speed of submitting a new job.

Connects #2117

Fixes #1903
Fixes #1894
Fixes #1535

Notes

This reduces the HTTP response time for all celery submit endpoints considerably. Locally, they return in ~250ms, as opposed to seconds on staging/production.

Testing Instructions

First round of testing is to ensure that all geoprocessing tasks work (except Mapshed which is still being refactored - however, the mapshed jobs should still be submitted correctly, the streams will fail).

• Notice that all async submissions (eg, /land, /soil, /tr55, /rwd, etc) respond in milliseconds and not seconds

Second round of testing is to make sure that the work is still adequately shared between workers and that stacks of another color don't process jobs submitted for another stack color.

Follow these instructions sequentially, as the steps build on each other.

Test multiple workers

1. ./scripts/debugcelery.sh to launch a worker thread on the Black stack
2. Open an additional terminal and launch another worker on the Black stack:

vagrant ssh worker -c "cd /opt/app/ && envdir /etc/mmw.d/env celery -A 'mmw.celery:app' worker --autoreload -l debug -n debug2@%n"

Notice the default key is task.Black in the output

--- ***** ----- [queues]
 -------------- .> Black            exchange=celery(direct) key=task.Black

1. Continue to use the app to submit analyze and tr55 jobs, they should be spread to both workers.

Test multiple stack colors

1. Open a third terminal, and ssh onto the worker: vagrant ssh worker
2. Edit the env file for stack color, change Black -> Blue
sudo vim /etc/mmw.d/env/MMW_STACK_COLOR
3. In /opt/app, launch a third celery worker, this one will start as if on a Blue stack.

envdir /etc/mmw.d/env celery -A 'mmw.celery:app' worker --autoreload -l debug -n debug3@%n

Notice the default key is task.Blue in the output

--- ***** ----- [queues]
 -------------- .> Blue             exchange=celery(direct) key=task.Blue

4. Continue to submit jobs, only the 2 Black Stack workers should be processing them, blue should not receive any.

Test switching stack task submission

1. Open and edit settings_base.py and set STACK_COLOR = 'Blue'
2. Submit jobs and see that only the Blue worker processes them.
3. Revert your changes to settings_base.py and /etc/mmw.d/env/MMW_STACK_COLOR

[rajadain]

Taking a look now.

[rajadain]

+1!! This works exceedingly well, leading to a more noticeable speedup to the end user than the GeoTrellis upgrade. Also very glad for the simpler code and easier routing. This is a solid win. 🎈

Great job on the testing instructions too. Tested on a single worker, multiple workers, multiple workers of different colors, and changing the API's preferred stack color on the fly. Messages were routed quickly and correctly.

[mmcfarland]

Great - thanks for that feedback.