-
Notifications
You must be signed in to change notification settings - Fork 31
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Collections API: Rework Celery task submission #2117
Comments
rajadain
added a commit
that referenced
this issue
Aug 18, 2017
We likely do not need to use `choose_worker` anymore, since each request is independent and can be run on any worker (in the right colored stack). However, this probably needs some more thought, and thus will be addressed in the separate issue #2117.
rajadain
added a commit
that referenced
this issue
Aug 22, 2017
We likely do not need to use `choose_worker` anymore, since each request is independent and can be run on any worker (in the right colored stack). However, this probably needs some more thought, and thus will be addressed in the separate issue #2117.
I think that we can remove the custom CELERY_DEFAULT_QUEUE = STACK_COLOR
CELERY_DEFAULT_ROUTING_KEY = "task.%s" % STACK_COLOR Testing that things are working properly may require adding another |
mmcfarland
pushed a commit
that referenced
this issue
Aug 31, 2017
We likely do not need to use `choose_worker` anymore, since each request is independent and can be run on any worker (in the right colored stack). However, this probably needs some more thought, and thus will be addressed in the separate issue #2117.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We rely heavily on querying Celery for a list of valid workers, one of which we use for a
routing_key
to keep jobs executing on the same worker, since the async SJS needed to be polled from the same worker who submitted the job. With that element removed and the GeoTrellis call converted to a synchronous call ( #2102 ) we can remove that limitation.The same technique is used to ensure that during a release, the stack color of the dark stack does not get selected to process tasks from the live production site. We'll have to preserve this method without relying on
celery.current_app.control.inspect().ping()
- presumably the stack color itself can be used as arouting_key
to ensure dark stack workers keep tasks within their group.The goal in removing the current workflow is to alleviate the latency it introduces as detailed in #1894
The text was updated successfully, but these errors were encountered: