Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Starts using celery queues #1067

Merged
merged 1 commit into from Jul 31, 2017
Merged

Starts using celery queues #1067

merged 1 commit into from Jul 31, 2017

Conversation

noirbizarre
Copy link
Contributor

This PR introduce Celery routing and routing usage.

This create by default 3 queues:

  • high for high priority tasks (ie. search indexation)
  • default for common tasks
  • low for slow tasks (ie. harvest tasks by example)

This allows to start workers like this:

celery -A udata.worker -Q high worker
celery -A udata.worker -Q high,default worker
celery -A udata.worker -Q high,default,low worker

and will ensure that high priority tasks are always handled and low priority tasks (slow tasks) won't flood the default queue.

If no -Q parameter is set, the worker still consume all the known queues.

@noirbizarre noirbizarre added this to the 1.1.1 milestone Jul 31, 2017
@noirbizarre noirbizarre requested a review from a team July 31, 2017 09:45
@noirbizarre noirbizarre merged commit 96d0052 into opendatateam:master Jul 31, 2017
@noirbizarre noirbizarre deleted the celery-routing branch July 31, 2017 10:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants