Tasks routing #1487
Tasks routing #1487
Conversation
udata/search/__init__.py
Outdated
| @@ -121,7 +121,7 @@ def get_i18n_analyzer(): | |||
| i18n_analyzer = LocalProxy(lambda: get_i18n_analyzer()) | |||
|
|
|||
|
|
|||
| @celery.task | |||
| @task(queue='high', routing_key='high.search') | |||
There was a problem hiding this comment.
why not route='high.search'? To stay consistent.
There was a problem hiding this comment.
Just forgot to remove the queue. I added the route after as a shortcut.
| @@ -26,6 +29,7 @@ def __call__(self, *args, **kwargs): | |||
|
|
|||
| class JobTask(ContextTask): | |||
| abstract = True | |||
| default_queue = 'low' | |||
There was a problem hiding this comment.
Maybe default? Because, you know, default ;-)
There was a problem hiding this comment.
This is default for tasks, jobs are meant to be long running tasks this is why I set them to low by default. But I can undo this, but this mean we have to declare queue=low manually for every job
| @job('test-low-queue', queue='low') | ||
| @job('test-default-queue', queue='default') | ||
| @job('test-high-queue', queue='high') | ||
| def queue_test(self, raises=False): |
There was a problem hiding this comment.
can't be defined in test files?
There was a problem hiding this comment.
No, this is intentional, like log-test and error-test: this allow to manually run these jobs with udata job run -d test-high-queue and see if this is properly routing.
You can also run udata job run -d test-high-queue raises=True and you can have the time for a tasks to be processed because the exception will appear in Sentry with a timestamp.
This PR refactor the task routing: the default route can be defined at declaration time, on both
taskandjob, by either using:queueand/orrouting_keyparametersrouteparameter from whichqueueandrouting_keywill be computed (route=high.somewheregivesqueue=highandrouting_key=high.somewhere)This also allows extension to define routing on their tasks.
Some route have been added to the existing tasks and jobs.