How to route a task to a different broker?

[patrickwolf]

We got 5 branches and would need 5 brokers as the connection isn't always reliable.
I would like to use Redis as the broker.
What I'm missing is how to route messages/tasks between different brokers with Celery.

Is there a way to make a task which purpose it just is to forward a task to a queue on a different broker.
ie tasksRemotePEK is send of in branch LAX.
a worker in LAX picks it up and notices its for PEK
the worker forwards it to PEK
in PEK a worker finds the tasks and executes it.

How could this be setup please with Celery?
Thanks!

PS: I know that RabbitMQ supports the forwarding but it would mean deploying 5 virtual machines with RabbitMQ on them while Redis can just run as another service on our servers ie it would be much easier to deploy

[RichardKnop]

If you were using RabbitMQ, you can create a RabbitMQ cluster from your 5 brokers, then hide the 5 RabbitMQ servers behind a reverse proxy (let's say Nginx).

Then just proceed as you would normally do (setting the reverse proxy as BROKER_URL) and the tasks will be distributed among 5 RabbitMQ servers. This would solve your connection reliability issues (I have done this in the past).

I believe this would work well for your requirement and deploying RabbitMQ clusters is quite easy with Puppet from my experience.

But since you want to use Redis, this might not be very helpful. I am not sure a setup like this can be achieved with Redis.

Another option would be to have multiple different brokers (this might work with Redis as well) and have Celery apps for each broker.

You could then use different Celery apps to send tasks to different brokers. But this is a different use case compared to what you are describing, if I understand correctly.

[RichardKnop]

If you were using RabbitMQ, you can create a RabbitMQ cluster from your 5 brokers, then hide the 5 RabbitMQ servers behind a reverse proxy (let's say Nginx).

Then just proceed as you would normally do (setting the reverse proxy as BROKER_URL) and the tasks will be distributed among 5 RabbitMQ servers. This would solve your connection reliability issues (I have done this in the past).

I believe this would work well for your requirement and deploying RabbitMQ clusters is quite easy with Puppet from my experience.

But since you want to use Redis, this might not be very helpful. I am not sure a setup like this can be achieved with Redis.

Another option would be to have multiple different brokers (this might work with Redis as well) and have Celery apps for each broker.

You could then use different Celery apps to send tasks to different brokers. But this is a different use case compared to what you are describing, if I understand correctly.

[malinoff]

Is there a way to make a task which purpose it just is to forward a task to a queue on a different broker.

Sure. You can simply do exactly what app.send_task does using custom producer which may be connected to a different broker.

But this sounds like a tricky issue and I don't think I fully understand the general purpose of such setup.
If you care about high availability, you should use clustered rabbitmq as broker instead of redis. (RabbitMQ in action may help with understanding its HA setup).

[malinoff]

Is there a way to make a task which purpose it just is to forward a task to a queue on a different broker.

Sure. You can simply do exactly what app.send_task does using custom producer which may be connected to a different broker.

But this sounds like a tricky issue and I don't think I fully understand the general purpose of such setup.
If you care about high availability, you should use clustered rabbitmq as broker instead of redis. (RabbitMQ in action may help with understanding its HA setup).