On_message not working with rabbitmq backend

[sanketsudake]

Checklist

• I have included the output of celery -A proj report in the issue.
  (if you are not able to do this, then at least specify the Celery
  version affected).
• I have verified that the issue exists against the master branch of Celery.

Celery report output.

[root@zm03 script]# celery -A celery_test report

software -> celery:4.0.0 (latentcall) kombu:4.0.0 py:2.7.9
            billiard:3.5.0.2 py-amqp:2.1.1
platform -> system:Linux arch:64bit, ELF imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:amqp results:amqp://guest@192.168.1.201//

Steps to reproduce

http://stackoverflow.com/questions/40784593/celery-raise-improperlyconfigured-exception

Expected behavior

Expected to work

Actual behavior

Traceback (most recent call last):   
File "test.py", line 8, in <module> print(r.get(on_message=on_raw_message, propagate=False))   
File "/usr/local/python2.7/lib/python2.7/site-packages/celery/result.py", line 189, in get on_message=on_message,   
File "/usr/local/python2.7/lib/python2.7/site-packages/celery/backends/base.py", line 460, in wait_for_pending 'Backend does not support on_message callback')  
celery.exceptions.ImproperlyConfigured: Backend does not support on_message callback

[sanketsudake]

@xbeastx Created issue for followup.

[xbeastx]

For some reason RabbitMQ not an 'AsyncBackendMixin' child https://github.com/celery/celery/blob/master/celery/backends/amqp.py#L40
but Redis Async: https://github.com/celery/celery/blob/master/celery/backends/redis.py#L91

Probably reason in that AMQP result backend deprecated now:

    """The AMQP result backend.
    Deprecated: Please use the RPC backend or a persistent backend.
    """

@ask Why AMQP backend not supported anymore? And why not inherited by AsyncBackendMixin?

[xbeastx]

@ssudake21 probably better move to Redis backend, with redis everythink must work.

[ask]

@xbeastx The amqp result backend is a horrible compromise at trying to implement a result backend
using AMQP that supports multiple consumers of the same result.

It tries to mimic how the persistent result backends work (those using a database), by using
one RabbitMQ queue per task id.

This does not scale well, and can lead to RabbitMQ struggling quickly when you have thousands of tasks executing at the same time, especially with the default result expiry of one day.

The RPC result backend is a much better implementation, but since it uses one queue per client, the result can only be consumed by the process that initiated the task.

So the AMQP result backend is deprecated simply because it does not perform well.

• If you want persistent results, where multiple processes can access the state of the task: use a persistent result backend backed by a database (like Redis)
• If you want traditional RPC, use the RPC result backend.

[rlam3]

yea whats up with this error after migrating to celery 4?

[auvipy]

can anyone verify the issue on current master?

[xbeastx]

Yep. And I think we can fix this problem. Because possibility of using on_message callback depends not on asyncing backends but on possibility using native joins. Give me some time I will make pull request for this.

[auvipy]

sure plz

[thedrow]

@xbeastx Do you have the time to resolve this issue?
If not, what is your proposed fix?

[auvipy]

didn't we remove rabbitmq backend?

[thedrow]

We did. I'm closing this issue.
If someone wants to provide a fix for 4.5, we'd consider it.