Qpid library needs to switch to using AMQP 1.0

[bmbouter]

Qpid is planning to drop support for pre AMQP 1.0 protocols. To align with their roadmap the Qpid transport in Kombu needs to switch to using the Python bindings for the Proton client. Today the client used is the "Python Qpid Messaging API".

The Qpid team is creating a new compatibility library which will allow 0-10 concepts such as exchanges, bindings, and queues to continue to work as expected even though the wire protocol will be AMQP 1.0.

• The Qpid 0-10/1.0 compatibility library is available for testing
• Migrate the broker management usage to use the new Qpid 0-10/1.0 compatibility library
• Migrate the send/receive functionality to use Qpid Proton directly instead of the "Qpid Messaging API"
• Test with usage against celery

[bmbouter]

The Qpid team is tracking their work as QPID-7439.

[timdawborn]

What is the current status of this? I spent the day trying to work out the current state of the union of Python library support for AMQP 1.0. Given that his has been recently added to the 5.0 milestone, does that mean this won't happen any time soon?

[timdawborn]

I forgot to add — the reason I'm looking at this is that Amazon's recently released Amazon MQ (AWS-managed ActiveMQ) only speaks AMQP 1.0, so it's not currently possible (from what I can see) to use celery/kombu with it.

[bmbouter]

This code works and is being tested where I work in various ways. It needs at least two things blocking its release:

1. There is at least one Python2 compatibility issue in the code related to the python2/3 changes w.r.t the Queue library. This should be easy to resolve, but I haven't found the time.
2. the qpid_tests need to be ported to also pass on Python3

As an aside, the AMQP 1.0 is specific to Qpid, and because of the AMQP 1.0 spec any implementation will have to be custom coded to the broker. This is because all the things Celery relies on (queues, routing keys, exchanges) are not present in AMQP 1.0 so you have to find some way to "emulate" AMQP<1.0 things even though the wire protocol is AMQP 1.0.

This means that being amqp 1.0 compliant tells you nothing about the ability for Celery to run on top of it or if any existing transport is compatible with that backend.

[timdawborn]

Righteo, thanks for the status update.

Yeah, I'm well aware of the 1.0 issue. If kombu was planning on supporting Qpid AMQP 1.0, it was clear that higher level concepts would have to be emulated on top of it in the application layer rather than the protocol layer. I was doing a general state of the union check as well as checking the support of kombu as one option for us would be to move off celery and use something more AMQP 1.0 native so we can use Amazon MQ.

[auvipy]

@bmbouter you could target the amqp 1.0 to python3 only and targeted to celery 5 which will be python thre only. after the final release of celery 4.2 in coming months I will start working on droping support for python2 from masters

[bmbouter]

That sounds like a fine plan. It works with Python3 and it's easier than supporting 2.7+. This means the only remaining work is that the test_qpid.py tests be forward ported to Python3.

[jheld]

Is there any update on this?

[bmbouter]

I wasn't able to continue this codebase. Someone else would have to take over the work from my branches.

Overall I think Celery should move away from kombu and adopt AMQP 1.0 natively. I don't expect that will happen though since RabbitMQ is really focused on 0-9-1 which is < 1.0.

[jheld]

If it dropped kombu then celery would lose the virtual broker support (redis etc), correct?

[bmbouter]

It would. It would really loose everything though because the entire exchange, queue, bind concepts are all done away with in AMQP 1.0. So in essence, celery would become a Python runtime on top of AMQP 1.0 that provides additional features that th protocol no longer does. I don't think celery will do this, it would be an entirely different project probably.

[andrewkrug]

Any progress on support for AMQP 1.0?

[auvipy]

slated for next major version of celery and kombu 5

[auvipy]

this will be included in kombu 4.7/4.8. I would request you to support the work for it :)

[yardensachs]

https://aws.amazon.com/about-aws/whats-new/2020/11/announcing-amazon-mq-rabbitmq/

AmazonMQ now supports rabbti!

[auvipy]

amqp 1.0 support is needed for other usecases