AttributeError: 'Connection' object has no attribute '_frame_writer'

[harlowja]

Seeing some interesting issue that show kombu might be incompatible with the current amqp library 2.0 release.

http://logs.openstack.org/46/322446/2/check/gate-oslo.messaging-dsvm-functional-py27-rabbit/2405378/console.html#_2016-05-29_08_40_36_651

Bug that openstack is tracking that has more details about this @ https://bugs.launchpad.net/oslo.messaging/+bug/1586840

[jjahns]

You'll want to add this bug, which documents the impact this has on the specific Openstack projects.
https://bugs.launchpad.net/oslo.messaging/+bug/1586979

Right now, we're working around the problem by removing 2.0 and adding 1.4.9 back in, but this required some manual intervention, and we're not real keen on having to automate a package remove / re-addition to resume.

[ask]

It IS incompatible, and this was planned many years ago by having Kombu 3.0.0 and up depend on amqp<2.0.

I'm not sure what's going on in your build tools, but if you use pip, installing kombu should refuse to upgrade to amqp 2.0

[ask]

The requirements are defined here: https://github.com/celery/kombu/blob/3.0/requirements/default.txt,
and this is what happens for me when I install kombu in a clean virtualenv:

$ pip install kombu
Downloading/unpacking kombu
  Downloading kombu-3.0.35-py2.py3-none-any.whl (240kB): 240kB downloaded
Downloading/unpacking amqp>=1.4.9,<2.0 (from kombu)
  Downloading amqp-1.4.9-py2.py3-none-any.whl (51kB): 51kB downloaded
Downloading/unpacking anyjson>=0.3.3 (from kombu)
  Downloading anyjson-0.3.3.tar.gz
  Running setup.py (path:/Users/asksol/.virtualenvs/testkombu/build/anyjson/setup.py) egg_info for package anyjson
Installing collected packages: kombu, amqp, anyjson
  Running setup.py install for anyjson
Successfully installed kombu amqp anyjson
Cleaning up...

[harlowja]

Ya, good question, let's see if we can get the pip install logs from folks that have been broken by this (since it seems like they somehow got amqp >= 2.0? )

[jjahns]

Here's what they're running into:

https://github.com/openstack/oslo.messaging/blob/stable/mitaka/requirements.txt#L36

Master shows this:

https://github.com/openstack/oslo.messaging/blob/master/requirements.txt#L36

The statement said all that was needed: "it IS incompatible"

So it looks like there's a critical issue that someone else needs to fix with an Openstack product, agree?

[harlowja]

Ya my guess is that https://github.com/openstack/oslo.messaging/blob/stable/mitaka/requirements.txt#L36 is the problem, this is the transitive dependency issue that pip really doesn't do very well so my guess is that line 36 there is pulling in a newer version, then line 37 is not pulling in a version that is less than (due to pip sort of sucking for these kinds of things) that (even though kombu says it only works with versions less than what line 36 brings in).

[ask]

If you depend on kombu, you shouldn't have to also list amqp as a dependency. This should only be done if you require some specific version in extraordinary cases (pinning the version to ==3.0.30 for example.

What's the traceback that they are experiencing? Seems to point to _frame_writer? What is it accessing that directly for?

While I did say it is incompatible, that's not the whole story. I only expected it to be incompatible at the time when I pinned the version requirement, from what i've heard people have already managed to get kombu 3.x working with amqp 2.x.

Kombu isn't accessing _frame_writer directly though, but it may be possible for me to add a compatibility interface for Openstack in a amqp 2.0.1 release. I just need to know what it requires from that object.

[jjahns]

Here's the bug I filed.
https://bugs.launchpad.net/oslo.messaging/+bug/1586979

We still haven't gotten a response on it just yet. I do know that a lot of people who haven't attached themselves to this bug have had the issue, which appears to me a discrepancy with the requirements and not necessarily Kombu's fault or AMQP's fault for that matter.

Many ways to fix the problem though, so thanks.

[AevaOnline]

Yes, the problem is with _frame_writer:

https://gist.github.com/devananda/354ef96bf1313dd8fd66eb42229ff593

It appears to be because of the explicit dependency on amqp causing pip to pull in the latest version of it. Removing amqp from oslo.messaging/requirements.txt seems to be one way to fix the problem (tested locally).

[jjahns]

Agree. Master branch should also work.

I'm not brave enough to run a production cloud against master though. So they need to copy the change in master over to the mitaka requirements. That seems like its the quickest solution.

[harlowja]

Okie dokie, closing this for now; thanks ask and others involved.

[ask]

I misread the traceback and _frame_writer is actually used by amqp itself, it did not even exist in amqp 1.4.

The problem was simply that in amqp 2.0 the Connection constructor does not have any side effects, so you have to call conn.connect() to establish the connection. I'm releasing amqp 2.0.1 now that will automatically connect when accessing these attributes with a deprecation warning.

[harlowja]

Much appreciated!

[jjahns]

That should take care of that one. A patch is pushed up for review for the requirements as well. Much thanks on the quick response!!!