New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RabbitMQ 3.3 changes QoS apply_global behavior #339

Closed
bialecki opened this Issue Apr 13, 2014 · 7 comments

Comments

Projects
None yet
4 participants
@bialecki

bialecki commented Apr 13, 2014

See these release notes:

http://www.rabbitmq.com/blog/2014/04/02/breaking-things-with-rabbitmq-3-3/
http://www.rabbitmq.com/consumer-prefetch.html

I believe the upshot of this is the apply_global defaulting to False won't actually update the prefetch_count of a consumer in RabbitMQ 3.3+ as it previously did. It will update the prefetch_count for any consumer added after it's set.

I haven't thought it all the way through, but to preserve the current, "old" behavior, that flag needs to be set to True. I'm guessing this will need to be done via some sort of flag or transport version sniffing because the RabbitMQ folks decided to straight up redefine the meaning of that parameter and I don't think there's a way to determine how a RabbitMQ will behave other than by version number.

We can across this when experimenting with 3.3 and noticed eta tasks in celery were blocking consumers from prefetching more tasks, which is potentially very problematic if you have a lot of eta tasks.

@ericworkman

This comment has been minimized.

Show comment
Hide comment
@ericworkman

ericworkman Apr 14, 2014

Contributor

I'm seeing this, too. I created a little sample project that reproduces this, available here.

Testing this out with the sample project with RabbitMQ 3.2.X and 3.3.0 reveals the same basic behavior up until kombu tries to change the basic.qos. I'm not sure if this is due to the apply_global flag or the change to the semantics of basic.qos.

Contributor

ericworkman commented Apr 14, 2014

I'm seeing this, too. I created a little sample project that reproduces this, available here.

Testing this out with the sample project with RabbitMQ 3.2.X and 3.3.0 reveals the same basic behavior up until kombu tries to change the basic.qos. I'm not sure if this is due to the apply_global flag or the change to the semantics of basic.qos.

@ask

This comment has been minimized.

Show comment
Hide comment
@ask

ask Apr 14, 2014

Member

Weird... the changelog description says the new behavior is granted to each consumer in the channel individually, which reads to me like the old behavior. Very confusing.

Anyway, we need a reliable way to detect this, we cannot use the version number because RabbitMQ built from hg uses the version %VSN% so cannot be relied upon.

Member

ask commented Apr 14, 2014

Weird... the changelog description says the new behavior is granted to each consumer in the channel individually, which reads to me like the old behavior. Very confusing.

Anyway, we need a reliable way to detect this, we cannot use the version number because RabbitMQ built from hg uses the version %VSN% so cannot be relied upon.

@ask

This comment has been minimized.

Show comment
Hide comment
@ask

ask Apr 14, 2014

Member

But yeah, there is nothing to detect, server_properties does not hold any information about qos behavior :(

Member

ask commented Apr 14, 2014

But yeah, there is nothing to detect, server_properties does not hold any information about qos behavior :(

@ask ask closed this in 453664f Apr 14, 2014

@ask

This comment has been minimized.

Show comment
Hide comment
@ask

ask Apr 14, 2014

Member

Fixed in Celery master, but also depends on kombu master.

But note, librabbitmq did not expose server_properties and I had to upgrade to rabbitmq-c 0.5.0 to get to that information, which was extremely tricky as lots have changed. I just released librabbitmq 1.5.0 to PyPI so be sure to upgrade before you test this fix.

Member

ask commented Apr 14, 2014

Fixed in Celery master, but also depends on kombu master.

But note, librabbitmq did not expose server_properties and I had to upgrade to rabbitmq-c 0.5.0 to get to that information, which was extremely tricky as lots have changed. I just released librabbitmq 1.5.0 to PyPI so be sure to upgrade before you test this fix.

@ericworkman

This comment has been minimized.

Show comment
Hide comment
@ericworkman

ericworkman Apr 15, 2014

Contributor

The fix has an issue for RabbitMQ < 3.3.0. I made a pull request to fix it. Thank you for looking at this. Otherwise, your updates and fixes work great! I'm seeing the proper behavior with 3.2.X and 3.3.0 again.

Contributor

ericworkman commented Apr 15, 2014

The fix has an issue for RabbitMQ < 3.3.0. I made a pull request to fix it. Thank you for looking at this. Otherwise, your updates and fixes work great! I'm seeing the proper behavior with 3.2.X and 3.3.0 again.

@rogerhu

This comment has been minimized.

Show comment
Hide comment
@rogerhu

rogerhu May 3, 2014

Contributor

Yes it seems that Celery for ETA tasks will increase the prefetch count (via increment_eventually() inside the QoS class). Without this patch, it seems that the existing workers are not able to increase their limit.

https://github.com/celery/celery/blob/master/celery/worker/strategy.py#L78

Contributor

rogerhu commented May 3, 2014

Yes it seems that Celery for ETA tasks will increase the prefetch count (via increment_eventually() inside the QoS class). Without this patch, it seems that the existing workers are not able to increase their limit.

https://github.com/celery/celery/blob/master/celery/worker/strategy.py#L78

@rogerhu

This comment has been minimized.

Show comment
Hide comment
@rogerhu

rogerhu May 5, 2014

Contributor

Also, please note that in order to upgrade to Celery v3.0.11 to include this fix, the librabbitmq library must be upgraded -- can you take a look at celery/librabbitmq#43 and see if it can be incorporated in a librabbitmq release soon?

Contributor

rogerhu commented May 5, 2014

Also, please note that in order to upgrade to Celery v3.0.11 to include this fix, the librabbitmq library must be upgraded -- can you take a look at celery/librabbitmq#43 and see if it can be incorporated in a librabbitmq release soon?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment