Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wascally wabbit on redhat losing consumers? #40

Closed
mxriverlynn opened this issue Feb 17, 2015 · 6 comments
Closed

wascally wabbit on redhat losing consumers? #40

mxriverlynn opened this issue Feb 17, 2015 · 6 comments

Comments

@mxriverlynn
Copy link
Contributor

have you seen wascally drop consumers, randomly?

i've got a situation that i can almost reproduce on command, in RedHat Linux 7 (64bit) where my consumers drop.

is there an event or something in wascally that i can latch on to, when this happens, so i can get notified?

@mxriverlynn
Copy link
Contributor Author

i'm digging in to logs and seeing this:

=INFO REPORT==== 17-Feb-2015::14:45:43 ===

accepting AMQP connection <0.1328.0> (10.130.25.83:38529 -> 10.130.25.83:5672)

...

=ERROR REPORT==== 17-Feb-2015::14:56:00 ===

Channel error on connection <0.1328.0> (10.130.25.83:38529 -> 10.130.25.83:5672, vhost: '/scheduler', user: 'scheduler'), channel 2:

{amqp_error,precondition_failed,"unknown delivery tag 9",'basic.ack'}

some google-fu turns up several stackoverflow posts that say a message is being ack'd on the wrong channel. this one, for example: http://stackoverflow.com/questions/25496882/why-is-rabbitmq-keep-logging-unknown-delivery-tag-basic-ack

at least one post said that trying to ack the same message more than once will do this, as well.

any way to know what channel wascally is using to ack the message?

@arobson
Copy link
Collaborator

arobson commented Feb 17, 2015

Which version of wascally and what version of Node?

If you're on 0.2.0, try adding the following before starting your service via node:

DEBUG=wascally:amqp-queue node [your service]

You should see messages about what tag is getting ack'd/nack'd/rejected on which queue/connection.

You can take it a step further and get pretty detailed reporting on how wascally is trying to calculate batches with the following:

DEBUG=wascally:acknack node [your service]

I'd be very interested to see the debug output from either of those when the error occurs - it would definitely help me reproduce the scenario and fix it quickly. (I haven't run into this yet)

@mxriverlynn
Copy link
Contributor Author

node v0.10.36 (just updated this), RMQ v3.4.4 (just updated to this, as well), wascally v0.2.0

I'll add the debug flag and see what it says. is there a particular folder this is logged to, or does it go to stdout?

crazy thing is, it only happens on my RHEL install, not on my dev box - even when i reproduce the circumstances (the best i can, at least... still trying)

@arobson
Copy link
Collaborator

arobson commented Feb 17, 2015

The debug flag just causes everything to go to stdout using the debug lib. Really odd that the OS seems to make a difference.

@mxriverlynn
Copy link
Contributor Author

it looks like v0.2.1 of wascally fixed this issue. i deployed the new version this morning and haven't been able to reproduce the problem yet. going to close this for now. if i see it happening again, i'll reopen the ticket.

@celsomarques
Copy link

Hey guys,

I'm facing this issue but I can't reproduce it.

I put wascally on debug mode and I deleted one queue, after that RabbiMQt lost consumers.
How can I handle that?
Is there any event I can listen to start subscription?

Thx

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants