Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Receiving empty messages on AMQP GELF input will result in requeueing and inifinite message processing loop #1018

Closed
dfch opened this issue Mar 3, 2015 · 4 comments
Assignees
Labels
bug
Milestone

Comments

@dfch
Copy link

@dfch dfch commented Mar 3, 2015

It seems that the GELF AMQP input will basicNack unhandled messages while procesing. On empty messages (where e.g. only an AMQP header is set) this will result in requeueing the same message. However this same message will subsequently be processed by the input plugin again and again resulting in a message loop.
This behaviour occurred with a message sent via RabbitMQ with only AMQP headers set (but with an empty message body) on a Graylog v1.0.0 server.

@bernd bernd added the bug label Mar 3, 2015
@bernd bernd added this to the 1.0.1 milestone Mar 3, 2015
@bernd
Copy link
Member

@bernd bernd commented Mar 3, 2015

Thanks for the report!

@bernd bernd self-assigned this Mar 4, 2015
@bernd
Copy link
Member

@bernd bernd commented Mar 4, 2015

Any GELF input requires a valid GELF message. Sending a GELF message with an empty body is an error.

I guess the problem here is that the AmqpConsumer is re-queueing messages that cannot be parsed. Maybe we should not re-queue the messages in case of an error and use channel.basicNack(deliveryTag, false, false); instead. (maybe make that configurable?)

Running into the same error with the same message sounds wrong to me.

Comments? /cc @dennisoelkers @joschi @kroepke

@dfch
Copy link
Author

@dfch dfch commented Mar 4, 2015

Hi Bernd, this is exactly what I mean. As the GELF AMQP plugin cannot enforce the format of the message that is sent by a publisher, this will lead to a re-processing of the message (in most cases). As an addition to just providing a flag whether or not to requeue, you could also write a qualified "warning" message to the system log (instead of just writing a full stack trace), like you do when a plugin fails or any other system event occurs.

@joschi
Copy link
Contributor

@joschi joschi commented Mar 4, 2015

+1 for adding a configuration flag for the handling of invalid messages to the GELF AMQP Input. (Discard invalid messages? true/false)

bernd added a commit that referenced this issue Mar 5, 2015
bernd added a commit that referenced this issue Mar 5, 2015
@bernd bernd closed this in 3856781 Mar 5, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.