Invalid gelf message causes processing to stop #2627
Comments
@colmaengus Do you have any other inputs that receive messages on that Graylog system or is the input with the faulty messages your only one that receives messages? |
The logs are coming from a docker container via fluentd. When I get time I'll try to isolate the error source to see if it will help you reproduce the blockage. Sent from my iPhone
|
We only have one input right now and fluentd is using it. Sent from my iPhone
|
This fixes an edge case where the journal grows when there is only one input and no message can be decoded. Fixes #2627
@colmaengus Thank you for the update! You are running into an edge case that happens when there is only one input and no message can be decoded correctly. In that case the journal just grows because we are not committing the processed offset back to the journal. If you would have another input that receives messages (which can be decoded), this issue wouldn't happen. This will be fixed with #2643. |
This fixes an edge case where the journal grows when there is only one input and no message can be decoded. Fixes #2627
Expected Behavior
I would expect that bad messages would be logged and dropped rather than block message processing.
Current Behavior
As soon as the bad message is received processing stops and the journal starts filling up an no further messages are written to elasticsearch. Restarting graylog has no impact.
Steps to Reproduce (for bugs)
I've not got an easy way to reproduce this but it occurs in our setup when we include logs including base64 encoded images (not intentionally but appear due to a kafka logging)
(see log snippet below)
Context
Your Environment
The text was updated successfully, but these errors were encountered: