Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
GELF input UDP stop work #5701
I receive some packet gelf that causes graylog input down. The last message in graylog-server.log is:
The web interface report that input is OK, but if I see the port open in machine, I see down:
With graylog 2.5 version works fine.
This is happening for us aswell. We are getting this error message in the log:
Afterwards the input on the node reports as "running", but in fact accepts no new messages. After a while all Gelf UDP inputs on all nodes receive a bad package and stop; effectively not accepting any messages through UDP anymore.
We do not know which system is producing these messages and therefore it would be complicated to capture a pcap during the time.
Would be cool if the input continues to work, despite the bad package.
we were able to reproduce this issue by actually sending a single byte to the UDP input. This directly causes the
Steps to reproduce
Verify that the UDP port is open:
Send a single byte over UDP:
Somehow the input does not stop immediately:
A couple of seconds later the port is closed:
It is important to have in mind what we highlight in the docs: http://docs.graylog.org/en/3.0/pages/gelf.html#gelf-via-udp
Happens to us as well, exact same symptoms,
After that the listener (all threads of it) close, and never comeback, all while the master web gui, shows listener as active on that node (and every node which has crashed...)
And since UDP doesn't have a transmission mechanism, the data is lost (since load balancer on udp cant really health check the port, and if it does, with this bug, it might just kill the server with a simple healthcheck - port check)
System - Docker:
I am somewhat glad this has been picked up by more graylog users already, And I'm not the only one experiencing it..