Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Syslog TCP RST send by Graylog #1105
I send syslogs message from a firewall over TCP to Graylog. I received those messages well but after a while, Graylog closes the TCP session with a RST packet. I do not understand this behavior
Could'you help me please ?
Here is versions I use:
rpm -qa | grep graylog graylog-1.0-repository-el6-1.2.0-1.noarch graylog-web-1.0.1-1.noarch graylog-server-1.0.1-1.noarch Kernel 2.6.32-358.el6.x86_64
And Input in Cluster
Syslog TCP 1514 (Syslog TCP) running Network IO: 16,9kiB 0B (total: 13,0GiB 0B ) Total connections: 11512 (2 active) recv_buffer_size: 1048576 port: 1514 tls_key_file: tls_key_password: ******* max_message_size: 2097152 override_source: allow_override_date: true bind_address: 0.0.0.0 tls_cert_file:
Another report from the mailing list that seems to be related:
I'm using syslog-ng to feed in data via a syslog/TCP channel and it's continually (every 10 seconds) dropping the TCP channel - forcing syslog-ng to restart it
tcpdump shows normal data flow followed by two TCP resets coming back from the graylog-1.1.5 server - so it's definitely graylog that's borking.
BTW, this system is working: I'm seeing these syslogs flowing in - can do searches/etc - but I assume I'm losing some records due to this issue. I even created a xinetd.d based tcp service on the graylog server that just logged what it received to a file, configured the syslog server to send to both tcp channels - and it's running fine with no restarts (ie tcpdump of both ports only shows TCP resets on the graylog port not the xinetd port). So I think that implies it isn't the OS (CentOS-7)
My syslog-ng.conf contains the following
I have the tcp channel aimed at an xinetd service that just writes the content to disk - it doesn't crash/TCP-reset. The "syslog" channel goes to the TCP syslog Input channel in graylog and shows the TCP-reset issue
The reason I'm using "syslog" is because graylog didn't like the "tcp" option - the content just blackholes all together. Graylog documentation about syslog-ng referred to using "syslog" (which I didn't even know existed as an option in syslog-ng) and that works - except for the TCP-reset issue
ngrep shows the traffic looking as follows - you can see the difference between the "tcp" (9876) and "syslog" (1514) channel formats
FYI I just moved off using syslog/TCP to syslog/UDP and immediately saw graylog messages/sec leap for a factor of 4 times!! We were losing 75% of all TCP syslogs due to this issue
Thankfully the central syslog server is in the same rack as the graylog server - so using UDP is OK - but that certainly wouldn't be possible over WANs/etc
In my case I had to setup rsyslog between syslog sender and graylog as a workaround
I using syslog/TCP to ensure message integrity and reading the last post I understand that I still could lose messages. I see bursts up to a 250 messages/second
Could TCP keepalive solve this issue if it avoids too many open TCP socket
added a commit
Aug 4, 2015
referenced this issue
Aug 4, 2015
The TCP RST was sent because Graylog closed the connection after an exception has been thrown. This just wasn't logged because we were missing a exception logger for the Netty pipeline. (will be fixed in #1340)
See #1339 for a fix. This will be released in Graylog 1.2.0.
Thank you very much for all the debugging information!