TCP bad data trace out on Keyword TCPAbortOnData-Detecting and alarm

[VM-Cloud-Solutions]

Hi. For a long time we are using netdata. Very good tool.

My question is about Network Stack and TCP connection aborts. baddata (TCPAbortOnData). We have a server that shows every minute a baddata (see image). However We can not trace this bad data. Could you help use with some tips to detect and follow this baddata?. Thanks

[cakrit]

@Ferroin do you know anything about this? Googled it and came in empty, by laptop has a lot more of those that what's shown on the pic.

[Saruspete]

The keyword "TCPAbortOnData" relates to constant LINUX_MIB_TCPABORTONDATA in Kernel.
From the sources, it's used in 2 functions:

• tcp_input.c :: tcp_rcv_state_process
• tcp.c :: tcp_close

so it gets incremented when :

• Your socket goes in state TCP_FIN_WAIT1, and the linger2 time was < 0 (that's the life time of orphaned fin-wait-2 state, set by setsockopt(SO_LINGER2))
• You receive an out of order TCP FIN after close() (when the socket is in TCP_FIN_WAIT1)
• Your socket is in any state among CLOSE_WAIT, CLOSING, LAST_ACK, FIN_WAIT1, FIN_WAIT2, and you are in this case:

  		if (sk->sk_shutdown & RCV_SHUTDOWN) {
  			if (TCP_SKB_CB(skb)->end_seq != TCP_SKB_CB(skb)->seq &&
  			    after(TCP_SKB_CB(skb)->end_seq - th->fin, tp->rcv_nxt)) {
  				NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPABORTONDATA);
  				tcp_reset(sk);
  				return 1;
  			}
  		}

• You close a TCP socket that had the SO_LINGER flag, and the sk_lingertime = 0

So I'd say, start by any software that messes with the SO_LINGER option.

[VM-Cloud-Solutions]

@Saruspete, thanks for the info. As I see, it is probably a bug of the application running in this server. I am going to pass them (development) this info and additional info finding in: http://deepix.github.io/2016/10/21/tcprst.html and https://www.nybek.com/blog/2015/03/05/cross-platform-testing-of-so_linger/#download_stuff.

Thanks again for your help.

[VM-Cloud-Solutions]

The origin of baddata for The keyword "TCPAbortOnData" (for our inqury) is the application Splunkforwarder (ver 7.1.3). We don´t know consecuences (the program is working as expected) but we are sending them this bug .
Thanks.

[cakrit]

The origin of baddata for The keyword "TCPAbortOnData" (for our inqury) is the application Splunkforwarder (ver 7.1.3).

How did you actually find the application though? I went through the links and didn't quite get how to use that info to find the application that messes with SO_LINGER.

[ilyam8]

We have a server that shows every minute a baddata (see image).

The origin of baddata for The keyword "TCPAbortOnData" (for our inqury) is the application Splunkforwarder (ver 7.1.3). We don´t know consecuences (the program is working as expected) but we are sending them this bug .

So there was a problem... It looks like we need an alarm for TCPAbortOnData.

@WebLiveVideo could you create the alarm and make the PR?

[VM-Cloud-Solutions]

We have a server that shows every minute a baddata (see image).

The origin of baddata for The keyword "TCPAbortOnData" (for our inqury) is the application Splunkforwarder (ver 7.1.3). We don´t know consecuences (the program is working as expected) but we are sending them this bug .

So there was a problem... It looks like we need an alarm for TCPAbortOnData.

@WebLiveVideo could you create the alarm and make the PR?

@ilyam8 Yes, it would be nice. We will try and post the results.

[VM-Cloud-Solutions]

The origin of baddata for The keyword "TCPAbortOnData" (for our inqury) is the application Splunkforwarder (ver 7.1.3).

How did you actually find the application though? I went through the links and didn't quite get how to use that info to find the application that messes with SO_LINGER.

@cakrit Really was easy. In this server (Linux) runs five applications. We shutdown all applicaciones (p.e service stop each one) and we started one by one monitoring key word TCPAbortOnData-baddata. When we started splunkforwarder then we saw the problem. After, to be sure: only stoped/started this application several times.
Thanks

[VM-Cloud-Solutions]

The origin of baddata for The keyword "TCPAbortOnData" (for our inqury) is the application Splunkforwarder (ver 7.1.3).

How did you actually find the application though? I went through the links and didn't quite get how to use that info to find the application that messes with SO_LINGER.

Really was easy. In this server (Linux) runs five applications. We shutdown all applicaciones (p.e service stop each one) and we started one by one monitoring key word TCPAbortOnData-baddata. When we started splunkforwarder then we saw the problem. After, to be sure: only stoped/started this application several times.
Thanks

Of course, we applied this method after understanding the question thanks to the contributions of saruspete and read his recomendations. Thanks

[VM-Cloud-Solutions]

The origin of baddata for The keyword "TCPAbortOnData" (for our inqury) is the application Splunkforwarder (ver 7.1.3).

How did you actually find the application though? I went through the links and didn't quite get how to use that info to find the application that messes with SO_LINGER.

@cakrit, excuse my last comment, it is not related to your question. It was only a way of saying why at first we could not follow up the problem, but thanks to the contributions of @Saruspete gave us a light to establish the origin. thanks

[VM-Cloud-Solutions]

@ilyam8, please your help. We create the alarm.conf file: TCPAbortOnData.conf in /usr/lib/netdata/conf.d/health.d/ with the following:

# Check Networking Stack - baddata received in keyword TCPAbortOnData

alarm: 1m_ipv4_TCPAbortOnData_baddata
  on: ip.tcpconnaborts
  os: linux
  hosts: *
  lookup: sum -1m unaligned absolute of baddata
  units: baddata
  every: 60s
  warn: $this > 0
  crit: $this > (($status == $WARNING) ? (0) : (100))
  info: number of baddata received during the last minute
  to: sysadmin

We restarting netdata but we don´t receive the alarm. What's wrong? Thanks

We are working with Netdata v1.15.0-42-g6d2ea32