Add TCP keepalive socket options support #17

dshcherb · 2019-02-28T22:05:50Z

Add support for printing tcp_keepalive_intvl, tcp_keepalive_probes and
tcp_keepalive_time.

keepalive_time and keepalive_intvl have to be divided by HZ as they are
multiplied by that value in the kernel code before being stored:

net/ipv4/tcp.c:

tp->keepalive_time = val * HZ;

tp->keepalive_intvl = val * HZ;

This is quite useful to inspect if an application overrides default
keepalive values somewhere in its code from system defaults which are
usually quite high (2 hours for tcp_keepalive_time by default).

Example:

tcp_keepalive_intvl = 15
tcp_keepalive_probes = 2
tcp_keepalive_time = 15

output:

// ... SO_KEEPALIVE=1,TCP_KEEPIDLE=15,TCP_KEEPCNT=2,TCP_KEEPINTVL=15

veithen · 2019-03-10T17:04:37Z

Thanks for submitting a patch. What kernel version did you test this with?

dshcherb · 2019-03-10T18:33:23Z

@veithen

Distribution kernel 4.15:

$ uname -a
Linux maas 4.15.0-34-generic #37-Ubuntu SMP Mon Aug 27 15:21:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

$ make
make -C /lib/modules/4.15.0-34-generic/build M=/home/ubuntu/knetstat modules
make[1]: Entering directory '/usr/src/linux-headers-4.15.0-34-generic'
Makefile:976: "Cannot use CONFIG_STACK_VALIDATION=y, please install libelf-dev, libelf-devel or elfutils-libelf-devel"
  CC [M]  /home/ubuntu/knetstat/knetstat.o
/home/ubuntu/knetstat/knetstat.c: In function ‘tcp_seq_show’:
/home/ubuntu/knetstat/knetstat.c:219:75: warning: comparison between pointer and integer
     } else if (tp->snd_nxt > tp->snd_una && tcp_time_stamp-tp->rcv_tstamp > HZ) {
                                                                           ^
  Building modules, stage 2.
  MODPOST 1 modules
  CC      /home/ubuntu/knetstat/knetstat.mod.o
  LD [M]  /home/ubuntu/knetstat/knetstat.ko
make[1]: Leaving directory '/usr/src/linux-headers-4.15.0-34-generic'

$ sudo insmod ./knetstat.ko

$ cat /proc/net/tcpstat 
Recv-Q Send-Q Local Address           Foreign Address         Stat Diag Options
     0      0 0.0.0.0:5247            0.0.0.0:*               LSTN      SO_REUSEADDR=1,SO_REUSEPORT=0,SO_KEEPALIVE=0,TCP_KEEPIDLE=0,TCP_KEEPCNT=0,TCP_KEEPINTVL=0,TCP_NODELAY=0,TCP_FASTOPEN=0,TCP_DEFER_ACCEPT=0
     0      0 0.0.0.0:5248            0.0.0.0:*               LSTN      SO_REUSEADDR=1,SO_REUSEPORT=0,SO_KEEPALIVE=0,TCP_KEEPIDLE=0,TCP_KEEPCNT=0,TCP_KEEPINTVL=0,TCP_NODELAY=0,TCP_FASTOPEN=0,TCP_DEFER_ACCEPT=0
     0      0 0.0.0.0:7911            0.0.0.0:*               LSTN      SO_REUSEADDR=1,SO_REUSEPORT=0,SO_KEEPALIVE=0,TCP_KEEPIDLE=0,TCP_KEEPCNT=0,TCP_KEEPINTVL=0,TCP_NODELAY=0,TCP_FASTOPEN=0,TCP_DEFER_ACCEPT=0
     0      0 192.0.2.2:53          0.0.0.0:*               LSTN      SO_REUSEADDR=1,SO_REUSEPORT=0,SO_KEEPALIVE=0,TCP_KEEPIDLE=0,TCP_KEEPCNT=0,TCP_KEEPINTVL=0,TCP_NODELAY=0,TCP_FASTOPEN=5,TCP_DEFER_ACCEPT=0
     0      0 127.0.0.1:53            0.0.0.0:*               LSTN      SO_REUSEADDR=1,SO_REUSEPORT=0,SO_KEEPALIVE=0,TCP_KEEPIDLE=0,TCP_KEEPCNT=0,TCP_KEEPINTVL=0,TCP_NODELAY=0,TCP_FASTOPEN=5,TCP_DEFER_ACCEPT=0
     0      0 127.0.0.53:53           0.0.0.0:*               LSTN      SO_REUSEADDR=1,SO_REUSEPORT=0,SO_KEEPALIVE=0,TCP_KEEPIDLE=0,TCP_KEEPCNT=0,TCP_KEEPINTVL=0,TCP_NODELAY=0,TCP_FASTOPEN=0,TCP_DEFER_ACCEPT=0
     0      0 0.0.0.0:22              0.0.0.0:*               LSTN      SO_REUSEADDR=1,SO_REUSEPORT=0,SO_KEEPALIVE=0,TCP_KEEPIDLE=0,TCP_KEEPCNT=0,TCP_KEEPINTVL=0,TCP_NODELAY=0,TCP_FASTOPEN=0,TCP_DEFER_ACCEPT=0
     0      0 127.0.0.1:5432          0.0.0.0:*               LSTN      SO_REUSEADDR=1,SO_REUSEPORT=0,SO_KEEPALIVE=0,TCP_KEEPIDLE=0,TCP_KEEPCNT=0,TCP_KEEPINTVL=0,TCP_NODELAY=0,TCP_FASTOPEN=0,TCP_DEFER_ACCEPT=0
     0      0 127.0.0.1:953           0.0.0.0:*               LSTN      SO_REUSEADDR=1,SO_REUSEPORT=0,SO_KEEPALIVE=0,TCP_KEEPIDLE=0,TCP_KEEPCNT=0,TCP_KEEPINTVL=0,TCP_NODELAY=0,TCP_FASTOPEN=64,TCP_DEFER_ACCEPT=0
     0      0 127.0.0.1:954           0.0.0.0:*               LSTN      SO_REUSEADDR=1,SO_REUSEPORT=0,SO_KEEPALIVE=0,TCP_KEEPIDLE=0,TCP_KEEPCNT=0,TCP_KEEPINTVL=0,TCP_NODELAY=0,TCP_FASTOPEN=64,TCP_DEFER_ACCEPT=0
     0      0 192.0.2.2:22          198.51.100.228:40586      ESTB      SO_REUSEADDR=1,SO_REUSEPORT=0,SO_KEEPALIVE=1,TCP_KEEPIDLE=0,TCP_KEEPCNT=0,TCP_KEEPINTVL=0,TCP_NODELAY=1,TCP_DEFER_ACCEPT=0

dshcherb · 2019-03-10T18:54:56Z

nc -l 4242 &
[1] 3118


$ python3
Python 3.6.7 (default, Oct 22 2018, 11:32:17) 
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> s.connect(('127.0.0.1', 4242))
>>> s.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)

$ grep 4242 /proc/net/tcpstat 
     0      0 0.0.0.0:4242            0.0.0.0:*               LSTN      SO_REUSEADDR=1,SO_REUSEPORT=1,SO_KEEPALIVE=0,TCP_KEEPIDLE=0,TCP_KEEPCNT=0,TCP_KEEPINTVL=0,TCP_NODELAY=0,TCP_FASTOPEN=0,TCP_DEFER_ACCEPT=0
     0      0 127.0.0.1:53754         127.0.0.1:4242          ESTB      SO_REUSEADDR=0,SO_REUSEPORT=0,SO_KEEPALIVE=1,TCP_KEEPIDLE=0,TCP_KEEPCNT=0,TCP_KEEPINTVL=0,TCP_NODELAY=0,TCP_DEFER_ACCEPT=0
     0      0 127.0.0.1:4242          127.0.0.1:53754         ESTB      SO_REUSEADDR=1,SO_REUSEPORT=1,SO_KEEPALIVE=0,TCP_KEEPIDLE=0,TCP_KEEPCNT=0,TCP_KEEPINTVL=0,TCP_NODELAY=0,TCP_DEFER_ACCEPT=0

>>> s.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPINTVL, 15)

$ grep 4242 /proc/net/tcpstat 
     0      0 0.0.0.0:4242            0.0.0.0:*               LSTN      SO_REUSEADDR=1,SO_REUSEPORT=1,SO_KEEPALIVE=0,TCP_KEEPIDLE=0,TCP_KEEPCNT=0,TCP_KEEPINTVL=0,TCP_NODELAY=0,TCP_FASTOPEN=0,TCP_DEFER_ACCEPT=0
     0      0 127.0.0.1:53754         127.0.0.1:4242          ESTB      SO_REUSEADDR=0,SO_REUSEPORT=0,SO_KEEPALIVE=1,TCP_KEEPIDLE=0,TCP_KEEPCNT=0,TCP_KEEPINTVL=15,TCP_NODELAY=0,TCP_DEFER_ACCEPT=0
     0      0 127.0.0.1:4242          127.0.0.1:53754         ESTB      SO_REUSEADDR=1,SO_REUSEPORT=1,SO_KEEPALIVE=0,TCP_KEEPIDLE=0,TCP_KEEPCNT=0,TCP_KEEPINTVL=0,TCP_NODELAY=0,TCP_DEFER_ACCEPT=0

>>> s.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPIDLE, 10)

$ grep 4242 /proc/net/tcpstat 
     0      0 0.0.0.0:4242            0.0.0.0:*               LSTN      SO_REUSEADDR=1,SO_REUSEPORT=1,SO_KEEPALIVE=0,TCP_KEEPIDLE=0,TCP_KEEPCNT=0,TCP_KEEPINTVL=0,TCP_NODELAY=0,TCP_FASTOPEN=0,TCP_DEFER_ACCEPT=0
     0      0 127.0.0.1:53754         127.0.0.1:4242          ESTB      SO_REUSEADDR=0,SO_REUSEPORT=0,SO_KEEPALIVE=1,TCP_KEEPIDLE=10,TCP_KEEPCNT=0,TCP_KEEPINTVL=15,TCP_NODELAY=0,TCP_DEFER_ACCEPT=0
     0      0 127.0.0.1:4242          127.0.0.1:53754         ESTB      SO_REUSEADDR=1,SO_REUSEPORT=1,SO_KEEPALIVE=0,TCP_KEEPIDLE=0,TCP_KEEPCNT=0,TCP_KEEPINTVL=0,TCP_NODELAY=0,TCP_DEFER_ACCEPT=0

>>> s.setsockopt(socket.IPPROTO_TCP, socket.TCP_KEEPCNT, 8)

$ grep 4242 /proc/net/tcpstat 
     0      0 0.0.0.0:4242            0.0.0.0:*               LSTN      SO_REUSEADDR=1,SO_REUSEPORT=1,SO_KEEPALIVE=0,TCP_KEEPIDLE=0,TCP_KEEPCNT=0,TCP_KEEPINTVL=0,TCP_NODELAY=0,TCP_FASTOPEN=0,TCP_DEFER_ACCEPT=0
     0      0 127.0.0.1:53754         127.0.0.1:4242          ESTB      SO_REUSEADDR=0,SO_REUSEPORT=0,SO_KEEPALIVE=1,TCP_KEEPIDLE=10,TCP_KEEPCNT=8,TCP_KEEPINTVL=15,TCP_NODELAY=0,TCP_DEFER_ACCEPT=0
     0      0 127.0.0.1:4242          127.0.0.1:53754         ESTB      SO_REUSEADDR=1,SO_REUSEPORT=1,SO_KEEPALIVE=0,TCP_KEEPIDLE=0,TCP_KEEPCNT=0,TCP_KEEPINTVL=0,TCP_NODELAY=0,TCP_DEFER_ACCEPT=0

veithen · 2019-03-10T19:38:27Z

Thanks. IIUC, zero values for those option mean that the kernel uses the default specified by the net.ipv4.tcp_* kernel parameters. In that case, I think it would make sense to only output the options if they have non zero values. WDYT?

dshcherb · 2019-03-10T20:54:23Z

@veithen Yes, if they are not overridden (set to 0) sysctls values are fetched instead:

https://elixir.bootlin.com/linux/v4.15.18/source/include/net/tcp.h#L1312

static inline int keepalive_intvl_when(const struct tcp_sock *tp)
{
	struct net *net = sock_net((struct sock *)tp);

	return tp->keepalive_intvl ? : net->ipv4.sysctl_tcp_keepalive_intvl;
}

static inline int keepalive_time_when(const struct tcp_sock *tp)
{
	struct net *net = sock_net((struct sock *)tp);

	return tp->keepalive_time ? : net->ipv4.sysctl_tcp_keepalive_time;
}
static inline int keepalive_probes(const struct tcp_sock *tp)
{
	struct net *net = sock_net((struct sock *)tp);

	return tp->keepalive_probes ? : net->ipv4.sysctl_tcp_keepalive_probes;
}

I think we could hide them if they contained non-zero values or we could say something like 0(sysctl_value=<x>) which would make the kernel behavior a bit more clear which is always helpful during debugging.

Another improvement would be to avoid printing these values in case SO_KEEPALIVE is 0 (sock_flag(sk, SOCK_KEEPOPEN) == 0).

https://elixir.bootlin.com/linux/v4.15.18/source/net/core/sock.c#L1148

	case SO_KEEPALIVE:
		v.val = sock_flag(sk, SOCK_KEEPOPEN);
		break;

So, in summary, I would settle on:

<parameter_name>=0(sysctl_value=<x>). sysctl values could be retrieved by calling the respective functions {keepalive_intvl_when, keepalive_time_when, keepalive_probes};
If sock_flag(sk, SOCK_KEEPOPEN) == 0 do not print any keepalive-related values.

Thoughts?

veithen · 2019-03-10T23:05:29Z

I would keep it simple and just suppress the output for each TCP_KEEP* option if its value is zero.

dshcherb · 2019-03-10T23:38:45Z

Pull-request updated, HEAD is now 234a906

Add support for printing tcp_keepalive_intvl, tcp_keepalive_probes and tcp_keepalive_time. keepalive_time and keepalive_intvl have to be divided by HZ as they are multiplied by that value in the kernel code before being stored: net/ipv4/tcp.c: tp->keepalive_time = val * HZ; tp->keepalive_intvl = val * HZ; This is quite useful to inspect if an application overrides default keepalive values somewhere in its code from system defaults which are usually quite high (2 hours for tcp_keepalive_time by default). If a given value is 0 the kernel takes a sysctl value for that parameter instead and knetstat will not print that value at all with this change. Example: tcp_keepalive_intvl = 15 tcp_keepalive_probes = 2 tcp_keepalive_time = 15 output: // ... SO_KEEPALIVE=1,TCP_KEEPIDLE=15,TCP_KEEPCNT=2,TCP_KEEPINTVL=15

dshcherb · 2019-03-10T23:49:25Z

Pull-request updated, HEAD is now 8798664

dshcherb · 2019-03-10T23:50:13Z

@veithen

Alright, let's keep it that way. I made some comments about that in the commit message and readme.

dshcherb · 2019-03-11T21:24:22Z

Thanks @veithen

veithen merged commit be75662 into veithen:master Mar 11, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add TCP keepalive socket options support #17

Add TCP keepalive socket options support #17

dshcherb commented Feb 28, 2019

veithen commented Mar 10, 2019

dshcherb commented Mar 10, 2019

dshcherb commented Mar 10, 2019

veithen commented Mar 10, 2019

dshcherb commented Mar 10, 2019

veithen commented Mar 10, 2019

dshcherb commented Mar 10, 2019

dshcherb commented Mar 10, 2019

dshcherb commented Mar 10, 2019

dshcherb commented Mar 11, 2019

Add TCP keepalive socket options support #17

Add TCP keepalive socket options support #17

Conversation

dshcherb commented Feb 28, 2019

veithen commented Mar 10, 2019

dshcherb commented Mar 10, 2019

dshcherb commented Mar 10, 2019

veithen commented Mar 10, 2019

dshcherb commented Mar 10, 2019

veithen commented Mar 10, 2019

dshcherb commented Mar 10, 2019

dshcherb commented Mar 10, 2019

dshcherb commented Mar 10, 2019

dshcherb commented Mar 11, 2019