Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Frequently TCP/TLS sessions remain in CLOSE_WAIT #1881

Closed
kvrban opened this issue Mar 5, 2019 · 7 comments

Comments

Projects
None yet
3 participants
@kvrban
Copy link

commented Mar 5, 2019

Description

On our setup with Kamailio 5.2.1 with users connected via TLS. i see frequently TCP/TLS sessions remain in CLOSE_WAIT:

netstat -ntp |grep CLOSE
tcp 238 0 172.17.217.10:5061 xxx.xxx.xxx.218:52613 CLOSE_WAIT 117912/kamailio
tcp 4681 0 172.17.217.10:5061 xxx.xxx.xxx.125:8892 CLOSE_WAIT 117912/kamailio

In tcpdump i see, that the client send a TCP FIN,ACK and and the OS which is hosting the Kamailio send TCP ACK. But Kamailio is not closing. So the session remain in CLOSE_WAIT for a long time (over a hour)

We also have configured:

tcp_connection_lifetime=65
modparam("tls", "connection_timeout", 63)

Which should finally close after 65 second. But no effect.

Troubleshooting

Reproduction

Register SIP clients via TLS, and then let the client close the TCP session ia TCP FIN,ACK

  • Operating System:

Debian GNU/Linux 9

@sergey-safarov

This comment has been minimized.

Copy link
Contributor

commented Mar 5, 2019

I have see similar behavior on CentOS with default network initialization subsystem.
After I reconfigured to use systemd-networkd instead default network initialization, then i not see more "CLOSE_WAIT".

Could you reconfigure network using systemd-networkd and then retest.

@kvrban

This comment has been minimized.

Copy link
Author

commented Mar 5, 2019

I have see similar behavior on CentOS with default network initialization subsystem.
After I reconfigured to use systemd-networkd instead default network initialization, then i not see more "CLOSE_WAIT".

Could you reconfigure network using systemd-networkd and then retest.

Hey sergey, i could try this too. But as far as I understand it. CLOSE_WAIT is a issue by the application, not finally closing the session.

As described by Crist Clark

CLOSE_WAIT means that the local end of the connection has received
a FIN from the other end, but the OS is waiting for the program at the
local end to actually close its connection.

The problem is your program running on the local machine is not closing
the socket. It is not a TCP tuning issue. A connection can (and quite
correctly) stay in CLOSE_WAIT forever while the program holds the
connection open.

@miconda

This comment has been minimized.

Copy link
Member

commented Mar 5, 2019

@kvrban

This comment has been minimized.

Copy link
Author

commented Mar 5, 2019

Try by setting this core parameter:

* https://www.kamailio.org/wiki/cookbooks/devel/core#tcp_linger2

Tried out just now with "tcp_linger2=5" and restarted Kamailio. No effect. Also documentation say it for FIN_WAIT2 not for CLOSE_WAIT state.

@miconda

This comment has been minimized.

Copy link
Member

commented Mar 8, 2019

Tune the tcp keepalive parameters to shorten the lifetime duration of these sockets -- there are couple of them, see:

Also, you can tune at OS level with sysctl. The OS value can be seen in the file:

/proc/sys/net/ipv4/tcp_keepalive_time
@miconda

This comment has been minimized.

Copy link
Member

commented Mar 22, 2019

Was the hint from previous comment fixing it?

@miconda

This comment has been minimized.

Copy link
Member

commented Apr 1, 2019

Reopen if still an issue after keepalive settings.

@miconda miconda closed this Apr 1, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.