xl2tpd 1.3.1 possible memory leak #23
Comments
|
Running 37a7cc3 I get a honking memory leak, 1.6GB for xl2tpd....nice. Was running for about four days. Affects both our LNS and LAC installations. Here have some valgrind lovin' xl2tpd[6042]: death_handler: Fatal signal 2 received |
|
Can you try valgrind with 95445f to see if it appears there too ? If not, then you and I can start bisecting this problem to see the commit that introduced it. |
|
Knew there was something else I wanted to mention, I do not see the problem on 1.3.1+dfsg-1 (Debian wheezy's ) release. So somewhere between and 95445f is the problem. So already tested... :) |
|
Can you give me the instructions to test the same way you are testing ? I will then test between 37a7cc3 and v1.3.6 |
|
Nothing exotic in our configuration (need to use the latest git tree though as we are doing multipath'd multilink PPP to bundle the throughput of our two uplinks): Our PPP options file is just: This is wrapped up all in a rather standard IPsec transport: There is definately something that triggers xl2tpd to start leaking memory as sometimes it runs for a day or so with no problems, then starts to grow, other times (like now) our LNS has been running fault free for over ten days and using only ~4MB of RAM. I suspect that the IPsec layer locks up and after a while naturally recovers, but of course xl2tpd is probably receiving a whole pile of errno's when sendmsg()'ing during the period. For reference, our LAC config is just: So, nothing exciting. However, I am convinced the RAM leak kicks off when there is a network interuption (probably an IPsec less setup would rarely if ever show the effect). |
|
I have a patch ready against 1.3.6. I will create a devel branch and push my patch there. I will also tag a devel version. Would you be willing to test it ? |
|
Sure! |
|
Please test v1.3.7dev1 and report back. I don't expect all leaks to be gone, but I think I got the main loop one, so that should be 90% of your report. |
|
The LAC is grumbling... |
|
meanwhile, the LNS side is just as unhappy... |
|
Dammit, I really thought I had it licked. The problem is that the code never free()s at the same level as the malloc(), so it's convoluted as hell. Sometimes I wonder why I just don't recode this whole thing using Python... Historically, it's been nice to have this in C to keep it all nice and small for routers, but show me a router that doesn't have 64MB of RAM now. OK, I will remove each extra free() I put in and give you a nudge for testing. |
|
=D shall we look forward to 1.3.7 very soon? |
|
Hi, I confirm that i also can help with testing and debug |
|
For those wanting something that does not kill your boxen, simply dump xl2tpd under runit and go back to drinking beer: |
|
I'm also seeing a memory leak on OpenWRT, version 1.3.6-5619e1771048e74b729804e8602f409af0f3faea, where xl2tpd is running as LAC. It does not seem to be a huge memory leak, because it takes several days for xl2tpd to get OOM-killed (at which point it uses 16 MB of RAM, out of the 32 MB available on the router). |
|
Can you ulimit it to a smaller amount of ram so that it gets a malloc failure? That way we can better see what object it is trying to allocate. How have you compiled it? |
|
Good idea, I've now limited xl2tpd to 7 MB of RAM, which is about twice as much as it takes normally. It's an OpenWRT router with a MIPS CPU, and I'm simply using the package for OpenWRT Barrier Breaker: http://downloads.openwrt.org/barrier_breaker/14.07/ar71xx/generic/packages/packages/ It is indeed used as a PPP/L2TP client, without IPsec. You're right, the last OOM-kill happened just after the LNS was not reachable (huge packet loss). |
|
Closing this ticket since user has work around. |
Hi,
I use xl2tpd-1.3.1 on Debian 6.0.7 x86_64 and it uses too much memory (already 1199808 kB)
On another server - about 6 Gb of memory, before server hangs.
Here is some info:
xl2tpd version: xl2tpd-1.3.1
pmap -x 2413
2413: /usr/sbin/xl2tpd
Address Kbytes RSS Dirty Mode Mapping
0000000000400000 0 92 0 r-x-- xl2tpd
000000000061a000 0 4 4 rw--- xl2tpd
000000000061b000 0 12 12 rw--- [ anon ]
0000000001b16000 0 1187564 1187564 rw--- [ anon ]
00007f2472d36000 0 16 0 r-x-- libnss_files-2.11.3.so
00007f2472d42000 0 0 0 ----- libnss_files-2.11.3.so
00007f2472f41000 0 4 4 r---- libnss_files-2.11.3.so
00007f2472f42000 0 4 4 rw--- libnss_files-2.11.3.so
00007f2472f43000 0 20 0 r-x-- libnss_nis-2.11.3.so
00007f2472f4d000 0 0 0 ----- libnss_nis-2.11.3.so
00007f247314c000 0 4 4 r---- libnss_nis-2.11.3.so
00007f247314d000 0 4 4 rw--- libnss_nis-2.11.3.so
00007f247314e000 0 24 0 r-x-- libnsl-2.11.3.so
00007f2473163000 0 0 0 ----- libnsl-2.11.3.so
00007f2473362000 0 4 4 r---- libnsl-2.11.3.so
00007f2473363000 0 4 4 rw--- libnsl-2.11.3.so
00007f2473364000 0 0 0 rw--- [ anon ]
00007f2473366000 0 12 0 r-x-- libnss_compat-2.11.3.so
00007f247336d000 0 0 0 ----- libnss_compat-2.11.3.so
00007f247356c000 0 4 4 r---- libnss_compat-2.11.3.so
00007f247356d000 0 4 4 rw--- libnss_compat-2.11.3.so
00007f247356e000 0 448 0 r-x-- libc-2.11.3.so
00007f24736c7000 0 0 0 ----- libc-2.11.3.so
00007f24738c6000 0 16 16 r---- libc-2.11.3.so
00007f24738ca000 0 4 4 rw--- libc-2.11.3.so
00007f24738cb000 0 20 20 rw--- [ anon ]
00007f24738d0000 0 92 0 r-x-- ld-2.11.3.so
00007f2473adf000 0 12 12 rw--- [ anon ]
00007f2473aeb000 0 8 8 rw--- [ anon ]
00007f2473aed000 0 4 4 r---- ld-2.11.3.so
00007f2473aee000 0 4 4 rw--- ld-2.11.3.so
00007f2473aef000 0 4 4 rw--- [ anon ]
00007fff0d0c3000 0 16 16 rw--- [ stack ]
00007fff0d1ff000 0 4 0 r-x-- [ anon ]
ffffffffff600000 0 0 0 r-x-- [ anon ]
total kB 1199808 1188408 1187700
xl2tpd.conf:
[global]
port = 1701
[lns default]
ip range = 10.3.0.2-255
assign ip = yes
local ip = 10.3.0.1
require authentication = yes
pppoptfile = /etc/ppp/options.xl2tpd
options.xl2tpd:
ms-dns 8.8.8.8
ms-dns 8.8.4.4
nomppe
require-mppe-128
+mschap-v2
+mschap
debug
plugin radius.so
plugin radattr.so
I can provide any other info to help debug/solve this problem, maby using valgrind. This is very important.
The text was updated successfully, but these errors were encountered: