L2TP connection issues, that crashes VPN server #1972

LinearAlpha · 2024-03-28T12:59:39Z

Are you using SoftEther VPN 5.x?

Yes, I'm using SoftEther VPN 5.x, not 4.x.

Version

5.02.5182

Component

VPN Server

Operating system & version

Ubuntu 22.04.4 LTS (GNU/Linux 6.5.0-26-generic x86_64)

Architecture or Hardware model

AMD64 (on the KVM hypervisor)

Steps to reproduce

vpn_server.zip

Connect VPN with SoftetherVPN client
Connect VPN server with L2TP protocol

Both connections got disconnected or the SoftetherVPN protocol disconnected first, and L2TP session disconnected 1 ~ 2 minutes later.

✔️ Expected Behavior

The VPN server can concurrently maintain sessions both from the SoftetherVPN protocol and L2TP protocol.

❌ Actual Behavior

Both of the connections are getting disconnected.

Anything else?

The VPN server setting file has been attached
vpn_server.zip

When I connected the VPN server with the L2TP protocol, another session connected by the SoftetherVPN client got disconnected.
Also, the client connected by L2TP is getting disconnected 2 minutes later.

LinearAlpha · 2024-04-01T01:33:24Z

For more information, this is the dmesg record when the VPN is dropping sessions.

[ 765.100798] vpnserver[1346]: segfault at 2 ip 00007522e682786d sp 00007522e5774418 error 4 in libmayaqua.so[7522e67f8000+66000] likely on CPU 0 (core 0, socket 0)
[ 765.100822] Code: bb fd ff 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa ba 14 00 00 00 e9 62 db fd ff 66 90 f3 0f 1e fa 48 85 ff 74 1f 48 8d 47 06 <80> 3f 00 75 16 48 83 c7 01 48 39 c7 75 f2 b8 01 00 00 00 c3 0f 1f

chipitsine · 2024-04-17T11:57:26Z

what is used as L2TP client ?

chipitsine · 2024-04-17T11:58:16Z

btw, if that's reproducible in your environment, do you mind collecting core dump and analyze it using gdb "bt" ? I can describe all steps

LinearAlpha · 2024-04-17T12:06:25Z

iPhone 12 pro (iOS 17.4.1), iPad Pro 3rd gen (with M1, iPadOS 17.4.1)

And yes, if you do not mind, could you please show me how to collect the core dump and analyze it using gdb "bt"?

chipitsine · 2024-04-17T12:17:05Z

ср, 17 апр. 2024 г. в 14:06, Minpyo Kim ***@***.***>:

iPhone 12 pro (iOS 17.4.1), iPad Pro 3rd gen (with M1, iPadOS 17.4.1)

I did have a hope that it is something on Linux (and thus we could add it to CI/CD). ok

And yes, if you do not mind, could you please show me how to collect the core dump and analyze it using gdb "bt"?

I'll write steps. thanks!

…

— Reply to this email directly, view it on GitHub <#1972 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQ5KUHOYDHRGSMRGYFGSK3Y5ZQVPAVCNFSM6AAAAABFMXWSSOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANRRGEYDMNJRGA> . You are receiving this because you commented.Message ID: ***@***.***>

chipitsine · 2024-04-17T13:44:48Z

as for collecting dump, I'd say there're two parts

enabling dumps on Linux (if you are familiar, you can skip)

sysctl part (please refer to your linux distro, there may be a wrapper)

sysctl kernel.core_pattern

shell part (I'd setup "ulimit -c unlimited")

ulimit -c

building SE with debug info. assuming we are building from source

(clang shows more friendly back traces, but you can try gcc as well)

mkdir build
cd build
export CC=clang
cmake -DCMAKE_C_FLAGS='-ggdb -fsanitize=address -O1' ..
cd ..
make -C build

after that, I'd invoke vpnserver as follow (to keep ulimit from current shell)

build/vpnserver execsvc

if you are lucky, there should be core dump.
try opening it using gdb

gdb build/vpnserver <path-to-core-dump>

and invoke either "bt" or "bt full" or "thread apply all bt full"

chipitsine · 2024-04-17T13:46:54Z

I've provided steps with ASAN. it should produce its own trace, in such case no coredump is needed. Even better.
if it will not produce a trace, we can try coredump/gdb

LinearAlpha · 2024-04-26T01:04:54Z

Hello, sorry for the delay.

While running VPNserver by
build/vpnserver execsvc

I'm getting the error shown below.

=================================================================
==6290==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000004 (pc 0x77206d4c49d0 bp 0x7720698bad90 sp 0x7720698ba558 T31)
==6290==The signal is caused by a READ memory access.
==6290==Hint: address points to the zero page.
#0 0x77206d4c49d0 string/../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:383
#1 0x5d5f83e855e1 in __asan_memcpy (/root/SE-debug/SoftEtherVPN/build/vpnserver+0xa05e1) (BuildId: 1687da78342b5a1c6f3f71a831ddcfd5801fc7c9)
#2 0x77206d8ce2a7 in Copy /root/SE-debug/SoftEtherVPN/src/Mayaqua/Memory.c:3981:2
#3 0x77206de8f60e in IPCIPv6AddRouterPrefixes /root/SE-debug/SoftEtherVPN/src/Cedar/IPC.c:2357:5
#4 0x77206de8dbbf in IPCProcessL3EventsEx /root/SE-debug/SoftEtherVPN/src/Cedar/IPC.c:1518:10
#5 0x77206df21c58 in PPPThread /root/SE-debug/SoftEtherVPN/src/Cedar/Proto_PPP.c:371:4
#6 0x77206d8c20b3 in ThreadPoolProc /root/SE-debug/SoftEtherVPN/src/Mayaqua/Kernel.c:872:3
#7 0x77206d9a44cd in UnixDefaultThreadProc /root/SE-debug/SoftEtherVPN/src/Mayaqua/Unix.c:1606:2
#8 0x77206d494ac2 in start_thread nptl/./nptl/pthread_create.c:442:8
#9 0x77206d52684f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV string/../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:383
Thread T31 created by T10 here:
#0 0x5d5f83e6f64c in __interceptor_pthread_create (/root/SE-debug/SoftEtherVPN/build/vpnserver+0x8a64c) (BuildId: 1687da78342b5a1c6f3f71a831ddcfd5801fc7c9)
#1 0x77206d9a0053 in UnixInitThread /root/SE-debug/SoftEtherVPN/src/Mayaqua/Unix.c:1685:6
#2 0x77206d8c29aa in NewThreadInternal /root/SE-debug/SoftEtherVPN/src/Mayaqua/Kernel.c:1200:7

Thread T10 created by T0 here:
#0 0x5d5f83e6f64c in __interceptor_pthread_create (/root/SE-debug/SoftEtherVPN/build/vpnserver+0x8a64c) (BuildId: 1687da78342b5a1c6f3f71a831ddcfd5801fc7c9)
#1 0x77206d9a0053 in UnixInitThread /root/SE-debug/SoftEtherVPN/src/Mayaqua/Unix.c:1685:6
#2 0x77206d8c29aa in NewThreadInternal /root/SE-debug/SoftEtherVPN/src/Mayaqua/Kernel.c:1200:7

==6290==ABORTING

LinearAlpha · 2024-04-26T01:06:14Z

However, I was not able to get a core dump.
Please let me know if I need to collect more data.

LinearAlpha · 2024-04-26T01:25:21Z

Also, during troubleshooting, I've found that this crash also occurs when I'm using the OpenVPN protocol.

This is the error message when I test with OpenVPN protocol.

AddressSanitizer:DEADLYSIGNAL

=================================================================
==7589==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000004 (pc 0x7b98e3ac49d0 bp 0x615000c443f7 sp 0x7b98decacb48 T30)
==7589==The signal is caused by a READ memory access.
==7589==Hint: address points to the zero page.
#0 0x7b98e3ac49d0 (/lib/x86_64-linux-gnu/libc.so.6+0xc49d0)
#1 0x7b98e4027ed5 in IPCIPv6AddRouterPrefixes /root/SE-debug/SoftEtherVPN/src/Cedar/IPC.c:2357
#2 0x7b98e4029ead in IPCProcessL3EventsEx /root/SE-debug/SoftEtherVPN/src/Cedar/IPC.c:1518
#3 0x7b98e4029ead in IPCProcessL3EventsEx /root/SE-debug/SoftEtherVPN/src/Cedar/IPC.c:1356
#4 0x7b98e402c733 in IPCProcessL3EventsIPv4Only /root/SE-debug/SoftEtherVPN/src/Cedar/IPC.c:1352
#5 0x7b98e409e847 in OvsRecvPacket /root/SE-debug/SoftEtherVPN/src/Cedar/Proto_OpenVPN.c:2295
#6 0x7b98e40a016b in OvsProcessDatagrams /root/SE-debug/SoftEtherVPN/src/Cedar/Proto_OpenVPN.c:238
#7 0x7b98e40a016b in OvsProcessDatagrams /root/SE-debug/SoftEtherVPN/src/Cedar/Proto_OpenVPN.c:227
#8 0x7b98e4059408 in ProtoSessionThread /root/SE-debug/SoftEtherVPN/src/Cedar/Proto.c:740
#9 0x7b98e4059408 in ProtoSessionThread /root/SE-debug/SoftEtherVPN/src/Cedar/Proto.c:719
#10 0x7b98e3cd7c62 in ThreadPoolProc /root/SE-debug/SoftEtherVPN/src/Mayaqua/Kernel.c:872
#11 0x7b98e3cd7c62 in ThreadPoolProc /root/SE-debug/SoftEtherVPN/src/Mayaqua/Kernel.c:827
#12 0x7b98e3d6cbb9 in UnixDefaultThreadProc /root/SE-debug/SoftEtherVPN/src/Mayaqua/Unix.c:1606
#13 0x7b98e3a94ac2 in start_thread nptl/pthread_create.c:442
#14 0x7b98e3b2684f (/lib/x86_64-linux-gnu/libc.so.6+0x12684f)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/lib/x86_64-linux-gnu/libc.so.6+0xc49d0)
Thread T30 created by T2 here:
#0 0x7b98e4458685 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:216
#1 0x7b98e3d6d4c7 in UnixInitThread /root/SE-debug/SoftEtherVPN/src/Mayaqua/Unix.c:1685

Thread T2 created by T0 here:
#0 0x7b98e4458685 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:216
#1 0x7b98e3d6d4c7 in UnixInitThread /root/SE-debug/SoftEtherVPN/src/Mayaqua/Unix.c:1685

==7589==ABORTING

chipitsine · 2024-04-26T05:13:29Z

thanks!

chipitsine · 2024-04-26T05:15:30Z

is it built using gcc ? can you try clang (I recall it produced more helpful info).

anyway, there are enough data to have a look at

LinearAlpha · 2024-04-26T05:36:51Z

It was built with clang.

Maybe I can try to build it again to see if it is giving a different message.

chipitsine · 2024-04-27T06:01:55Z

@LinearAlpha , can you try the following ?

#1990

LinearAlpha · 2024-04-27T10:34:18Z

I tried, and still the same

AddressSanitizer:DEADLYSIGNAL

=================================================================
==2440==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000002 (pc 0x7f1f52d0c98f bp 0x7f1f4efc7e10 sp 0x7f1f4efc7cf0 T29)
==2440==The signal is caused by a READ memory access.
==2440==Hint: address points to the zero page.
#0 0x7f1f52d0c98f in IsMacInvalid /root/SE-debug/SoftEtherVPN/src/Mayaqua/Network.c:7490:7
#1 0x7f1f52d0c98f in IsMacUnicast /root/SE-debug/SoftEtherVPN/src/Mayaqua/Network.c:7561:6
#2 0x7f1f5328f892 in IPCIPv6AssociateOnNDTEx /root/SE-debug/SoftEtherVPN/src/Cedar/IPC.c:2185:10
#3 0x7f1f5328dc2e in IPCProcessL3EventsEx /root/SE-debug/SoftEtherVPN/src/Cedar/IPC.c
#4 0x7f1f53321c78 in PPPThread /root/SE-debug/SoftEtherVPN/src/Cedar/Proto_PPP.c:371:4
#5 0x7f1f52cc20b3 in ThreadPoolProc /root/SE-debug/SoftEtherVPN/src/Mayaqua/Kernel.c:872:3
#6 0x7f1f52da44cd in UnixDefaultThreadProc /root/SE-debug/SoftEtherVPN/src/Mayaqua/Unix.c:1606:2
#7 0x7f1f52894ac2 in start_thread nptl/./nptl/pthread_create.c:442:8
#8 0x7f1f5292684f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /root/SE-debug/SoftEtherVPN/src/Mayaqua/Network.c:7490:7 in IsMacInvalid
Thread T29 created by T27 here:
#0 0x5e3b66a0464c in __interceptor_pthread_create (/root/SE-debug/SoftEtherVPN/build/vpnserver+0x8a64c) (BuildId: 1687da78342b5a1c6f3f71a831ddcfd5801fc7c9)
#1 0x7f1f52da0053 in UnixInitThread /root/SE-debug/SoftEtherVPN/src/Mayaqua/Unix.c:1685:6
#2 0x7f1f52cc29aa in NewThreadInternal /root/SE-debug/SoftEtherVPN/src/Mayaqua/Kernel.c:1200:7

Thread T27 created by T25 here:
#0 0x5e3b66a0464c in __interceptor_pthread_create (/root/SE-debug/SoftEtherVPN/build/vpnserver+0x8a64c) (BuildId: 1687da78342b5a1c6f3f71a831ddcfd5801fc7c9)
#1 0x7f1f52da0053 in UnixInitThread /root/SE-debug/SoftEtherVPN/src/Mayaqua/Unix.c:1685:6
#2 0x7f1f52cc29aa in NewThreadInternal /root/SE-debug/SoftEtherVPN/src/Mayaqua/Kernel.c:1200:7

Thread T25 created by T0 here:
#0 0x5e3b66a0464c in __interceptor_pthread_create (/root/SE-debug/SoftEtherVPN/build/vpnserver+0x8a64c) (BuildId: 1687da78342b5a1c6f3f71a831ddcfd5801fc7c9)
#1 0x7f1f52da0053 in UnixInitThread /root/SE-debug/SoftEtherVPN/src/Mayaqua/Unix.c:1685:6
#2 0x7f1f52cc29aa in NewThreadInternal /root/SE-debug/SoftEtherVPN/src/Mayaqua/Kernel.c:1200:7

==2440==ABORTING

chipitsine · 2024-04-27T10:45:14Z

I tried, and still the same

AddressSanitizer:DEADLYSIGNAL

================================================================= ==2440==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000002 (pc 0x7f1f52d0c98f bp 0x7f1f4efc7e10 sp 0x7f1f4efc7cf0 T29) ==2440==The signal is caused by a READ memory access. ==2440==Hint: address points to the zero page. #0 0x7f1f52d0c98f in IsMacInvalid /root/SE-debug/SoftEtherVPN/src/Mayaqua/Network.c:7490:7 #1 0x7f1f52d0c98f in IsMacUnicast /root/SE-debug/SoftEtherVPN/src/Mayaqua/Network.c:7561:6 #2 0x7f1f5328f892 in IPCIPv6AssociateOnNDTEx /root/SE-debug/SoftEtherVPN/src/Cedar/IPC.c:2185:10 #3 0x7f1f5328dc2e in IPCProcessL3EventsEx /root/SE-debug/SoftEtherVPN/src/Cedar/IPC.c #4 0x7f1f53321c78 in PPPThread /root/SE-debug/SoftEtherVPN/src/Cedar/Proto_PPP.c:371:4 #5 0x7f1f52cc20b3 in ThreadPoolProc /root/SE-debug/SoftEtherVPN/src/Mayaqua/Kernel.c:872:3 #6 0x7f1f52da44cd in UnixDefaultThreadProc /root/SE-debug/SoftEtherVPN/src/Mayaqua/Unix.c:1606:2 #7 0x7f1f52894ac2 in start_thread nptl/./nptl/pthread_create.c:442:8 #8 0x7f1f5292684f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

AddressSanitizer can not provide additional info. SUMMARY: AddressSanitizer: SEGV /root/SE-debug/SoftEtherVPN/src/Mayaqua/Network.c:7490:7 in IsMacInvalid Thread T29 created by T27 here: #0 0x5e3b66a0464c in __interceptor_pthread_create (/root/SE-debug/SoftEtherVPN/build/vpnserver+0x8a64c) (BuildId: 1687da78342b5a1c6f3f71a831ddcfd5801fc7c9) #1 0x7f1f52da0053 in UnixInitThread /root/SE-debug/SoftEtherVPN/src/Mayaqua/Unix.c:1685:6 #2 0x7f1f52cc29aa in NewThreadInternal /root/SE-debug/SoftEtherVPN/src/Mayaqua/Kernel.c:1200:7

Thread T27 created by T25 here: #0 0x5e3b66a0464c in __interceptor_pthread_create (/root/SE-debug/SoftEtherVPN/build/vpnserver+0x8a64c) (BuildId: 1687da78342b5a1c6f3f71a831ddcfd5801fc7c9) #1 0x7f1f52da0053 in UnixInitThread /root/SE-debug/SoftEtherVPN/src/Mayaqua/Unix.c:1685:6 #2 0x7f1f52cc29aa in NewThreadInternal /root/SE-debug/SoftEtherVPN/src/Mayaqua/Kernel.c:1200:7

Thread T25 created by T0 here: #0 0x5e3b66a0464c in __interceptor_pthread_create (/root/SE-debug/SoftEtherVPN/build/vpnserver+0x8a64c) (BuildId: 1687da78342b5a1c6f3f71a831ddcfd5801fc7c9) #1 0x7f1f52da0053 in UnixInitThread /root/SE-debug/SoftEtherVPN/src/Mayaqua/Unix.c:1685:6 #2 0x7f1f52cc29aa in NewThreadInternal /root/SE-debug/SoftEtherVPN/src/Mayaqua/Kernel.c:1200:7

==2440==ABORTING

really helpful!

despite it looks the same, it is not.
the first issue has gone, here's another one.

Evengard · 2024-04-27T18:22:17Z

I'll look into this, thanks

Evengard · 2024-04-27T18:58:45Z

I've updated my pr #1990 you can retest

LinearAlpha · 2024-04-27T19:35:46Z

Based on short testing, it is running without crashing, and the problem has been resolved!

When will this PR be merged into the main branch?

Evengard · 2024-04-27T19:50:16Z

I guess in a day or two, just want the latest change to be reviewed.

Fix memory access error when IPv6 prefix reading, should resolve #1972

chipitsine · 2024-04-28T15:25:56Z

thank you for cooperation!

LinearAlpha added the needs-triage label Mar 28, 2024

LinearAlpha changed the title ~~L2TP Connection issue update since two weeks ago~~ L2TP connection issues, that crashes VPN server Apr 1, 2024

chipitsine added bug protocol/L2TP and removed needs-triage labels Apr 18, 2024

Evengard added a commit that referenced this issue Apr 26, 2024

Fix memory access error when IPv6 prefix reading, should resolve #1972

c9b5e25

Evengard mentioned this issue Apr 26, 2024

Fix memory access error when IPv6 prefix reading, should resolve #1972 #1990

Merged

chipitsine closed this as completed in #1990 Apr 28, 2024

chipitsine added a commit that referenced this issue Apr 28, 2024

Merge pull request #1990 from SoftEtherVPN/fix1972

9ce27f3

Fix memory access error when IPv6 prefix reading, should resolve #1972

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

L2TP connection issues, that crashes VPN server #1972

L2TP connection issues, that crashes VPN server #1972

LinearAlpha commented Mar 28, 2024

LinearAlpha commented Apr 1, 2024 •

edited

chipitsine commented Apr 17, 2024

chipitsine commented Apr 17, 2024

LinearAlpha commented Apr 17, 2024

chipitsine commented Apr 17, 2024 via email

chipitsine commented Apr 17, 2024

chipitsine commented Apr 17, 2024

LinearAlpha commented Apr 26, 2024 •

edited

LinearAlpha commented Apr 26, 2024

LinearAlpha commented Apr 26, 2024 •

edited

chipitsine commented Apr 26, 2024

chipitsine commented Apr 26, 2024

LinearAlpha commented Apr 26, 2024

chipitsine commented Apr 27, 2024

LinearAlpha commented Apr 27, 2024

chipitsine commented Apr 27, 2024

Evengard commented Apr 27, 2024

Evengard commented Apr 27, 2024

LinearAlpha commented Apr 27, 2024

Evengard commented Apr 27, 2024

chipitsine commented Apr 28, 2024

L2TP connection issues, that crashes VPN server #1972

L2TP connection issues, that crashes VPN server #1972

Comments

LinearAlpha commented Mar 28, 2024

Are you using SoftEther VPN 5.x?

Version

Component

Operating system & version

Architecture or Hardware model

Steps to reproduce

✔️ Expected Behavior

❌ Actual Behavior

Anything else?

LinearAlpha commented Apr 1, 2024 • edited

chipitsine commented Apr 17, 2024

chipitsine commented Apr 17, 2024

LinearAlpha commented Apr 17, 2024

chipitsine commented Apr 17, 2024 via email

chipitsine commented Apr 17, 2024

chipitsine commented Apr 17, 2024

LinearAlpha commented Apr 26, 2024 • edited

LinearAlpha commented Apr 26, 2024

LinearAlpha commented Apr 26, 2024 • edited

chipitsine commented Apr 26, 2024

chipitsine commented Apr 26, 2024

LinearAlpha commented Apr 26, 2024

chipitsine commented Apr 27, 2024

LinearAlpha commented Apr 27, 2024

chipitsine commented Apr 27, 2024

Evengard commented Apr 27, 2024

Evengard commented Apr 27, 2024

LinearAlpha commented Apr 27, 2024

Evengard commented Apr 27, 2024

chipitsine commented Apr 28, 2024

LinearAlpha commented Apr 1, 2024 •

edited

LinearAlpha commented Apr 26, 2024 •

edited

LinearAlpha commented Apr 26, 2024 •

edited