[Crash] Coturn crash in server->send_turn_session_info (func report_turn_session_info)

[hangocanh2303]

When I'm using turnutils_uclient to test for coturn server. I get error in this func
Here is text log:
Program received signal SIGABRT, Aborted.
0x00007f8a325c91d7 in raise () from /lib64/libc.so.6
(gdb) bt
#0 0x00007f8a325c91d7 in raise () from /lib64/libc.so.6
#1 0x00007f8a325ca8c8 in abort () from /lib64/libc.so.6
#2 0x00007f8a32608f07 in __libc_message () from /lib64/libc.so.6
#3 0x00007f8a32611d4d in _int_malloc () from /lib64/libc.so.6
#4 0x00007f8a32613fbc in malloc () from /lib64/libc.so.6
#5 0x00007f8a336a8b48 in evbuffer_chain_new (size=1880, size@entry=1832) at buffer.c:168
#6 0x00007f8a336ab7a0 in evbuffer_add (buf=0x7f89b00033a0, data_in=0x7f8a2cb30180, datlen=1832) at buffer.c:1552
#7 0x0000000000422e4a in send_turn_session_info (tsi=0x7f8a2cb30180) at src/apps/relay/turn_admin_server.c:1468
#8 0x000000000044a58d in report_turn_session_info (server=0x7f8a2cb32050, ss=0x7f8a10014840, force_invalid=0)
at src/server/ns_turn_server.c:594
#9 0x0000000000455f2b in read_client_connection (server=0x7f8a2cb32050, ss=0x7f8a10014840, in_buffer=0x7f8a2cb45308,
can_resume=1, count_usage=1) at src/server/ns_turn_server.c:4558
#10 0x0000000000456e1c in client_input_handler (s=0x7f8a10000e60, event_type=2, data=0x7f8a2cb45308,
arg=0x7f8a10014840, can_resume=1) at src/server/ns_turn_server.c:4851
#11 0x000000000045684a in open_client_connection_session (server=0x7f8a2cb32050, sm=0x7f8a2cb45300)
at src/server/ns_turn_server.c:4706
#12 0x000000000042e9fc in handle_udp_packet (server=0x7f8a2cb44e90, sm=0x7f8a2cb452f0, ioa_eng=0x7f8a2cb32200,
ts=0x7f8a2cb32050) at src/apps/relay/dtls_listener.c:470
#13 0x000000000042f570 in udp_server_input_handler (fd=211, what=2, arg=0x7f8a2cb44e90)
at src/apps/relay/dtls_listener.c:728
#14 0x00007f8a336a50b4 in event_process_active_single_queue (activeq=0x7f8a10000d30, base=0x7f8a100008f0)
at event.c:1350
#15 event_process_active (base=) at event.c:1420
#16 event_base_loop (base=0x7f8a100008f0, flags=0) at event.c:1621
#17 0x000000000041b03d in run_events (eb=0x7f8a100008f0, e=0x7f8a2cb32200) at src/apps/relay/netengine.c:1579
#18 0x000000000041b534 in run_general_relay_thread (arg=0x7f8a2cb32010) at src/apps/relay/netengine.c:1700
#19 0x00007f8a3295cdc5 in start_thread () from /lib64/libpthread.so.0
#20 0x00007f8a3268b73d in clone () from /lib64/libc.so.6
Thank you.

[misi]

It is mem allocation issue.
Please check if you do not run out of memory. Have you enough?
Which OS and which libc version do you use?

[hangocanh2303]

@misi My OS: CentOS Linux release 7.3.1611 (Core)
Coturn version: 4.5.1.0
How can I check run out of memory in coturn ?

[misi]

Check how much free memory do you have.
See top or vmstat command outputs

[misi]

I was wrong sorry. My collegue corrected me. Malloc will not die due to out of memory.
What is your architecture? 64bit? Arm?

cat /proc/cpuinfo ?

[hangocanh2303]

Here is my server architecture:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 79
model name : Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
stepping : 1
microcode : 0xb000020
cpu MHz : 2200.147
cache size : 25600 KB
physical id : 0
siblings : 16
core id : 0
cpu cores : 16
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm xsaveopt
bogomips : 4400.18
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 79
model name : Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
stepping : 1
microcode : 0xb000020
cpu MHz : 2200.147
cache size : 25600 KB
physical id : 0
siblings : 16
core id : 1
cpu cores : 16
apicid : 2
initial apicid : 2
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm xsaveopt
bogomips : 4400.18
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:

processor : 2
vendor_id : GenuineIntel
cpu family : 6
model : 79
model name : Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
stepping : 1
microcode : 0xb000020
cpu MHz : 2200.147
cache size : 25600 KB
physical id : 0
siblings : 16
core id : 2
cpu cores : 16
apicid : 4
initial apicid : 4
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm xsaveopt
bogomips : 4400.18
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:

processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 79
model name : Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
stepping : 1
microcode : 0xb000020
cpu MHz : 2200.147
cache size : 25600 KB
physical id : 0
siblings : 16
core id : 3
cpu cores : 16
apicid : 6
initial apicid : 6
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm xsaveopt
bogomips : 4400.18
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:

processor : 4
vendor_id : GenuineIntel
cpu family : 6
model : 79
model name : Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
stepping : 1
microcode : 0xb000020
cpu MHz : 2200.147
cache size : 25600 KB
physical id : 0
siblings : 16
core id : 4
cpu cores : 16
apicid : 8
initial apicid : 8
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm xsaveopt
bogomips : 4400.18
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:

processor : 5
vendor_id : GenuineIntel
cpu family : 6
model : 79
model name : Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
stepping : 1
microcode : 0xb000020
cpu MHz : 2200.147
cache size : 25600 KB
physical id : 0
siblings : 16
core id : 5
cpu cores : 16
apicid : 10
initial apicid : 10
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm xsaveopt
bogomips : 4400.18
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:

processor : 6
vendor_id : GenuineIntel
cpu family : 6
model : 79
model name : Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
stepping : 1
microcode : 0xb000020
cpu MHz : 2200.147
cache size : 25600 KB
physical id : 0
siblings : 16
core id : 6
cpu cores : 16
apicid : 12
initial apicid : 12
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm xsaveopt
bogomips : 4400.18
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:

processor : 7
vendor_id : GenuineIntel
cpu family : 6
model : 79
model name : Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
stepping : 1
microcode : 0xb000020
cpu MHz : 2200.147
cache size : 25600 KB
physical id : 0
siblings : 16
core id : 7
cpu cores : 16
apicid : 14
initial apicid : 14
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm xsaveopt
bogomips : 4400.18
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:

processor : 8
vendor_id : GenuineIntel
cpu family : 6
model : 79
model name : Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
stepping : 1
microcode : 0xb000020
cpu MHz : 2200.147
cache size : 25600 KB
physical id : 0
siblings : 16
core id : 8
cpu cores : 16
apicid : 16
initial apicid : 16
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm xsaveopt
bogomips : 4400.18
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:

processor : 9
vendor_id : GenuineIntel
cpu family : 6
model : 79
model name : Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
stepping : 1
microcode : 0xb000020
cpu MHz : 2200.147
cache size : 25600 KB
physical id : 0
siblings : 16
core id : 9
cpu cores : 16
apicid : 18
initial apicid : 18
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm xsaveopt
bogomips : 4400.18
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:

processor : 10
vendor_id : GenuineIntel
cpu family : 6
model : 79
model name : Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
stepping : 1
microcode : 0xb000020
cpu MHz : 2200.147
cache size : 25600 KB
physical id : 0
siblings : 16
core id : 10
cpu cores : 16
apicid : 20
initial apicid : 20
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm xsaveopt
bogomips : 4400.18
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:

processor : 11
vendor_id : GenuineIntel
cpu family : 6
model : 79
model name : Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
stepping : 1
microcode : 0xb000020
cpu MHz : 2200.147
cache size : 25600 KB
physical id : 0
siblings : 16
core id : 11
cpu cores : 16
apicid : 22
initial apicid : 22
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm xsaveopt
bogomips : 4400.18
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:

processor : 12
vendor_id : GenuineIntel
cpu family : 6
model : 79
model name : Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
stepping : 1
microcode : 0xb000020
cpu MHz : 2200.147
cache size : 25600 KB
physical id : 0
siblings : 16
core id : 12
cpu cores : 16
apicid : 24
initial apicid : 24
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm xsaveopt
bogomips : 4400.18
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:

processor : 13
vendor_id : GenuineIntel
cpu family : 6
model : 79
model name : Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
stepping : 1
microcode : 0xb000020
cpu MHz : 2200.147
cache size : 25600 KB
physical id : 0
siblings : 16
core id : 13
cpu cores : 16
apicid : 26
initial apicid : 26
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm xsaveopt
bogomips : 4400.18
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:

processor : 14
vendor_id : GenuineIntel
cpu family : 6
model : 79
model name : Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
stepping : 1
microcode : 0xb000020
cpu MHz : 2200.147
cache size : 25600 KB
physical id : 0
siblings : 16
core id : 14
cpu cores : 16
apicid : 28
initial apicid : 28
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm xsaveopt
bogomips : 4400.18
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:

processor : 15
vendor_id : GenuineIntel
cpu family : 6
model : 79
model name : Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
stepping : 1
microcode : 0xb000020
cpu MHz : 2200.147
cache size : 25600 KB
physical id : 0
siblings : 16
core id : 15
cpu cores : 16
apicid : 30
initial apicid : 30
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm fsgsbase bmi1 hle avx2 smep bmi2 erms invpcid rtm xsaveopt
bogomips : 4400.18
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:

[misi]

Many thanks! So it is x86 64bit.
Can you please share the config and your parameters?
I want to reproduce your env to understand more to reproduce it.

[hangocanh2303]

thank misi
Here is my config: In turnserver.conf file, I enabled log file folder and cli password. Everything set default
When i start turn server, I used -o -v -c parameters

[misi]

I can't find your config.. So sorry I misunderstood your comment.
Many Thanks.

[misi]

For me it works fine.
I have installed centOS your version is not recent please update it!

[root@centos coturn]# cat /etc/redhat-release 
CentOS Linux release 7.6.1810 (Core)

git clone https://github.com/coturn/coturn.git

cd coturn
cd rpm
./CentOS7.pre.build.sh
./build.sh
cd  /root/rpmbuild/RPMS/x86_64/
./install.sh 

My/etc/turnserver/turnserver.conf config.

verbose
fingerprint
lt-cred-mech
user=ninefingers:youhavetoberealistic
syslog
allow-loopback-peers
pidfile="/var/run/turnserver/turnserver.pid"
cli-password=qwerty

[hangocanh2303]

Thank @misi
Problem Resolved

[misi]

Great!
Notice that allow-loopback-peers is not for production, it was only added to run tests on loopback interface.