rtpengine-recording: core dumps

[oeboema]

Hi Richard,

We're running duration tests with the combo rtpengine and rtpengine-recording. We're running about 200 concurrent calls on a virtual machine. In the last days we had 7 core dumps, all related to memory corruption. The output of GDB is add below.
We are using the source of the master branch, I've compiled the debian packages at 20-6-2023.
Can you see why this is happening?
I'll add the dump file as attachment.

Best regards,
Sjoerd.

File: core.rtpengine-recor.111.e68ed577500a4078b99515c94852ddef.26103.1688448003000000
(gdb) bt full
#0 0x000014d98634d8eb in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
set =
{__val = {18446744067266837211, 22924122456576, 956160, 22924504671565, 22924123427264, 17664845887130449408, 0, 22924500550889, 0, 17664845887130449408, 22924123427264, 22924119754944, 22924302749020, 22924500523792, 1, 22924500200019}}
pid =
tid =
#1 0x000014d986338535 in __GI_abort () at abort.c:79
save_stage = 1
act =
{__sigaction_handler = {sa_handler = 0x14d9680969c0, sa_sigaction = 0x14d9680969c0}, sa_mask = {__val = {22924503102714, 0, 17664845887130449408, 22923985906112, 17664845887130449408, 22923985847792, 22923985605952, 22923985606248, 22923985606304, 22924188155024, 22924188155008, 0, 17664845887130449408, 22923987173056, 22924302749264, 22924302749520}}, sa_flags = 2062290512, sa_restorer = 0x1000}
sigs = {__val = {32, 0 <repeats 15 times>}}
#2 0x000014d98638f648 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x14d9864992a0 "%s\n") at ../sysdeps/posix/libc_fatal.c:181
ap = {{gp_offset = 24, fp_offset = 0, overflow_arg_area = 0x14d97aec0f60, reg_save_area = 0x14d97aec0ef0}}
fd = 2
list =
nlist =
cp =
written =
#3 0x000014d986395d6a in malloc_printerr (str=str@entry=0x14d98649b010 "double free or corruption (!prev)") at malloc.c:5359
#4 0x000014d9863978ac in _int_free (av=0x14d9864d0c40 <main_arena>, p=0x14d9684e78c0, have_lock=) at malloc.c:4324
size = 95920
fb =
nextchunk = 0x14d9684fef70
nextsize =
nextinuse =
prevsize =
bck =
fwd =
PRETTY_FUNCTION = "_int_free"
#5 0x000055edc7177f93 in stream_handler (handler=) at stream.c:88
stream = 0x14d9600a9a30
buf = 0x14d9684e78d0 "\200"
#6 0x000055edc7173c69 in poller_thread (ptr=) at epoll.c:67
handler =
ret = 1
__cancel_buf =
{__cancel_jmp_buf = {{__cancel_jmp_buf = {4, -21036514804007189, 140727225714878, 140727225714879, 22924302755584, 0, -3024247695047164181, 6084025858971762411}, __mask_was_saved = 0}}, __pad = {0x14d97aec10d0, 0x0, 0x0, 0x0}}
__cancel_routine = 0x55edc71739e0 <poller_thread_end>
__not_first_call =
epev = {events = 1, data = {ptr = 0x14d9600a9a80, fd = 1611307648, u32 = 1611307648, u64 = 22923851766400}}
me_num =
#7 0x000014d9864ddfa3 in start_thread (arg=) at pthread_create.c:486
ret =
pd =
now =
unwind_buf =
{cancel_jmp_buf = {{jmp_buf = {22924302755584, -21036514804007189, 140727225714878, 140727225714879, 22924302755584, 0, -3024247695074427157, -3024642151530529045}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call =
#8 0x000014d98640f06f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

[oeboema]

core.rtpengine-recor.111.e68ed577500a4078b99515c94852ddef.26103.zip
Here's the original core dump.

[rfuchs]

Can you open the core in gdb and post the output of

frame 5
p *stream
p buf

[oeboema]

Here it is:
(gdb) frame 5
#5 0x000055edc7177f93 in stream_handler (handler=) at stream.c:88
88 stream.c: No such file or directory.
(gdb) p *stream
$1 = {lock = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0},
name = 0x14d968048870 "tag-1-media-1-component-1-RTP-id-0", metafile = 0x14d9680885f0, id = 0, tag = 1, fd = -1, handler = {func = 0x55edc7177e50 <stream_handler>, ptr = 0x14d9600a9a30}, forwarding_on = 0,
start_time = 1688447883.6719911}
(gdb) p buf
$2 = (unsigned char *) 0x14d9684e78d0 "\200"

Best regards,
Sjoerd.