Recording crash

[nicoonline]

Hello,

The RTP Recording crash after 20 minutes of operation :)

I get this log in the syslog 👍
proxy1 kernel: rtpengine-recor[24317] general protection ip:3ee165eac4 sp:7fd291b55b00 error:0 in libglib-2.0.so.0.2800.8[3ee1600000+115000]

Can you help me to find the dump file and extract root cause ?

Thanks

[nicoonline]

seems the error isn't always the same..

Nov 25 09:25:55 proxy1 kernel: rtpengine-recor[24317] general protection ip:3ee165eac4 sp:7fd291b55b00 error:0 in libglib-2.0.so.0.2800.8[3ee1600000+115000]

Nov 25 10:24:04 proxy1 kernel: rtpengine-recor[26825]: segfault at 6ebc ip 000000000040d085 sp 00007fa84cfeeb80 error 4 in rtpengine-recording[400000+16000]

Nov 25 10:43:26 proxy1 kernel: rtpengine-recor[29242] general protection ip:3edf27b93c sp:7f55af1cbba8 error:0 in libc-2.12.so[3edf200000+18b000]

[rfuchs]

Open the core dump in gdb and post the output of bt full

If you're using a Debian packaged build, make sure the debug package is installed first.

[nicoonline]

can you refresh me where is the dump file ?

[rfuchs]

Really depends on your system, under systemd try /var/lib/systemd/coredump/
Otherwise the local directory where the daemon was started from

[nicoonline]

centos 6 and i don't find any coredump directory on the system.
the daemon was started in /usr/sbin/

[rfuchs]

Can't help with CentOS unfortunately

[rfuchs]

Syntax to start gdb is gdb /usr/sbin/rtpengine-recording /tmp/core.13822

[nicoonline]

Core was generated by `/usr/sbin/rtpengine-recording --config-file=/etc/rtpengine-recording.conf --pid'.
Program terminated with signal 11, Segmentation fault.
#0 0x0000003ee165eac4 in g_slice_alloc () from /lib64/libglib-2.0.so.0

[rfuchs]

And the rest of the backtrace?

Also which version is this?

[nicoonline]

#0 0x0000003ee165eac4 in g_slice_alloc () from /lib64/libglib-2.0.so.0
No symbol table info available.
#1 0x0000003ee165f186 in g_slice_alloc0 () from /lib64/libglib-2.0.so.0
No symbol table info available.
#2 0x00000000004074f1 in packet_process (stream=0x7faedc00d800, buf=0x7faedc14eb70 "E\240", len=200) at packet.c:133
packet =
bufstr = {s = 0x1001f <Address 0x1001f out of bounds>, len = -16}
ssrc_num =
ssrc =
#3 0x0000000000407153 in stream_handler (handler=) at stream.c:79
stream = 0x7faedc00d800
buf = 0x7faedc14eb70 "E\240"
ret = 200
#4 0x0000000000404ed6 in poller_thread (ptr=) at epoll.c:65
handler =
ret = 1
__cancel_buf = {__cancel_jmp_buf = {{__cancel_jmp_buf = {0, -6688637537828829702, 270037795680, 140389406542272, 0, 3, 6661042587786184186, -6688638089388849670}, __mask_was_saved = 0}}, __pad = {0x7faef4b4fdb0, 0x0, 0x0, 0x0}}
__cancel_routine = 0x404d30 <poller_thread_end>
__cancel_arg = 0x0
not_first_call =
epev = {events = 1, data = {ptr = 0x7faedc00d848, fd = -603924408, u32 = 3691042888, u64 = 140388992079944}}
me_num = 3
#5 0x0000003edf607aa1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#6 0x0000003edf2e8c4d in clone () from /lib64/libc.so.6

[root@proxy1 tmp]# /usr/sbin/rtpengine-recording -v
Version: git-mr6.5.5-44828a6

[rfuchs]

Looks like some sort of memory corruption issue. Are you able to run this through valgrind? Something like

G_SLICE=always-malloc valgrind /usr/sbin/rtpengine-recording --config-file=/etc/rtpengine-recording.conf --.... remaining options

[nicoonline]

==20322== Memcheck, a memory error detector
==20322== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al.
==20322== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info
==20322== Command: /usr/sbin/rtpengine-recording --config-file=/etc/rtpengine-recording.conf
==20322==
==20322==
==20322== HEAP SUMMARY:
==20322== in use at exit: 57,598 bytes in 1,122 blocks
==20322== total heap usage: 1,743 allocs, 621 frees, 197,678 bytes allocated
==20322==
==20322== LEAK SUMMARY:
==20322== definitely lost: 1,420 bytes in 5 blocks
==20322== indirectly lost: 2,675 bytes in 39 blocks
==20322== possibly lost: 232 bytes in 2 blocks
==20322== still reachable: 53,271 bytes in 1,076 blocks
==20322== suppressed: 0 bytes in 0 blocks
==20322== Rerun with --leak-check=full to see details of leaked memory
==20322==
==20322== For counts of detected and suppressed errors, rerun with: -v
==20322== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 8 from 6)

[rfuchs]

Ah you have to add --foreground to the options

[nicoonline]

valgrind --leak-check=full /usr/sbin/rtpengine-recording --config-file=/etc/rtpengine-recording.conf
ok launched.

[rfuchs]

Don't really need the leak check, just keep an eye on reported errors like invalid reads or writes.

[nicoonline]

Okay but i have to output on a file ?

[rfuchs]

Depends on how much output you get. If it's too much for your terminal, you can write to a file with --log-file=...

[nicoonline]

strange no crash anymore when launched with G_SLICE=always-malloc valgrind /usr/sbin/rtpengine-recording ...

[rfuchs]

Maybe an issue with the glib slice allocator then. Try again with G_SLICE=always-malloc but without valgrind. If that still fixes it, then it might be a problem within glib. Which version of the glib2 package do you have?

[nicoonline]

glib2.x86_64 2.28.8-10.el6

[nicoonline]

I have added the G_SLICE in init script and since then no crash anymore....

start() {
echo -n $"Starting $prog: "
if [[ -n "$RE_USER" ]]
then
# shellcheck disable=SC2086
daemon --user "$RE_USER" --pidfile="${pidfile}" "$runfile" $OPTS
else
# shellcheck disable=SC2086
daemon --pidfile="${pidfile}" "G_SLICE=always-malloc $runfile" $OPTS
fi

[rfuchs]

Might be a problem with your glib then. Try looking for an updated package