debugger module: Kamailio crashes when shutting down

[foucse]

Hello everyone,

version: kamailio 4.4.0-dev7 (i386/linux) b8e3af-dirty
flags: STATS: Off, EXTRA_DEBUG, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: b8e3af -dirty
compiled on 10:15:37 Dec 16 2015 with gcc 4.9.3

I have a strange crash while Kamailio shuts down with debugging module active.

Here is the backtrace:

#0  0xb183dadc in dbg_get_mod_debug_level (mname=0x8362d2d "core", mnlen=4, mlevel=0xbfc3122c) at debugger_api.c:1357
#1  0x080aa0c3 in get_debug_level (mname=0x8362d2d "core", mnlen=4) at dprint.c:150
#2  0x08233eca in pkg_destroy_manager () at mem/pkg.c:88
#3  0x080d5d39 in cleanup (show_status=1) at main.c:589
#4  0x080d658e in shutdown_children (sig=15, show_status=1) at main.c:665
#5  0x080d6e33 in handle_sigs () at main.c:695
#6  0x080dfc38 in main_loop () at main.c:1722
#7  0x080e4ca1 in main (argc=17, argv=0xbfc31734) at main.c:2597

Here is the full backtrace:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0xb183dadc in dbg_get_mod_debug_level (mname=0x8362d2d "core", mnlen=4, mlevel=0xbfc3122c) at debugger_api.c:1357
1357            if(cfg_get(dbg, dbg_cfg, mod_level_mode)==0)
#0  0xb183dadc in dbg_get_mod_debug_level (mname=0x8362d2d "core", mnlen=4, mlevel=0xbfc3122c) at debugger_api.c:1357
        idx = 1
        hid = 3077953848
        it = 0xb775d000
#1  0x080aa0c3 in get_debug_level (mname=0x8362d2d "core", mnlen=4) at dprint.c:150
        mlevel = 138227952
#2  0x08233eca in pkg_destroy_manager () at mem/pkg.c:88
        __FUNCTION__ = "pkg_destroy_manager"
#3  0x080d5d39 in cleanup (show_status=1) at main.c:589
        memlog = 4
        __FUNCTION__ = "cleanup"
#4  0x080d658e in shutdown_children (sig=15, show_status=1) at main.c:665
        __FUNCTION__ = "shutdown_children"
#5  0x080d6e33 in handle_sigs () at main.c:695
        chld = 1
        chld_status = 1
        memlog = -1304824452
        __FUNCTION__ = "handle_sigs"
#6  0x080dfc38 in main_loop () at main.c:1722
        i = 4
        pid = 31542
        si = 0x0
        si_desc = "udp receiver child=3 sock=192.168.100.1:5060\000\025ÿ\227\210\322$\240\026ÿp\026ÿ\b\025ÿ\352m\025\b8G\344\266>\212\061\b\005\000\000\000\201\220\021\b\345\003\000\000d\303\063\b\006", '\000' <repeats 11 times>, "\377\377\377\377\070\025ÿ\275n\025\bp\222D\bX\320ֶ\003\000\000"
        r = 30
        nrprocs = 4
        woneinit = 1
        __FUNCTION__ = "main_loop"
#7  0x080e4ca1 in main (argc=17, argv=0xbfc31734) at main.c:2597
        cfg_stream = 0x8622008
        c = -1
        r = 0
        tmp = 0xbfc32788 ""
        tmp_len = -1219166520
        port = -1219130376
        proto = -1217065642
        options = 0x830f268 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:"
        ret = -1
        seed = 2559363386
        rfd = 4
        debug_save = 0
        debug_flag = 0
        dont_fork_cnt = 0
        n_lst = 0x0
        p = 0x8d3 <error: Cannot access memory at address 0x8d3>
        st = {st_dev = 17, __pad1 = 0, st_ino = 22917, st_mode = 16877, st_nlink = 2, st_uid = 120, st_gid = 121, st_rdev = 0, __pad2 = 0, st_size = 60, st_blksize = 4096, st_blocks = 0,
          st_atim = {tv_sec = 1450080275, tv_nsec = 852601183}, st_mtim = {tv_sec = 1450199479, tv_nsec = 415493962}, st_ctim = {tv_sec = 1450199479, tv_nsec = 415493962},
          __glibc_reserved4 = 0, __glibc_reserved5 = 0}
        __FUNCTION__ = "main"

Debugger module has the following configuration:

    modparam("debugger", "log_level"            , 1                 )
    modparam("debugger", "mod_hash_size"        , 6                 )
    modparam("debugger", "mod_level_mode"       , 1                 )
    modparam("debugger", "mod_level"            , "core=2"          )
    modparam("debugger", "mod_level"            , "db_postgres=-1"  )

No logging messages for now. I will update if I have more information.

[smititelu]

do you also have:

modparam("debugger", "mod_hash_size", 2(or any size))
modparam("debugger", "mod_level_mode", 1)

?

If no, can you try with them and see if the crash is still happening?

However, I'll have a look and try to solve the crash in this case.

[foucse]

Yes I forgot to mention those, they were in my shared configuration

[smititelu]

Managed to reproduce and fix the issue. Can you try again with the latest upstream master?

[foucse]

Hello smititelu,

version: kamailio 4.4.0-dev7 (i386/linux) 645838-dirty
flags: STATS: Off, EXTRA_DEBUG, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 645838 -dirty
compiled on 12:40:47 Dec 16 2015 with gcc 4.9.3

Ok no more crashing/core dump on shutdown with the same configuration

Thank you

[smititelu]

Thank you also for the nice description of the problem :D

[miconda]

I think this needs to be backported at least to 4.3. Not sure if 4.2 has the feature.

[smititelu]

Backported this to 4.X for the part of the feature each branch had.

[miconda]

Thanks!