Kamailio-5.4.: Crashs from keepalive module during adding destination

[buzhaodi]

Description

I just update kamailio 5.2.1 to 5.4.1. then there are some crashs from keepalive module.

Troubleshooting

Reproduction

Debugging Data

Thread 1 (Thread 0x2b4d0df90d00 (LWP 30239)):
#0  0x00000000004ef1a7 in atomic_cmpxchg_int (var=0x0, old=0, new_v=1) at core/mem/../atomic/atomic_x86.h:224
#1  0x00000000004ef21a in futex_get (lock=0x0) at core/mem/../futexlock.h:99
#2  0x00000000004f339a in timer_add_safe (tl=0x2b4d13af7928, delta=8) at core/timer.c:543
#3  0x00002b4d133b32c3 in ka_add_dest (uri=0x7ffe571d1440, owner=0x7ffe571d1430, flags=0, ping_interval=3, statechanged_clb=0x0, response_clb=0x0, user_attr=0x0) at keepalive_api.c:120
#4  0x00002b4d133b95ff in ka_mod_add_destination (type=2147483649, val=0x2b4d0ed29300) at keepalive_mod.c:244
#5  0x00000000006615b8 in set_mod_param_regex (regex=0x2b4d0ed29130 "keepalive", name=0x2b4d0ed29218 "destination", type=1, val=0x2b4d0ed29300) at core/modparam.c:125
#6  0x00000000007becec in yyparse () at core/cfg.y:1842
#7  0x000000000042f216 in main (argc=5, argv=0x7ffe571d2578) at main.c:2320
#3  0x00002b4d133b32c3 in ka_add_dest (uri=0x7ffe571d1440, owner=0x7ffe571d1430, flags=0, ping_interval=3, statechanged_clb=0x0, response_clb=0x0, user_attr=0x0) at keepalive_api.c:120
120             if(timer_add(dest->timer, MS_TO_TICKS(KA_FIRST_TRY_DELAY)) < 0){
$2 = {next = 0x0, prev = 0x0, expire = 0, initial_timeout = 0, data = 0x2b4d13af7738, f = 0x2b4d133b4fbf <ka_check_timer>, flags = 0, slow_idx = 0}
_uri = {user = {s = 0x0, len = 0}, passwd = {s = 0x0, len = 0}, host = {s = 0x2b4d13af783c "172.18.67.47", len = 12}, port = {s = 0x0, len = 0}, params = {s = 0x0, len = 0}, sip_params = {s = 0x0, len = 0},
  headers = {s = 0x0, len = 0}, port_no = 0, proto = 0, type = SIP_URI_T, flags = (unknown: 0), transport = {s = 0x0, len = 0}, ttl = {s = 0x0, len = 0}, user_param = {s = 0x0, len = 0}, maddr = {s = 0x0,
    len = 0}, method = {s = 0x0, len = 0}, lr = {s = 0x0, len = 0}, r2 = {s = 0x0, len = 0}, gr = {s = 0x0, len = 0}, transport_val = {s = 0x0, len = 0}, ttl_val = {s = 0x0, len = 0}, user_param_val = {
    s = 0x0, len = 0}, maddr_val = {s = 0x0, len = 0}, method_val = {s = 0x0, len = 0}, lr_val = {s = 0x0, len = 0}, r2_val = {s = 0x0, len = 0}, gr_val = {s = 0x0, len = 0}}
dest = 0x2b4d13af7738
hollow = 0x0
__FUNCTION__ = "ka_add_dest"
#2  0x00000000004f339a in timer_add_safe (tl=0x2b4d13af7928, delta=8) at core/timer.c:543
543             LOCK_TIMER_LIST();
ret = 0
__FUNCTION__ = "timer_add_safe"

Log Messages

Sep 14 15:49:34 localhost kamailio: INFO: pv [pv_shv.c:60]: shvar_init_locks(): locks array size 16
Sep 14 15:49:34 localhost kernel: kamailio[30186]: segfault at 0 ip 00000000004ef1a7 sp 00007ffdd2db9170 error 6 in kamailio[400000+4ae000]

SIP Traffic

(paste your sip traffic here)

Possible Solutions

Additional Information

• Kamailio Version - output of kamailio -v

version: kamailio 5.4.1 (x86_64/linux) 99b2a0
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 99b2a0 
compiled on 10:28:36 Sep 14 2020 with gcc 4.8.5

• Operating System:

Linux version 3.10.0-327.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Thu Nov 19 22:10:57 UTC 2015

[henningw]

Hello, thanks for the report. You might be interested in this #2448 bug, it deals with (probably different) bugs of this particular module as well.

[miconda]

This looks like adding the destination into timer list before the timer process has been initialized. Seems to happen in the modparam function callback and likely the timer is created after config parsing to get the values for real-time parameters (http://kamailio.org/wiki/cookbooks/5.4.x/core#real-time_parameters).

The approach should be to create the list of the items via the modparam callback and then add them to the timer in mod_init. This has to be done in the c code. Maybe @NGSegovia can check if my assumptions are correct, he seems to be familiar with the module

An workaround for now is to load htable module, define event_route[htable:mod-init] and there add the destinations for keepalive with the function.

[NGSegovia]

I think you are right @miconda, and I think your solution is right too.

[buzhaodi]

Thanks everyone，So will there be a patch? or use define event_route[htable:mod-init] ? @NGSegovia

[NGSegovia]

Yeah, that will be patched. Just need some time to work on that @buzhaodi .

[NGSegovia]

Hi @buzhaodi! Sorry for the delay. I have been quite busy these weeks.
I opened #2579 fixing this problem.

If you can take a look and let me know if looks good for you would be great.

Regards!

[buzhaodi]

@NGSegovia thanks for this,i will test in next week(because today is Friday in china:smile:),and report you.

[buzhaodi]

i have beet test it. it work's.thanks