Kamailio became unresponsive during runtime

[ObakeFilter]

Description

Hi, I ran into an issue where the Kamailio service seemingly froze (didn't crash, but failed to receive and deliver calls) while running, I went through the logs a little bit, and saw the same log pattern (pasted below) recouring every few months or so, on an older installation (5.2.0). Usually, it ended up automatically restarting the service. But not this time, I had to reload it manually to make the service work again.

Is this a known issue?
How can I make sure that the service will be able to restart the next time it happens?

Edward

Troubleshooting

Debugging Data

GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/sbin/kamailio...done.

warning: exec file is newer than core file.
[New LWP 11413]
Core was generated by `kamailio'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007f4e4a52e428 in ?? ()
(gdb) Quit
(gdb) bt full
#0  0x00007f4e4a52e428 in ?? ()
No symbol table info available.
#1  0x00007f4e4a53002a in ?? ()
No symbol table info available.
#2  0x0000000000000020 in ?? ()
No symbol table info available.
#3  0x0000000000000000 in ?? ()
No symbol table info available.
(gdb) info locals
No symbol table info available.
(gdb) list
1871                           & now if we don't need it */
1872    #ifdef USE_SLOW_TIMER
1873                    + 1 /* slow timer process */
1874    #endif
1875    #ifdef USE_TCP
1876                    +((!tcp_disable)?( 1/* tcp main */ + tcp_listeners ):0)
1877    #endif
1878    #ifdef USE_SCTP
1879                    +((!sctp_disable)?sctp_listeners:0)
1880    #endif
(gdb)

Log Messages

Jun 28 16:35:22 kamprodegres kamailio: INFO: <core> [core/sctp_core.c:74]: sctp_core_check_support(): SCTP API not enabled - if you want to use it, load sctp module
Jun 28 16:35:22 kamprodegres kamailio: WARNING: <core> [core/socket_info.c:1394]: fix_hostname(): could not rev. resolve XX.XX.XX.XX
Jun 28 16:35:22 kamprodegres kamailio: WARNING: <core> [core/socket_info.c:1394]: fix_hostname(): could not rev. resolve XX.XX.XX.XX
Jun 28 16:35:22 kamprodegres kamailio: INFO: <core> [core/tcp_main.c:5042]: init_tcp(): using epoll_lt as the io watch method (auto detected)
Jun 28 16:35:22 kamprodegres kamailio[8489]: INFO: jsonrpcs [jsonrpcs_sock.c:197]: jsonrpc_dgram_mod_init(): the socket /var/run/kamailio/kamailio_rpc.sock already exists, trying to delete it...
Jun 28 16:35:22 kamprodegres kamailio[8489]: INFO: rr [../outbound/api.h:52]: ob_load_api(): unable to import bind_ob - maybe module is not loaded
Jun 28 16:35:22 kamprodegres kamailio[8489]: INFO: rr [rr_mod.c:177]: mod_init(): outbound module not available
Jun 28 16:35:22 kamprodegres kamailio[8489]: INFO: <core> [main.c:2779]: main(): processes (at least): 40 - shm size: 67108864 - pkg size: 8388608
Jun 28 16:35:22 kamprodegres kamailio[8489]: INFO: <core> [core/udp_server.c:154]: probe_max_receive_buffer(): SO_RCVBUF is initially 212992
Jun 28 16:35:22 kamprodegres kamailio[8489]: INFO: <core> [core/udp_server.c:206]: probe_max_receive_buffer(): SO_RCVBUF is finally 425984
Jun 28 16:35:22 kamprodegres kamailio[8489]: INFO: <core> [core/udp_server.c:154]: probe_max_receive_buffer(): SO_RCVBUF is initially 212992
Jun 28 16:35:22 kamprodegres kamailio[8489]: INFO: <core> [core/udp_server.c:206]: probe_max_receive_buffer(): SO_RCVBUF is finally 425984
Jun 28 16:35:22 kamprodegres kamailio[8489]: INFO: <core> [core/udp_server.c:154]: probe_max_receive_buffer(): SO_RCVBUF is initially 212992
Jun 28 16:35:22 kamprodegres kamailio[8489]: INFO: <core> [core/udp_server.c:206]: probe_max_receive_buffer(): SO_RCVBUF is finally 425984
Jun 28 16:35:22 kamprodegres kamailio[8518]: INFO: jsonrpcs [jsonrpcs_sock.c:443]: jsonrpc_dgram_process(): a new child 0/8518
Jun 28 16:35:22 kamprodegres kamailio[8519]: INFO: ctl [io_listener.c:214]: io_listen_loop(): io_listen_loop:  using epoll_lt io watch method (config)
Jun 29 18:13:37 kamprodegres kamailio[8513]: INFO: {1 581461 CANCEL 402954192_49936786@67.231.13.146} tm [t_reply.c:478]: _reply_light(): can't generate 487 reply when a final 404 was sent out
Jun 29 18:27:00 kamprodegres kamailio[8508]: INFO: {1 211236 CANCEL 405019408_127896329@67.231.13.146} tm [t_reply.c:478]: _reply_light(): can't generate 487 reply when a final 404 was sent out
Jun 29 18:51:10 kamprodegres /usr/local/sbin/kamailio[2269]: NOTICE: <core> [main.c:725]: handle_sigs(): Thank you for flying kamailio!!!
Jun 29 18:51:10 kamprodegres /usr/local/sbin/kamailio[2326]: INFO: <core> [main.c:847]: sig_usr(): signal 15 received

Additional Information

• Kamailio Version - output of kamailio -v

version: kamailio 5.3.2 (x86_64/linux) 7ba545
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 7ba545
compiled on 16:37:32 Jun 14 2020 with gcc 5.4.0

• Operating System:

Ubuntu 16.04

[henningw]

Hello, you need to install kamailio debugging symbols to get meaningful information from the backtrace. From the log message it looks ok. You should first try to update to the latest 5.3 version (e.g. 5.3.5 released recently), as it might be already fixed.

[ObakeFilter]

Hello, you need to install kamailio debugging symbols to get meaningful information from the backtrace. From the log message it looks ok. You should first try to update to the latest 5.3 version (e.g. 5.3.5 released recently), as it might be already fixed.

Can I do that after the coredump has already been generated?
If so, can you point me to how can I do that?

[henningw]

Yes, look for a packaged called kamailio-dbg, kamailio-debug or similar and install it.

[ObakeFilter]

Hmm. I did install it, but I get the same output.

root@kamprodegres:/# gdb /usr/local/sbin/kamailio core
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/sbin/kamailio...done.

warning: exec file is newer than core file.
[New LWP 11413]
Core was generated by `kamailio'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f4e4a52e428 in ?? ()
(gdb) bt full
#0 0x00007f4e4a52e428 in ?? ()
No symbol table info available.
#1 0x00007f4e4a53002a in ?? ()
No symbol table info available.
#2 0x0000000000000020 in ?? ()
No symbol table info available.
#3 0x0000000000000000 in ?? ()
No symbol table info available.
(gdb) info locals
No symbol table info available.
(gdb) Quit
(gdb) quit
root@kamprodegres:/# apt-get install kamailio-dbg
Reading package lists... Done
Building dependency tree
Reading state information... Done
kamailio-dbg is already the newest version (4.3.4-1.1ubuntu2.1).
The following packages were automatically installed and are no longer required:
gyp javascript-common libjs-inherits libjs-jquery libjs-node-uuid libjs-underscore libuv1 libuv1-dev node-abbrev node-ansi node-ansi-color-table node-archy node-async node-block-stream node-combined-stream node-cookie-jar
node-delayed-stream node-forever-agent node-form-data node-fstream node-fstream-ignore node-github-url-from-git node-glob node-graceful-fs node-gyp node-inherits node-ini node-json-stringify-safe node-lockfile node-lru-cache
node-mime node-minimatch node-mkdirp node-mute-stream node-node-uuid node-nopt node-normalize-package-data node-npmlog node-once node-osenv node-qs node-read node-read-package-json node-request node-retry node-rimraf node-semver
node-sha node-sigmund node-slide node-tar node-tunnel-agent node-underscore node-which python-pkg-resources
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 442 not upgraded.
root@kamprodegres:/#

[henningw]

Ok, in this case it the backtrace information might be from somewhere else. But I would suggest to update to the lastest kamailio version first. It looks a bit like a crash during shutdown, which might be already fixed.

[ObakeFilter]

Are you aware of similar bug that has been fixed? Edward

…

On Mon, Jun 29, 2020, 23:24 Henning Westerholt ***@***.***> wrote: Ok, in this case it the backtrace information might be from somewhere else. But I would suggest to update to the lastest kamailio version first. It looks a bit like a crash during shutdown, which might be already fixed. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2380 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACM76S472YM6GDETFOBEX4LRZD2A5ANCNFSM4OLN6KAQ> .

[henningw]

This is hard to say without more details. You'll find all fixes in the Changelog: https://www.kamailio.org/pub/kamailio/5.3.5/ChangeLog
Close this one, please re-open if the bug appears again after you updated to 5.3.5.