Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ims_ipsec_pcscf crashes after the call #2970

Closed
Dgeka25594 opened this issue Dec 10, 2021 · 20 comments
Closed

ims_ipsec_pcscf crashes after the call #2970

Dgeka25594 opened this issue Dec 10, 2021 · 20 comments

Comments

@Dgeka25594
Copy link

Description

Hi, I am a junior with Kamailio, if I show you where I am wrong, I will. Often during a call between two sip clients (located on the same subnet with pcscf) Kamailio P-CSCF crashes with a core dump. But sometimes the call goes through normally, I can’t see why this is happening.

Troubleshooting

Reproduction

It is reproduced often, the error is most likely somewhere with me.

Debugging Data

GNU gdb (Debian 10.1-1.7) 10.1.90.20210103-git
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "i686-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/kamailio...
(No debugging symbols found in /usr/sbin/kamailio)

warning: Can't open file /dev/zero (deleted) during file-backed mapping note processing
[New LWP 14196]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0xb6f1eab9 in ?? () from /usr/lib/i386-linux-gnu/kamailio/modules/ims_ipsec_pcscf.so
(gdb) bt full
#0  0xb6f1eab9 in ?? () from /usr/lib/i386-linux-gnu/kamailio/modules/ims_ipsec_pcscf.so
No symbol table info available.
#1  0xb6f25b1e in ipsec_forward () from /usr/lib/i386-linux-gnu/kamailio/modules/ims_ipsec_pcscf.so
No symbol table info available.
#2  0xb6f2901a in ?? () from /usr/lib/i386-linux-gnu/kamailio/modules/ims_ipsec_pcscf.so
No symbol table info available.
#3  0x004bc49d in do_action ()
No symbol table info available.
#4  0x004bae10 in run_actions ()
No symbol table info available.
#5  0x004ca3a0 in run_top_route ()
No symbol table info available.
#6  0xb73c1695 in reply_received () from /usr/lib/i386-linux-gnu/kamailio/modules/tm.so
No symbol table info available.
#7  0x0052fa14 in ?? ()
No symbol table info available.
#8  0x005b9170 in receive_msg ()
No symbol table info available.
#9  0x006bb830 in udp_rcv_loop ()
No symbol table info available.
#10 0x004b74e8 in main_loop ()
No symbol table info available.
#11 0x004ab0f1 in main ()
No symbol table info available.

Log Messages

Dec 10 15:33:07 pcscf /usr/sbin/kamailio[14194]: ERROR: rtpengine [rtpengine.c:2957]: select_rtpp_set():  script error-invalid id_set to be selected
Dec 10 15:33:07 pcscf /usr/sbin/kamailio[14194]: ERROR: rtpengine [rtpengine.c:3236]: set_rtpengine_set_from_avp(): could not locate engine set 2
Dec 10 15:35:14 pcscf /usr/sbin/kamailio[14204]: ERROR: <script>: INVITE (sip:bob@ims.mnc001.mcc001.3gppnetwork.org (192.168.56.1:55149) to sip:alice@ims.mnc001.mcc001.3gppnetwork.org, dca98f98-8bb9-c01d-ff0c-e55794e9bcaa)
Dec 10 15:35:14 pcscf /usr/sbin/kamailio[14204]: ERROR: ims_ipsec_pcscf [cmd.c:806]: ipsec_forward(): Contact doesn't exist
Dec 10 15:35:14 pcscf /usr/sbin/kamailio[14204]: ERROR: ims_ipsec_pcscf [cmd.c:806]: ipsec_forward(): Contact doesn't exist
Dec 10 15:35:14 pcscf /usr/sbin/kamailio[14193]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact
Dec 10 15:35:14 pcscf /usr/sbin/kamailio[14195]: ERROR: <script>: INVITE (sip:bob@ims.mnc001.mcc001.3gppnetwork.org (192.168.56.106:6060) to sip:alice@ims.mnc001.mcc001.3gppnetwork.org, dca98f98-8bb9-c01d-ff0c-e55794e9bcaa)
Dec 10 15:35:14 pcscf /usr/sbin/kamailio[14195]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact
Dec 10 15:35:14 pcscf /usr/sbin/kamailio[14195]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact
Dec 10 15:35:14 pcscf /usr/sbin/kamailio[14195]: ERROR: rtpengine [rtpengine.c:2957]: select_rtpp_set():  script error-invalid id_set to be selected
Dec 10 15:35:14 pcscf /usr/sbin/kamailio[14195]: ERROR: rtpengine [rtpengine.c:3236]: set_rtpengine_set_from_avp(): could not locate engine set 2
Dec 10 15:35:14 pcscf /usr/sbin/kamailio[14205]: ERROR: rtpengine [rtpengine.c:2957]: select_rtpp_set():  script error-invalid id_set to be selected
Dec 10 15:35:14 pcscf /usr/sbin/kamailio[14205]: ERROR: rtpengine [rtpengine.c:3236]: set_rtpengine_set_from_avp(): could not locate engine set 2
Dec 10 15:35:14 pcscf /usr/sbin/kamailio[14210]: ERROR: ims_ipsec_pcscf [cmd.c:252]: fill_contact(): Reply No contact headers
Dec 10 15:35:14 pcscf /usr/sbin/kamailio[14210]: ERROR: ims_ipsec_pcscf [cmd.c:799]: ipsec_forward(): Error filling in contact data
Dec 10 15:35:14 pcscf /usr/sbin/kamailio[14201]: ERROR: rtpengine [rtpengine.c:2957]: select_rtpp_set():  script error-invalid id_set to be selected
Dec 10 15:35:14 pcscf /usr/sbin/kamailio[14201]: ERROR: rtpengine [rtpengine.c:3236]: set_rtpengine_set_from_avp(): could not locate engine set 2
Dec 10 15:35:14 pcscf /usr/sbin/kamailio[14274]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 22
Dec 10 15:35:14 pcscf /usr/sbin/kamailio[14191]: ALERT: <core> [main.c:782]: handle_sigs(): child process 14196 exited by a signal 11
Dec 10 15:35:14 pcscf /usr/sbin/kamailio[14191]: ALERT: <core> [main.c:785]: handle_sigs(): core was generated
Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14324]: WARNING: <core> [core/daemonize.c:596]: mem_lock_pages(): failed to lock the memory pages (disable swap): Cannot allocate memory [12]
Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14324]: ERROR: rtpengine [rtpengine.c:2957]: select_rtpp_set():  script error-invalid id_set to be selected
Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14324]: WARNING: tm [tm.c:521]: fixup_routes(): t_on_failure("NATMANAGE"): empty/non existing route
Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14324]: ERROR: <script>: event_route[htable:mod-init] {
Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14324]: ERROR: ims_ipsec_pcscf [cmd.c:950]: ipsec_destroy(): Contact doesn't exist
Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14400]: WARNING: ims_usrloc_pcscf [usrloc_db.c:67]: connect_db(): DB connection already open... continuing
Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14406]: WARNING: ims_usrloc_pcscf [usrloc_db.c:67]: connect_db(): DB connection already open... continuing
Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14403]: WARNING: ims_usrloc_pcscf [usrloc_db.c:67]: connect_db(): DB connection already open... continuing
Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14405]: WARNING: ims_usrloc_pcscf [usrloc_db.c:67]: connect_db(): DB connection already open... continuing
Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14401]: WARNING: ims_usrloc_pcscf [usrloc_db.c:67]: connect_db(): DB connection already open... continuing
Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14404]: WARNING: ims_usrloc_pcscf [usrloc_db.c:67]: connect_db(): DB connection already open... continuing
Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14402]: WARNING: ims_usrloc_pcscf [usrloc_db.c:67]: connect_db(): DB connection already open... continuing
Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14399]: WARNING: ims_usrloc_pcscf [usrloc_db.c:67]: connect_db(): DB connection already open... continuing
Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14396]: WARNING: ims_usrloc_pcscf [usrloc_db.c:67]: connect_db(): DB connection already open... continuing
Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14334]: ERROR: <script>: PRACK (sip:bob@ims.mnc001.mcc001.3gppnetwork.org (192.168.56.1:55149) to sip:alice@ims.mnc001.mcc001.3gppnetwork.org, dca98f98-8bb9-c01d-ff0c-e55794e9bcaa)
Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14334]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact
Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14334]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact
Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14337]: ERROR: <script>: PRACK (sip:bob@ims.mnc001.mcc001.3gppnetwork.org (192.168.56.106:6060) to sip:alice@ims.mnc001.mcc001.3gppnetwork.org, dca98f98-8bb9-c01d-ff0c-e55794e9bcaa)
Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14337]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact
Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14337]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact
Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14329]: ERROR: <script>: PRACK (sip:bob@ims.mnc001.mcc001.3gppnetwork.org (192.168.56.1:55149) to sip:alice@ims.mnc001.mcc001.3gppnetwork.org, dca98f98-8bb9-c01d-ff0c-e55794e9bcaa)
Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14329]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact
Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14329]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact
Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14341]: ERROR: <script>: PRACK (sip:bob@ims.mnc001.mcc001.3gppnetwork.org (192.168.56.106:6060) to sip:alice@ims.mnc001.mcc001.3gppnetwork.org, dca98f98-8bb9-c01d-ff0c-e55794e9bcaa)
Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14341]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact
Dec 10 15:35:15 pcscf /usr/sbin/kamailio[14341]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact
Dec 10 15:35:17 pcscf /usr/sbin/kamailio[14328]: ERROR: <script>: ACK (sip:bob@ims.mnc001.mcc001.3gppnetwork.org (192.168.56.1:55149) to sip:alice@ims.mnc001.mcc001.3gppnetwork.org, dca98f98-8bb9-c01d-ff0c-e55794e9bcaa)
Dec 10 15:35:17 pcscf /usr/sbin/kamailio[14328]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact
Dec 10 15:35:17 pcscf /usr/sbin/kamailio[14328]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact
Dec 10 15:35:17 pcscf /usr/sbin/kamailio[14326]: ERROR: <script>: ACK (sip:bob@ims.mnc001.mcc001.3gppnetwork.org (192.168.56.106:6060) to sip:alice@ims.mnc001.mcc001.3gppnetwork.org, dca98f98-8bb9-c01d-ff0c-e55794e9bcaa)
Dec 10 15:35:17 pcscf /usr/sbin/kamailio[14326]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact
Dec 10 15:35:17 pcscf /usr/sbin/kamailio[14326]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact
Dec 10 15:35:26 pcscf /usr/sbin/kamailio[14327]: ERROR: <script>: BYE (sip:bob@ims.mnc001.mcc001.3gppnetwork.org (192.168.56.1:55149) to sip:alice@ims.mnc001.mcc001.3gppnetwork.org, dca98f98-8bb9-c01d-ff0c-e55794e9bcaa)
Dec 10 15:35:26 pcscf /usr/sbin/kamailio[14327]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact
Dec 10 15:35:26 pcscf /usr/sbin/kamailio[14327]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact
Dec 10 15:35:26 pcscf /usr/sbin/kamailio[14325]: ERROR: <script>: BYE (sip:bob@ims.mnc001.mcc001.3gppnetwork.org (192.168.56.106:6060) to sip:alice@ims.mnc001.mcc001.3gppnetwork.org, dca98f98-8bb9-c01d-ff0c-e55794e9bcaa)
Dec 10 15:35:26 pcscf /usr/sbin/kamailio[14325]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact
Dec 10 15:35:26 pcscf /usr/sbin/kamailio[14325]: ERROR: ims_ipsec_pcscf [cmd.c:816]: ipsec_forward(): No security parameters found in contact
Dec 10 15:35:26 pcscf /usr/sbin/kamailio[14325]: ERROR: rtpengine [rtpengine.c:2957]: select_rtpp_set():  script error-invalid id_set to be selected
Dec 10 15:35:26 pcscf /usr/sbin/kamailio[14325]: ERROR: rtpengine [rtpengine.c:3236]: set_rtpengine_set_from_avp(): could not locate engine set 2
Dec 10 15:35:30 pcscf /usr/sbin/kamailio[14396]: ERROR: <script>: Preloading NAT-PING. Rows: 921
Dec 10 15:35:30 pcscf /usr/sbin/kamailio[14396]: ERROR: <script>: OPTIONS to sip:alice@192.168.56.107:57838;transport=udp via sip:192.168.56.107:57838...

SIP Traffic

(paste your sip traffic here)



pcscf_dump.zip

Possible Solutions

Additional Information

  • Kamailio Version - output of kamailio -v
version: kamailio 5.4.4 (i386/linux) 
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown 
compiled with gcc 10.2.1
  • Operating System:
Linux pcscf.ims.mnc001.mcc001.3gppnetwork.org 5.10.0-9-686-pae #1 SMP Debian 5.10.70-1 (2021-09-30) i686 GNU/Linux

kamailio_cfg.zip
kamailio_log.zip

@miconda miconda changed the title Kamailio crashes after the call ims_ipsec_pcscf crashes after the call Dec 13, 2021
@Dgeka25594
Copy link
Author

Colleagues, there is an opportunity to give a hint why it crashes ims_ipsec_pcscf ?

@henningw
Copy link
Contributor

Thanks for the report, can you install maybe kamailio debug symbols (e.g. from packages) and do the backtrace again? The initial one is missing any detailed information.

@Dgeka25594
Copy link
Author

Thanks for the answer! Did I understand correctly that I need to install "kamailio-dbg"? I installed Kamailio via "apt install" from the repository.

@henningw
Copy link
Contributor

kamailio-dbg should do the trick

@Dgeka25594
Copy link
Author

kamailio-dbg должен помочь

I apologize, but the packages cannot install kamailio-dbg on Debian, the repositories do not have any kamailio-dbg, kamailio-debbug, kamailio-dbg. Maybe I need to include from the debbig module parameters https://www.kamailio.org/docs/modules/devel/modules/debugger.html?

@henningw
Copy link
Contributor

At least in the kamailio debian repository they are there:

root@sbct1: # apt-cache policy kamailio-dbg
kamailio-dbg:
Installiert: (keine)
Installationskandidat: 5.5.3+bpo11
Versionstabelle:
5.5.3+bpo11 500
500 http://deb.kamailio.org/kamailio55 bullseye/main amd64 Packages

@kamailio-sync
Copy link

kamailio-sync commented Dec 28, 2021 via email

@Dgeka25594
Copy link
Author

Thanks for the tips. The old core is gone, but the new one.

GNU gdb (Debian 10.1-1.7) 10.1.90.20210103-git
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "i686-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /sbin/kamailio...
Reading symbols from /usr/lib/debug/.build-id/f0/69ca129a40c5415b1162a2672ca9a1eb6443a0.debug...

warning: Can't open file /dev/zero (deleted) during file-backed mapping note processing
[New LWP 32251]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg'.
Program terminated with signal SIGABRT, Aborted.
#0  0xb7fcb559 in __kernel_vsyscall ()
(gdb) bt full
#0  0xb7fcb559 in __kernel_vsyscall ()
No symbol table info available.
#1  0xb7cc2e02 in __libc_signal_restore_set (set=0xbfb295ec) at ../sysdeps/unix/sysv/linux/internal-signals.h:86
        resultvar = <optimized out>
#2  __GI_raise (sig=6) at ../sysdeps/unix/sysv/linux/raise.c:48
        set = {__val = {8192, 0, 0, 2021, 0, 0, 3083523171, 400, 0, 3216152096, 3085381632, 3085383488, 79848, 1, 0, 1, 23021476, 23021468, 15, 23021468, 1844847104, 3216152452, 0, 3083925755, 
            3085383488, 23021448, 3086841612, 3083491989, 5012815, 3085381632, 0, 3216152352}}
        pid = <optimized out>
        tid = <optimized out>
        ret = 0
#3  0xb7cab306 in __GI_abort () at abort.c:79
        save_stage = <optimized out>
        act = {__sigaction_handler = {sa_handler = 0xb7fc5c70, sa_sigaction = 0xb7fc5c70}, sa_mask = {__val = {0, 10800, 22903096, 0, 0, 0, 0, 0, 4979331, 1, 0, 9883664, 3086866326, 4979331, 
              3086977408, 3216152596, 3086977856, 3086770192, 1, 1, 0, 0, 9883648, 3086974976, 4927488, 3216152744, 3084441447, 130, 3083435384, 3216152632, 0, 3086866153}}, sa_flags = 9883648, 
          sa_restorer = 0x897078}
        sigs = {__val = {32, 0 <repeats 31 times>}}
#4  0x004d14cf in sig_alarm_abort (signo=14) at main.c:699
        __func__ = "sig_alarm_abort"
#5  <signal handler called>
No symbol table info available.
#6  0xb7fcb557 in __kernel_vsyscall ()
No symbol table info available.
#7  0xb7d8da07 in syscall () at ../sysdeps/unix/sysv/linux/i386/syscall.S:29
No locals.
#8  0xb6f3f936 in futex_get (lock=0xb08d3b6c) at ../../core/mem/../futexlock.h:121
        v = 1
        i = -1225347072
#9  0xb6f41668 in lock_ulslot (_d=0xb08e5b88, i=206) at udomain.c:295
No locals.
#10 0xb6f40be1 in free_udomain (_d=0xb08e5b88) at udomain.c:156
        i = 206
        __func__ = "free_udomain"
#11 0xb6f36508 in free_all_udomains () at dlist.c:296
        ptr = 0xb08e5b04
        __func__ = "free_all_udomains"
#12 0xb6f39c69 in destroy () at ims_usrloc_pcscf_mod.c:253
No locals.
#13 0x006d3660 in destroy_modules () at core/sr_module.c:839
        t = 0xb75253a4
        foo = 0xb7524d3c
        __func__ = "destroy_modules"
#14 0x004d002b in cleanup (show_status=1) at main.c:575
        memlog = 0
        __func__ = "cleanup"
#15 0x004d182b in shutdown_children (sig=15, show_status=1) at main.c:718
        __func__ = "shutdown_children"
#16 0x004d4612 in handle_sigs () at main.c:816
        chld = 0
        chld_status = 139
        any_chld_stopped = 1
        memlog = 5041487
        __func__ = "handle_sigs"
#17 0x004e0af6 in main_loop () at main.c:1903
--Type <RET> for more, q to quit, c to continue without paging--
        i = 16
        pid = 32331
        si = 0x0
        si_desc = "udp receiver child=15 sock=0.0.0.0:5064\000\060:5060\000\277\000Ж\000\371}\000\000\254\245\262\277裲\277\a\267e\000\210\234V\267\f\000\000\000\060\232Q\267ζe\000\000\000\000\000\226\301\215\000\005\000\000\000\000\000\000\000\334uTue De\030\244\262\277\365\267e\000\344(\232\000\332\vN\000\062\061\n"
        nrprocs = 16
        woneinit = 1
        __func__ = "main_loop"
#18 0x004e9bd0 in main (argc=9, argv=0xbfb2a884) at main.c:3053
        cfg_stream = 0x15101c0
        c = -1
        r = 0
        tmp = 0xbfb2ae7a ""
        tmp_len = 1
        port = 0
        proto = -1209441992
        ahost = 0x0
        aport = 0
        options = 0x899ef0 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:"
        ret = -1
        seed = 2727312077
        rfd = 4
        debug_save = 0
        debug_flag = 0
        dont_fork_cnt = 0
        n_lst = 0xbfb2a67f
        p = 0x1 <error: Cannot access memory at address 0x1>
        st = {st_dev = 22, __pad1 = 0, st_ino = 931, st_mode = 16888, st_nlink = 2, st_uid = 111, st_gid = 115, st_rdev = 0, __pad2 = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, 
          st_atim = {tv_sec = 1640709398, tv_nsec = 701244783}, st_mtim = {tv_sec = 1640709398, tv_nsec = 701244783}, st_ctim = {tv_sec = 1640709398, tv_nsec = 701244783}, __glibc_reserved4 = 0, 
          __glibc_reserved5 = 0}
        tbuf = "\000\000\000\000\000\000\000\000\004\000\000\020\000\360\377\377Linux", '\000' <repeats 60 times>, "pcscf.ims.mnc001.mcc001.3gppnetwork.org", '\000' <repeats 26 times>, "\065.10.0-9-686-pae", '\000' <repeats 22 times>, "\fw\375\267\225ZʷO}L", '\000' <repeats 16 times>...
        option_index = 0
        long_options = {{name = 0x89c26a "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x8974d6 "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x89c26f "alias", has_arg = 1, 
            flag = 0x0, val = 1024}, {name = 0x89c275 "subst", has_arg = 1, flag = 0x0, val = 1025}, {name = 0x89c27b "substdef", has_arg = 1, flag = 0x0, val = 1026}, {
            name = 0x89c284 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {name = 0x89c28e "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x89c298 "loadmodule", has_arg = 1, 
            flag = 0x0, val = 1029}, {name = 0x89c2a3 "modparam", has_arg = 1, flag = 0x0, val = 1030}, {name = 0x89c2ac "log-engine", has_arg = 1, flag = 0x0, val = 1031}, {
            name = 0x89c2b7 "debug", has_arg = 1, flag = 0x0, val = 1032}, {name = 0x89c2bd "cfg-print", has_arg = 0, flag = 0x0, val = 1033}, {name = 0x89c2c7 "atexit", has_arg = 1, flag = 0x0, 
            val = 1034}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}}
        __func__ = "main"

@Dgeka25594
Copy link
Author

Updated Kamailio to:

version: kamailio 5.5.3 (i386/linux) 
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown 
compiled with gcc 10.2.1

@henningw
Copy link
Contributor

It looks like the second core dump back trace was from another child that stopped because of an error in another child. Do you have only this core dump, or are there others? If more, try to find the one with the process ID from the first log file output related to the crash.

@Dgeka25594
Copy link
Author

Sorry, wrong core

root@pcscf:/usr/local/src/kamailio# gdb /sbin/kamailio /tmp/core-kamailio-11-111-115-32593-1640711154
GNU gdb (Debian 10.1-1.7) 10.1.90.20210103-git
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "i686-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /sbin/kamailio...
Reading symbols from /usr/lib/debug/.build-id/f0/69ca129a40c5415b1162a2672ca9a1eb6443a0.debug...

warning: Can't open file /dev/zero (deleted) during file-backed mapping note processing
[New LWP 32593]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/kamailio -P /run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.cfg'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0xb6f06afa in fill_contact (ci=0xbfd2249c, m=0xb7555b1c) at cmd.c:271

warning: Source file is more recent than executable.
271	    ci->received_host.len = ip_addr2sbuf(&req->rcv.src_ip, srcip, 50);
(gdb) bt full
#0  0xb6f06afa in fill_contact (ci=0xbfd2249c, m=0xb7555b1c) at cmd.c:271
        t = 0xb092ef0c
        cb = 0xb75a9d54
        vb = 0xb75cc99c
        req = 0xb09308e8
        srcip = 0x0
        __func__ = "fill_contact"
#1  0xb6f107e3 in ipsec_forward (m=0xb7555b1c, d=0xb08e6b88, _cflags=0) at cmd.c:799
        ci = {searchflag = 0, extra_search_criteria = 0, aor = {s = 0x0, len = 0}, via_host = {
            s = 0x9bf4e6 <buf+454> "192.168.56.1:64334;rport=64334;received=192.168.56.1;branch=z9hG4bK527528804\r\nRecord-Route: <sip:mt@192.168.56.120;lr=on;ftag=527524296;rm=7>\r\nRecord-Route: <sip:mt@192.168.56.106:6060;lr=on;ftag=5275"..., len = 12}, via_port = 64334, via_prot = 1, received_host = {s = 0x0, len = 0}, received_port = 0, received_proto = 0, path = 0x0, expires = 0, 
          callid = 0x0, public_ids = 0x0, num_public_ids = 0, service_routes = 0x0, num_service_routes = 0, rx_regsession_id = 0x0, reg_state = PCONTACT_ANY}
        pcontact = 0x0
        ret = -1
        dst_proto = 1 '\001'
        dst_port = 0
        src_port = 0
        via_host = {af = 3075613068, len = 3075665560, u = {addrl = {6627768, 3076040100, 3084758229, 9719808}, addr32 = {6627768, 3076040100, 3084758229, 9719808}, addr16 = {8632, 101, 42404, 
              46936, 44245, 47069, 20480, 148}, addr = "\270!e\000\244\245X\267լݷ\000P\224"}}
        req = 0xb09308e8
        __func__ = "ipsec_forward"
        s = 0x0
        buf = "\345\211M\000\024'ҿxx\\\267\034[U\267\240\200M\000\027\264\260\364h/\362\313\000\000\000\000p'ҿ\020\300H\267\320%ҿ\004`[\267\067WH\267\000\000\000\000\034[U\267\220\230\\\267\000#c\027\000\000\200\000l\270j\000\064\031\017\000\000P\224\000<8ҿ\024*ҿ\030'ҿB\"e\000\364\063ҿ\034[U\267p'ҿ\244\225T\267\000P\224\000\034[U\267xx\\\267\024'ҿ\000P\224\000\000\000\000\000\377\377\377\377(\354H\267\311\000\000\000\000P\224\000\350&ҿ", '\000' <repeats 16 times>, "h\332d\000\364\063ҿ\220\241T\267\034[U\267N\331d\000\001\000\000\000\000\000\000\000\370&ҿ\000\000\000\000"...
        buf_len = 0
        client_sock = 0x0
        dst_info = {send_sock = 0x5f1b92 <pv_value_destroy+187>, to = {s = {sa_family = 9620, sa_data = "ҿ\000\000\000\000\020\000\000\000\000#c\027"}, sin = {sin_family = 9620, 
              sin_port = 49106, sin_addr = {s_addr = 0}, sin_zero = "\020\000\000\000\000#c\027"}, sin6 = {sin6_family = 9620, sin6_port = 49106, sin6_flowinfo = 0, sin6_addr = {__in6_u = {
                  __u6_addr8 = "\020\000\000\000\000#c\027\000P\224\000<8ҿ", __u6_addr16 = {16, 0, 8960, 5987, 20480, 148, 14396, 49106}, __u6_addr32 = {16, 392372992, 9719808, 3218225212}}}, 
              sin6_scope_id = 3218220520}, sas = {ss_family = 9620, 
              __ss_padding = "ҿ\000\000\000\000\020\000\000\000\000#c\027\000P\224\000<8ҿ\350%ҿh\332d\000\364\063ҿP\223T\267\034[U\267\000#c\027 `[\267\214=ҿx%ҿ\000P\224\000<8ҿ\024*ҿ8&ҿ\241\221M\000\204%ҿ\220\241T\267\034[U\267\004\221M\000\200%ҿ\034[U\267\220\241T\267\000#c\027\200%ҿլݷ\000P\224", __ss_align = 9719808}}, id = -1076742084, send_flags = {f = 20480, 
            blst_imask = 148}, proto = -40 '\330', proto_pad0 = 38 '&', proto_pad1 = -16430}
#2  0xb6f1a793 in w_forward (_m=0xb7555b1c, _d=0xb08e6b88 "\004k\216\260", _cflags=0x0) at ims_ipsec_pcscf_mod.c:421
No locals.
#3  0x004cb1fd in do_action (h=0xbfd23264, a=0xb75c918c, msg=0xb7555b1c) at core/action.c:1082
        ret = -5
        v = 0
        dst = {send_sock = 0x7757ce <qm_info+12>, to = {s = {sa_family = 49168, sa_data = "H\267\000P\224\000H.ҿ\020\300H\267"}, sin = {sin_family = 49168, sin_port = 46920, sin_addr = {
                s_addr = 9719808}, sin_zero = "H.ҿ\020\300H\267"}, sin6 = {sin6_family = 49168, sin6_port = 46920, sin6_flowinfo = 9719808, sin6_addr = {__in6_u = {
                  __u6_addr8 = "H.ҿ\020\300H\267\005\000\000\000\000\260H\267", __u6_addr16 = {11848, 49106, 49168, 46920, 5, 0, 45056, 46920}, __u6_addr32 = {3218222664, 3074998288, 5, 
                    3074994176}}}, sin6_scope_id = 3218222584}, sas = {ss_family = 49168, 
              __ss_padding = "H\267\000P\224\000H.ҿ\020\300H\267\005\000\000\000\000\260H\267\370-ҿ\252WH\267\020\300H\267\320-ҿ\330\310\\\267\067WH\267\020.ҿ|.ҿx.ҿ6\365R\000\000\000\200\000\224\270j\000\330\030\017\000pЎ\000\350\203\025\000\004\000\000\000\020\000\000\000\000#c\027b\000\000\000\224z5\267\330.ҿ\a\266}\000pЎ\000\000\000\000\000\000\004\000", 
              __ss_align = 3218222648}}, id = -1221231911, send_flags = {f = 24448, blst_imask = 45177}, proto = 0 '\000', proto_pad0 = 0 '\000', proto_pad1 = 0}
        tmp = 0xb7d4ea7f <__strftime_internal+1359> "\203\304\020\003l$\030\003|$\b\351`\373\377\377\213\\$\004\203\376E\017\205\310\375\377\377\351\063\375\377\377\211\\$\004\061\366\213D$ \205\300\017DD$\034\211D$\034\203\376E\017\204\021\375\377\377\213D$$\213X\020\211ڃ\376O\017\204\350\024"
        new_uri = 0xc859 <error: Cannot access memory at address 0xc859>
        end = 0x1dad7a8 ""
        crt = 0x1dab7b2 "45 /usr/sbin/kamailio[32593]: ERROR: <script>: NOTIFY (sip:711122330@ims.mnc001.mcc001.3gppnetwork.org (192.168.56.106:6060) to sip:711122330@ims.mnc001.mcc001.3gppnetwork.org, bddc50a5-f6da-67cd-0222"...
--Type <RET> for more, q to quit, c to continue without paging-- 
        cmd = 0xb7526698
        len = -1076745526
        user = 2
        uri = {user = {s = 0x17632300 <error: Cannot access memory at address 0x17632300>, len = 31111072}, passwd = {s = 0xe9 <error: Cannot access memory at address 0xe9>, len = -1211037445}, 
          host = {s = 0xb7e74000 "lM\036", len = 21}, port = {s = 0xb7e74000 "lM\036", len = -1076745032}, params = {s = 0xb7d8e41d <__vsyslog_internal+733> "\203\304\020\213E\344e+\005\024", 
            len = 31111072}, sip_params = {
            s = 0x1dab7a0 "\260\222\332\001\020 \314\001 28 20:05:45 /usr/sbin/kamailio[32593]: ERROR: <script>: NOTIFY (sip:711122330@ims.mnc001.mcc001.3gppnetwork.org (192.168.56.106:6060) to sip:711122330@ims.mnc001.mcc001.3gppnetwork.org, bddc5"..., len = 233}, headers = {s = 0x4000 <error: Cannot access memory at address 0x4000>, len = -1219295780}, port_no = 11572, proto = 49106, 
          type = 31119264, flags = (unknown: 0xbfd22ce8), transport = {s = 0xb70ca50a "%s: %.*s%s%s%s%.*s", len = 11}, ttl = {s = 0x1 <error: Cannot access memory at address 0x1>, 
            len = 31042192}, user_param = {s = 0xbfd22d50 "\350\203\025", len = -1218584096}, maddr = {s = 0x61cb43e9 <error: Cannot access memory at address 0x61cb43e9>, len = 31111072}, 
          method = {s = 0xe9 <error: Cannot access memory at address 0xe9>, len = 45}, lr = {s = 0x5 <error: Cannot access memory at address 0x5>, len = 20}, r2 = {
            s = 0x1c <error: Cannot access memory at address 0x1c>, len = 11}, gr = {s = 0x79 <error: Cannot access memory at address 0x79>, len = 2}, transport_val = {
            s = 0x169 <error: Cannot access memory at address 0x169>, len = 0}, ttl_val = {s = 0x2a30 <error: Cannot access memory at address 0x2a30>, len = 30972216}, user_param_val = {
            s = 0x4 <error: Cannot access memory at address 0x4>, len = -1210209067}, maddr_val = {s = 0x945000 "\260\236K", len = 7821290}, method_val = {s = 0xbfd22cc0 "\203", len = 0}, 
          lr_val = {s = 0x1c <error: Cannot access memory at address 0x1c>, len = 5516737}, r2_val = {s = 0xbfd22ca8 "\370-ҿ\345\211M", len = -1210204419}, gr_val = {s = 0x945000 "\260\236K", 
            len = 392372992}}
        next_hop = {user = {s = 0xb744b841 "tm: t_reply.c", len = -1220128768}, passwd = {s = 0xbfd22ed8 "h0", len = 9719808}, host = {s = 0xbfd2383c "", len = 9719808}, port = {
            s = 0xbfd22df8 "\330.ҿ\a\266}", len = 5081573}, params = {s = 0xbfd233f4 "", len = -1219190040}, sip_params = {s = 0xb7555b1c "\n", len = 5079200}, headers = {
            s = 0x83 <error: Cannot access memory at address 0x83>, len = -1223908086}, port_no = 11496, proto = 49106, type = ERROR_URI_T, flags = (unknown: 0xbfd2383c), transport = {
            s = 0xbfd22fd4 "ب\\\267\001", len = -1076744968}, ttl = {s = 0xb70c385b <xlog_helper+3092> "\203\304\060\213u\234\213\206\364\377\377\377\213", len = 131}, user_param = {
            s = 0xb7ddacd5 <__memset_sse2_rep+37> "\201\303;\206\004", len = 9719808}, maddr = {s = 0x7757ea <qm_info+40> "\203\304\020\213E\364\213P\004\213E\f\211\020\213E\f\307@\024\004", 
            len = -1076744896}, method = {s = 0x0, len = 28}, lr = {s = 0x7757ce <qm_info+12> "\005\062\370\034", len = 168}, r2 = {
            s = 0xb75de1e0 "NOTIFY (sip:711122330@ims.mnc001.mcc001.3gppnetwork.org (192.168.56.106:6060) to sip:711122330@ims.mnc001.mcc001.3gppnetwork.org, bddc50a5-f6da-67cd-0222-9eef1e619cdc)\n", len = -1218584096}, gr = {s = 0xb748c010 "\001", len = -1076744864}, transport_val = {s = 0xb748b000 "\b\016\001", len = -1076744856}, ttl_val = {
            s = 0xb74857aa <pkg_proc_update_stats+127> "\203\304\020\213\223h\f", len = -1219969008}, user_param_val = {s = 0xbfd22d40 "\372\030", len = -1218673468}, maddr_val = {
            s = 0x787ef8 <parse_headers+17> "\201\303\b\321\033", len = 0}, method_val = {s = 0x0, len = -1076744728}, lr_val = {s = 0x0, len = 6394}, r2_val = {s = 0x0, len = 8}, gr_val = {
            s = 0x0, len = 1410024}}
        u = 0x0
        port = 0
        dst_host = 0xb7e74740 <main_arena>
        i = 8180
        flags = -1218559940
        avp = 0xb7e1b0ee
        st = {flags = 3085385728, id = 18240, name = {n = 240, s = {s = 0xf0 <error: Cannot access memory at address 0xf0>, len = 0}, re = 0xf0}, avp = 0x5}
        sct = 0x1dab7ac
        sjt = 0x17632300
        rve = 0xb75c7ebc
        mct = 0xf
        rv = 0x3
        rv1 = 0x2
        c1 = {cache_type = 392372992, val_type = RV_NONE, c = {avp_val = {n = 9719808, s = {s = 0x945000 "\260\236K", len = -1076744808}, re = 0x945000}, pval = {rs = {s = 0x945000 "\260\236K", 
                len = -1076744808}, ri = 5517120, flags = 0}}, i2s = "լݷ\000P\224\000\352Ww\000\320-ҿ\000\000\000\000\034"}
        s = {s = 0xb7d0e2e9 <__GI__IO_str_seekoff+9> "\201\303\027]\026", len = -1209581568}
        srevp = {0x4, 0x12}
        evp = {data = 0x0, obuf = {s = 0x0, len = 0}, rcv = 0x0, dst = 0x0, req = 0x0, rpl = 0x0, rplcode = 0, mode = 0}
        mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, 
              str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, 
              select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, 
              string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, 
              select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, 
              string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}}
        __func__ = "do_action"
#4  0x004d89e5 in run_actions (h=0xbfd23264, a=0xb75c90e0, msg=0xb7555b1c) at core/action.c:1581
--Type <RET> for more, q to quit, c to continue without paging--
        t = 0xb75c918c
        ret = 1
        tvb = {tv_sec = 0, tv_usec = 0}
        tve = {tv_sec = 0, tv_usec = 0}
        tz = {tz_minuteswest = 10220721, tz_dsttime = 32}
        tdiff = 0
        __func__ = "run_actions"
#5  0x004d92ab in run_top_route (a=0xb75c90e0, msg=0xb7555b1c, c=0xbfd23264) at core/action.c:1666
        ctx = {rec_lev = 8018553, run_flags = -1076743968, last_retcode = -1219142884, jmp_env = {{__jmpbuf = {10220616, -1218664232, 1, 10220670, 9298587, -1220742057}, 
              __mask_was_saved = 9299260, __saved_mask = {__val = {3074998288, 3076311284, 0, 1024, 3218223144, 3074225352, 2962419868, 0, 1, 3, 9334117, 5, 0, 3074838528, 3218225212, 
                  3218223176, 3074229440, 2962419868, 3218223184, 3076303036, 3074229424, 3218225212, 3074225146, 3218223224, 5436726, 2962419868, 3074838528, 5462975, 3218223224, 5463391, 0, 
                  3218223464}}}}}
        p = 0xbfd23264
        ret = 2
        sfbk = 0
#6  0xb73ebb7b in reply_received (p_msg=0xb7555b1c) at t_reply.c:2541
        msg_status = 183
        last_uac_status = 100
        ack = 0xb75ca8d0 "H\001"
        ack_len = 7821262
        branch = 0
        reply_status = 9299260
        onreply_route = 3
        cancel_data = {cancel_bitmap = 0, reason = {cause = 0, u = {text = {s = 0x0, len = 0}, e2e_cancel = 0x0, packed_hdrs = {s = 0x0, len = 0}}}}
        uac = 0xb092f0e4
        t = 0xb092ef0c
        lack_dst = {send_sock = 0x9bf4aa <buf+394>, to = {s = {sa_family = 43188, sa_data = "\\\267\000#c\027+\000\000\000լݷ"}, sin = {sin_family = 43188, sin_port = 46940, sin_addr = {
                s_addr = 392372992}, sin_zero = "+\000\000\000լݷ"}, sin6 = {sin6_family = 43188, sin6_port = 46940, sin6_flowinfo = 392372992, sin6_addr = {__in6_u = {
                  __u6_addr8 = "+\000\000\000լݷ\000P\224\000\352Ww", __u6_addr16 = {43, 0, 44245, 47069, 20480, 148, 22506, 119}, __u6_addr32 = {43, 3084758229, 9719808, 7821290}}}, 
              sin6_scope_id = 3218223680}, sas = {ss_family = 43188, 
              __ss_padding = "\\\267\000#c\027+\000\000\000լݷ\000P\224\000\352Ww\000@2ҿլݷ\372\030\000\000\000\000\000\000P2ҿ\000\000\000\000\034\000\000\000\316Ww\000\020\300H\267լݷ\000P\224\000\020\300H\267p2ҿլݷ\000P\224\000\352Ww\000\200\062ҿ\000\000\000\000\034\000\000\000\316Ww\000\000\000\200\000\000\260H\267\230\062ҿ\370~x\000\000\000\000", __ss_align = 0}}, 
          id = -1076743512, send_flags = {f = 22442, blst_imask = 46920}, proto = -6 '\372', proto_pad0 = 24 '\030', proto_pad1 = 0}
        backup_user_from = 0x9cf330 <def_list+8>
        backup_user_to = 0x9cf334 <def_list+12>
        backup_domain_from = 0x9cf338 <def_list+16>
        backup_domain_to = 0x9cf33c <def_list+20>
        backup_uri_from = 0x9cf328 <def_list>
        backup_uri_to = 0x9cf32c <def_list+4>
        backup_xavps = 0x9cf3d4 <_xavp_list_head>
        backup_xavus = 0x9cf3d8 <_xavu_list_head>
        backup_xavis = 0x9cf3dc <_xavi_list_head>
        replies_locked = 1
        branch_ret = -1332853704
        prev_branch = -1219973120
        blst_503_timeout = -1218656180
        hf = 0x1
        onsend_params = {req = 0x8de29b, rpl = 0xbfd231cc, param = 0x18fa, code = 0, flags = 2, branch = 0, t_rbuf = 0x0, dst = 0xb748c010, send_buf = {s = 0xb75ca884 "\f", len = -1218651916}}
        ctx = {rec_lev = 1, run_flags = 0, last_retcode = -1, jmp_env = {{__jmpbuf = {9719808, -1076742084, -1219142412, -1076744312, -190565353, -873320600}, __mask_was_saved = 0, 
              __saved_mask = {__val = {1394140, 1410024, 4, 23, 3075824412, 8, 9719808, 3218223832, 5517120, 0, 0, 0, 5516737, 6392, 0, 2296, 0, 8299492, 3084758229, 9719808, 5436726, 10221705, 
                  3218223880, 152, 0, 16, 0, 248, 0, 24, 0, 3218224088}}}}}
        bctx = 0x0
        keng = 0x0
--Type <RET> for more, q to quit, c to continue without paging--
        ret = 9719808
        evname = {s = 0xb744e8bb "on_sl_reply", len = 11}
        __func__ = "reply_received"
#7  0x00556110 in do_forward_reply (msg=0xb7555b1c, mode=0) at core/forward.c:764
        new_buf = 0x0
        dst = {send_sock = 0x0, to = {s = {sa_family = 0, sa_data = '\000' <repeats 13 times>}, sin = {sin_family = 0, sin_port = 0, sin_addr = {s_addr = 0}, 
              sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, 
                  __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}, sas = {ss_family = 0, __ss_padding = '\000' <repeats 121 times>, __ss_align = 0}}, 
          id = 0, send_flags = {f = 0, blst_imask = 0}, proto = 0 '\000', proto_pad0 = 0 '\000', proto_pad1 = 0}
        new_len = 4
        r = 2
        ip = {af = 7783723, len = 0, u = {addrl = {9719808, 3074937938, 3074994176, 3218224120}, addr32 = {9719808, 3074937938, 3074994176, 3218224120}, addr16 = {20480, 148, 54354, 46919, 
              45056, 46920, 13304, 49106}, addr = "\000P\224\000R\324G\267\000\260H\267\370\063ҿ"}}
        s = 0xbfd23390 "\376\377\377\377"
        len = -1219142912
        __func__ = "do_forward_reply"
#8  0x00558062 in forward_reply (msg=0xb7555b1c) at core/forward.c:865
No locals.
#9  0x00609def in receive_msg (
    buf=0x9bf320 <buf> "SIP/2.0 183 Session in Progress\r\nFrom: <sip:711122330@ims.mnc001.mcc001.3gppnetwork.org>;tag=527524296\r\nTo: <sip:711122331@ims.mnc001.mcc001.3gppnetwork.org>;tag=18016628\r\nContact: <sip:711122331@192."..., len=1385, rcv_info=0xbfd23774) at core/receive.c:587
        msg = 0xb7555b1c
        ctx = {rec_lev = 4763648, run_flags = -1076742504, last_retcode = -1209552940, jmp_env = {{__jmpbuf = {-1208194672, -1211497464, 9242371, -1076742596, -1208097047, 9719808}, 
              __mask_was_saved = -1218656232, __saved_mask = {__val = {3218225200, 3218224808, 3086895344, 3218225020, 3084528336, 56335, 3086981504, 744, 7583757, 44055, 7583287, 3086774304, 
                  7583694, 1, 0, 1024, 12, 1385, 10220320, 3076310936, 3076310936, 0, 0, 3218224756, 3085481001, 9719808, 9, 10220320, 5436726, 0, 3076310936, 3218224868}}}}}
        bctx = 0x0
        ret = 0
        tvb = {tv_sec = 0, tv_usec = 0}
        tve = {tv_sec = 0, tv_usec = 0}
        diff = 0
        inb = {
          s = 0x9bf320 <buf> "SIP/2.0 183 Session in Progress\r\nFrom: <sip:711122330@ims.mnc001.mcc001.3gppnetwork.org>;tag=527524296\r\nTo: <sip:711122331@ims.mnc001.mcc001.3gppnetwork.org>;tag=18016628\r\nContact: <sip:711122331@192."..., len = 1385}
        netinfo = {data = {s = 0x0, len = 0}, rcv = 0x0, dst = 0x0}
        keng = 0x0
        evp = {data = 0xbfd23580, obuf = {s = 0x0, len = 0}, rcv = 0xbfd23774, dst = 0x0, req = 0x0, rpl = 0x0, rplcode = 0, mode = 0}
        cidlockidx = 0
        cidlockset = 0
        errsipmsg = 0
        exectime = 0
        __func__ = "receive_msg"
#10 0x00744552 in udp_rcv_loop () at core/udp_server.c:543
        len = 1385
        buf = "SIP/2.0 183 Session in Progress\r\nFrom: <sip:711122330@ims.mnc001.mcc001.3gppnetwork.org>;tag=527524296\r\nTo: <sip:711122331@ims.mnc001.mcc001.3gppnetwork.org>;tag=18016628\r\nContact: <sip:711122331@192."...
        tmp = 0x0
        fromaddr = 0xb75cc798
        fromaddrlen = 16
        rcvi = {src_ip = {af = 2, len = 4, u = {addrl = {1782098112, 0, 0, 0}, addr32 = {1782098112, 0, 0, 0}, addr16 = {43200, 27192, 0, 0, 0, 0, 0, 0}, 
              addr = "\300\250\070j", '\000' <repeats 11 times>}}, dst_ip = {af = 2, len = 4, u = {addrl = {2016979136, 0, 0, 0}, addr32 = {2016979136, 0, 0, 0}, addr16 = {43200, 30776, 0, 0, 0, 
                0, 0, 0}, addr = "\300\250\070x", '\000' <repeats 11 times>}}, src_port = 6060, dst_port = 5060, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {sa_family = 2, 
              sa_data = "\027\254\300\250\070j\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 44055, sin_addr = {s_addr = 1782098112}, 
              sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 44055, sin6_flowinfo = 1782098112, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, 
--Type <RET> for more, q to quit, c to continue without paging--
                  __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}, sas = {ss_family = 2, 
              __ss_padding = "\027\254\300\250\070j", '\000' <repeats 115 times>, __ss_align = 0}}, bind_address = 0xb7517f20, rflags = 0, proto = 1 '\001', proto_pad0 = 0 '\000', proto_pad1 = 0}
        evp = {data = 0x0, obuf = {s = 0x0, len = 0}, rcv = 0x0, dst = 0x0, req = 0x0, rpl = 0x0, rplcode = 0, mode = 0}
        printbuf = '\000' <repeats 16 times>, "\066\365R", '\000' <repeats 14 times>, "#c\027\000\000\000\000\360!\235\000\300!\235\000\000P\224\000\000P\224\000\214=ҿ\030\071ҿ\025\273j\000\020\000\000\000\004\000\000\000\326&\373\323F\267j\000(\336cE%\371ЩJO\004a\000#c\027jπ\000j\354sY\310\070ҿn\251\200\000`+\235\000`,\235\000\000P\224\000\364\233R\267\000\000\000\000\000P\224\000\350\070ҿA\251\200\000X+\235\000\340!\235\000 \000\000\000\001\000\000\000\000\000\000\000\000P\224\000\b9ҿ6\365R\000X+\235\000\340!\235\000 \000\000\000\000#c\027\000\000\000\000\000\000\000\000\070\071ҿ"...
        i = 15
        j = 0
        l = 0
        __func__ = "udp_rcv_loop"
#11 0x004b5d52 in main_loop () at main.c:1730
        i = 15
        pid = 0
        si = 0xb7517f20
        si_desc = "udp receiver child=15 sock=192.168.56.120:5060\000\277\000P\224\000;\177\000\000\214=ҿ\310;ҿ\a7c\000\210\254V\267\000\004\000\000\060\252Q\267\316\066c\000Ie\213\000\226A\213\000\005\000\000\000\000\000\000\000܅Tue De\370;ҿ\365\067c\000䨗\000ڋK\000\062\061\n"
        nrprocs = 16
        woneinit = 1
        __func__ = "main_loop"
#12 0x004c1bd0 in main (argc=9, argv=0xbfd24064) at main.c:3053
        cfg_stream = 0x1cc21c0
        c = -1
        r = 0
        tmp = 0xbfd24e7a ""
        tmp_len = 1
        port = 0
        proto = -1209437896
        ahost = 0x0
        aport = 0
        options = 0x871ef0 ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:x:X:Y:"
        ret = -1
        seed = 2926354237
        rfd = 4
        debug_save = 0
        debug_flag = 0
        dont_fork_cnt = 0
        n_lst = 0xbfd23e5f
        p = 0x1 <error: Cannot access memory at address 0x1>
        st = {st_dev = 22, __pad1 = 0, st_ino = 952, st_mode = 16888, st_nlink = 2, st_uid = 111, st_gid = 115, st_rdev = 0, __pad2 = 0, st_size = 40, st_blksize = 4096, st_blocks = 0, 
          st_atim = {tv_sec = 1640710116, tv_nsec = 690868980}, st_mtim = {tv_sec = 1640710116, tv_nsec = 690868980}, st_ctim = {tv_sec = 1640710116, tv_nsec = 690868980}, __glibc_reserved4 = 0, 
          __glibc_reserved5 = 0}
        tbuf = "\000\000\000\000\000\000\000\000\004\000\000\020\000\360\377\377Linux", '\000' <repeats 60 times>, "pcscf.ims.mnc001.mcc001.3gppnetwork.org", '\000' <repeats 26 times>, "\065.10.0-9-686-pae", '\000' <repeats 22 times>, "\f\207\375\267\225jʷO\375I", '\000' <repeats 16 times>...
        option_index = 0
        long_options = {{name = 0x87426a "help", has_arg = 0, flag = 0x0, val = 104}, {name = 0x86f4d6 "version", has_arg = 0, flag = 0x0, val = 118}, {name = 0x87426f "alias", has_arg = 1, 
            flag = 0x0, val = 1024}, {name = 0x874275 "subst", has_arg = 1, flag = 0x0, val = 1025}, {name = 0x87427b "substdef", has_arg = 1, flag = 0x0, val = 1026}, {
            name = 0x874284 "substdefs", has_arg = 1, flag = 0x0, val = 1027}, {name = 0x87428e "server-id", has_arg = 1, flag = 0x0, val = 1028}, {name = 0x874298 "loadmodule", has_arg = 1, 
            flag = 0x0, val = 1029}, {name = 0x8742a3 "modparam", has_arg = 1, flag = 0x0, val = 1030}, {name = 0x8742ac "log-engine", has_arg = 1, flag = 0x0, val = 1031}, {
            name = 0x8742b7 "debug", has_arg = 1, flag = 0x0, val = 1032}, {name = 0x8742bd "cfg-print", has_arg = 0, flag = 0x0, val = 1033}, {name = 0x8742c7 "atexit", has_arg = 1, flag = 0x0, 
            val = 1034}, {name = 0x0, has_arg = 0, flag = 0x0, val = 0}}
        __func__ = "main"
(gdb) 

@Dgeka25594
Copy link
Author

Can this bt help with the concept of the problem? Or is it better to take the dump again, attach the log and bt?

@henningw
Copy link
Contributor

henningw commented Jan 3, 2022

Hello, it crashed during the processing of an 183 message. Crashed function was fill_contact(..) in frame 0. You could investigate the passed data in this function to see why it crashed. Another idea would be to look to the SIP message that was processed to see if there was something unusual in it.

@kamailio-sync
Copy link

kamailio-sync commented Jan 6, 2022 via email

@henningw
Copy link
Contributor

henningw commented Jan 6, 2022

@alexyosifov If I do not misread it, I think the second backtrace was already done with debug symbols installed.

@Dgeka25594
Copy link
Author

Hi Eugeniy, You need to install debug symbols as Henning mentioned and provide the backtrace with symbols to investigate. If possible provide P-cscf logs and Wireshark trace. BR Aleksandar Yosifov

On Tue, Dec 28, 2021 at 2:52 PM Eugeniy @.> wrote: Colleagues, there is an opportunity to give a hint why it crashes ims_ipsec_pcscf ? — Reply to this email directly, view it on GitHub <#2970 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABO7UZO5JE2IPFKDMH4ESLLUTGXFNANCNFSM5JZDNT7Q . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you are subscribed to this thread.Message ID: @.> _______________________________________________ Kamailio (SER) - Development Mailing List @.*** https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-dev

Hi, Aleksandar.

In the archive in the attachment a new dump, log and bt with debug symbols
pcscf_crashes.zip

@Dgeka25594
Copy link
Author

Dgeka25594 commented Jan 10, 2022

Hello, it crashed during the processing of an 183 message. Crashed function was fill_contact(..) in frame 0. You could investigate the passed data in this function to see why it crashed. Another idea would be to look to the SIP message that was processed to see if there was something unusual in it.

Hey!
Thanks for the advice, but don't tell me where the fill_contact fields are filled in from the SIP packets?

@YaZasnyal
Copy link

YaZasnyal commented Jan 12, 2022

It seems that problem is not in contact parsing. I think that object are in some failed state

In core dump we have what it seems to be a valid Contact field

#0  0x00007f0b29e26979 in fill_contact (ci=0x7fffff6755a0, m=0x7f0b2a68e780) at cmd.c:271
"<sip:alice@10.2.16.36:5060;gr=00F54E59-1172-EC11-9AC7-7D467C83D9C2>
----
INVITE sip:bob@ims.mnc001.mcc001.3gppnetwork.org SIP/2.0
Via: SIP/2.0/UDP 10.2.16.36:5060;branch=z9hG4bK8019dd952e72ec119bd47d467c83d9c2;rport
From: "PhonerLite" <sip:alice@ims.mnc001.mcc001.3gppnetwork.org>;tag=4116302762
To: <sip:bob@ims.mnc001.mcc001.3gppnetwork.org>
Call-ID: 8019DD95-2E72-EC11-9BD3-7D467C83D9C2@10.2.16.36
CSeq: 1 INVITE
Contact: <sip:alice@10.2.16.36:5060;gr=00F54E59-1172-EC11-9AC7-7D467C83D9C2>
Content-Type: application/sdp
Allow: INVITE, ACK, BYE, CANCEL, INFO, MESSAGE, NOTIFY, OPTIONS, REFER, UPDATE, PRACK
Max-Forwards: 69
Supported: 100rel, replaces, from-change, gruu
User-Agent: PhonerLite/2.97
P-Preferred-Identity: <sip:alice@ims.mnc001.mcc001.3gppnetwork.org>
Content-Length:   340

We have normal pointer to parsed object but object itself contains garbage. This is why no parsing is performed and this check passes.

(gdb) p req->contact.parsed
$45 = (void *) 0x7f0b2a69b778
(gdb) p *(contact_body_t*)req->contact.parsed
$46 = {
  star = 3 '\003', <---- this
  contacts = 0x38  <---- and this
}

If we take next header than the parsed pointer will contain even stranger things

(gdb) p *req->contact.next
$49 = {
  type = HDR_CONTENTTYPE_T,
  name = {
    s = 0x7f0b238ab6c3 "Content-Type: application/sdp\r\nAllow: INVITE, ACK, BYE, CANCEL, INFO, MESSAGE, NOTIFY, OPTIONS, REFER, UPDATE, PRACK\r\nMax-Forwards: 69\r\nSupported: 100rel, replaces, from-change, gruu\r\nUser-Agent: Phon"...,
    len = 12
  },
  body = {
    s = 0x7f0b238ab6d1 "application/sdp\r\nAllow: INVITE, ACK, BYE, CANCEL, INFO, MESSAGE, NOTIFY, OPTIONS, REFER, UPDATE, PRACK\r\nMax-Forwards: 69\r\nSupported: 100rel, replaces, from-change, gruu\r\nUser-Agent: PhonerLite/2.97\r\nP"...,
    len = 15
  },
  len = 31,
  parsed = 0x30003, <------- this
  next = 0x7f0b238ac088
}

I am not much into this code and unfortunately I am unable to find where this values are set. Are messages reused or created from scratch each time? The m variable (reply message) that passed into this method seems to be ok by the way.

@Dgeka25594
Copy link
Author

Friends, do you have any ideas about the problem?

Copy link

github-actions bot commented Nov 8, 2023

This issue is stale because it has been open 6 weeks with no activity. Remove stale label or comment or this will be closed in 2 weeks.

@github-actions github-actions bot added the Stale label Nov 8, 2023
@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Nov 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants