Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

segfault v 2.2 on reactor init #779

Closed
cepehutu opened this issue Feb 2, 2016 · 26 comments
Closed

segfault v 2.2 on reactor init #779

cepehutu opened this issue Feb 2, 2016 · 26 comments
Assignees
Milestone

Comments

@cepehutu
Copy link

cepehutu commented Feb 2, 2016

eb 2 11:02:05 front--voice-4 kernel: opensips[59678]: segfault at 70 ip 000000000056655a sp 00007ffc3153d5f0 error 4 in opensips[400000+1df000]
Feb 2 11:02:05 front--voice-4 kernel: opensips[59682]: segfault at 70 ip 000000000056655a sp 00007ffc3153d5f0 error 4 in opensips[400000+1df000]
Feb 2 11:02:05 front--voice-4 opensips_log[59670]: INFO:core:handle_sigs: child process 59678 exited by a signal 11
Feb 2 11:02:05 front--voice-4 opensips_log[59670]: INFO:core:handle_sigs: core was not generated
Feb 2 11:02:05 front--voice-4 opensips_log[59670]: INFO:core:handle_sigs: child process 59682 exited by a signal 11
Feb 2 11:02:05 front--voice-4 opensips_log[59670]: INFO:core:handle_sigs: core was not generated
Feb 2 11:02:05 front--voice-4 opensips_log[59670]: INFO:core:handle_sigs: terminating due to SIGCHLD

version: opensips 2.2-dev (x86_64/linux)
flags: STATS: On, EXTRA_DEBUG, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, F_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
main.c compiled on 10:29:53 Feb 2 2016 with gcc 4.4.7

@cepehutu
Copy link
Author

cepehutu commented Feb 2, 2016

All log:

Feb 2 11:05:51 front--voice-4 opensips: INFO:core:fix_poll_method: using epoll_lt as the IO watch method (auto detected)
Feb 2 11:05:51 front--voice-4 /-voice/sbin/opensips[59731]: NOTICE:core:main: version: opensips 2.2-dev (x86_64/linux)
Feb 2 11:05:51 front--voice-4 /-voice/sbin/opensips[59731]: INFO:core:main: using 32 Mb shared memory
Feb 2 11:05:51 front--voice-4 /-voice/sbin/opensips[59731]: INFO:core:main: using 2 Mb private memory per process
Feb 2 11:05:51 front--voice-4 /-voice/sbin/opensips[59731]: WARNING:core:init_reactor_size: shrinking reactor size from 102400 (autodetected via rlimit) to 6553 (limited by memory of 10% from 2097152Mb)
Feb 2 11:05:51 front--voice-4 /-voice/sbin/opensips[59731]: WARNING:core:init_reactor_size: use 'open_files_limit' to enforce other limit or increase PKG memory
Feb 2 11:05:51 front--voice-4 /-voice/sbin/opensips[59731]: INFO:core:evi_publish_event: Registered event <E_CORE_THRESHOLD(0)>
Feb 2 11:05:51 front--voice-4 /-voice/sbin/opensips[59731]: INFO:core:evi_publish_event: Registered event <E_CORE_SHM_THRESHOLD(1)>
Feb 2 11:05:51 front--voice-4 /-voice/sbin/opensips[59731]: INFO:core:evi_publish_event: Registered event <E_CORE_PKG_THRESHOLD(2)>
Feb 2 11:05:51 front--voice-4 /seder-voice/sbin/opensips[59731]: INFO:tm:mod_init: TM - initializing...
Feb 2 11:05:51 front--voice-4 /-voice/sbin/opensips[59731]: INFO:sl:mod_init: Initializing StateLess engine
Feb 2 11:05:51 front--voice-4 /-voice/sbin/opensips[59731]: NOTICE:signaling:mod_init: initializing module ...
Feb 2 11:05:51 front--voice-4 /-voice/sbin/opensips[59731]: INFO:rr:mod_init: rr - initializing
Feb 2 11:05:51 front--voice-4 /-voice/sbin/opensips[59731]: INFO:maxfwd:mod_init: initializing...
Feb 2 11:05:51 front--voice-4 /-voice/sbin/opensips[59731]: INFO:sipmsgops:mod_init: initializing...
Feb 2 11:05:51 front--voice-4 /-voice/sbin/opensips[59731]: INFO:usrloc:ul_init_locks: locks array size 512
Feb 2 11:05:51 front--voice-4 /-voice/sbin/opensips[59731]: INFO:core:evi_publish_event: Registered event <E_UL_AOR_INSERT(3)>
Feb 2 11:05:51 front--voice-4 /-voice/sbin/opensips[59731]: INFO:core:evi_publish_event: Registered event <E_UL_AOR_DELETE(4)>
Feb 2 11:05:51 front--voice-4 /-voice/sbin/opensips[59731]: INFO:core:evi_publish_event: Registered event <E_UL_CONTACT_INSERT(5)>
Feb 2 11:05:51 front--voice-4 /-voice/sbin/opensips[59731]: INFO:core:evi_publish_event: Registered event <E_UL_CONTACT_DELETE(6)>
Feb 2 11:05:51 front--voice-4 /-voice/sbin/opensips[59731]: INFO:core:evi_publish_event: Registered event <E_UL_CONTACT_UPDATE(7)>
Feb 2 11:05:51 front--voice-4 /-voice/sbin/opensips[59731]: INFO:registrar:mod_init: initializing...
Feb 2 11:05:51 front--voice-4 /-voice/sbin/opensips[59731]: INFO:acc:mod_init: initializing...
Feb 2 11:05:51 front--voice-4 /-voice/sbin/opensips[59731]: INFO:core:evi_publish_event: Registered event <E_ACC_EVENT(8)>
Feb 2 11:05:51 front--voice-4 /-voice/sbin/opensips[59731]: INFO:core:evi_publish_event: Registered event <E_ACC_CDR(9)>
Feb 2 11:05:51 front--voice-4 /-voice/sbin/opensips[59731]: INFO:core:evi_publish_event: Registered event <E_ACC_MISSED_EVENT(10)>
Feb 2 11:05:51 front--voice-4 /-voice/sbin/opensips[59731]: INFO:core:probe_max_sock_buff: using rcv buffer of 6962 kb
Feb 2 11:05:51 front--voice-4 kernel: opensips[59735]: segfault at 70 ip 000000000056655a sp 00007fff5ee74c00 error 4 in opensips[400000+1df000]
Feb 2 11:05:51 front--voice-4 kernel: opensips[59736]: segfault at 70 ip 000000000056655a sp 00007fff5ee74c00 error 4 in opensips[400000+1df000]
Feb 2 11:05:51 front--voice-4 opensips: INFO:core:daemonize: pre-daemon process exiting with 0
Feb 2 11:05:51 front--voice-4 kernel: opensips[59738]: segfault at 70 ip 000000000056655a sp 00007fff5ee74c00 error 4 in opensips[400000+1df000]
Feb 2 11:05:51 front--voice-4 kernel: opensips[59737]: segfault at 70 ip 000000000056655a sp 00007fff5ee74c00 error 4 in opensips[400000+1df000]
Feb 2 11:05:51 front--voice-4 /-voice/sbin/opensips[59731]: INFO:core:handle_sigs: child process 59738 exited by a signal 11
Feb 2 11:05:51 front--voice-4 /-voice/sbin/opensips[59731]: INFO:core:handle_sigs: core was generated
Feb 2 11:05:51 front--voice-4 /-voice/sbin/opensips[59731]: INFO:core:handle_sigs: terminating due to SIGCHLD
Feb 2 11:05:51 front--voice-4 /-voice/sbin/opensips[59732]: INFO:core:sig_usr: signal 15 received
Feb 2 11:05:51 front--voice-4 /-voice/sbin/opensips[59733]: INFO:core:sig_usr: signal 15 received
Feb 2 11:05:51 front--voice-4 /-voice/sbin/opensips[59734]: INFO:core:sig_usr: signal 15 receivedFeb 2 11:05:51 front--voice-4 /-voice/sbin/opensips[59731]: INFO:core:cleanup: cleanup

@bogdan-iancu
Copy link
Member

@cepehutu , could you extract a backtrace from the core file ?

@bogdan-iancu bogdan-iancu self-assigned this Feb 2, 2016
@bogdan-iancu bogdan-iancu added this to the 2.1 milestone Feb 2, 2016
@cepehutu
Copy link
Author

cepehutu commented Feb 2, 2016

Couldn't find core file.
Log with strace:

LOG removed as long and irrelevant ( @bogdan-iancu )

@bogdan-iancu
Copy link
Member

The strace log is not helpfull here.

But I see you get a core file:

Feb 2 11:05:51 front--voice-4 /-voice/sbin/opensips[59731]: INFO:core:handle_sigs: child process 59738 exited by a signal 11
Feb 2 11:05:51 front--voice-4 /-voice/sbin/opensips[59731]: INFO:core:handle_sigs: core was generated
Feb 2 11:05:51 front--voice-4 /-voice/sbin/opensips[59731]: INFO:core:handle_sigs: terminating due to SIGCHLD

So, look for the core file into the working dir of opensips try the / or /root or home dir.

@cepehutu
Copy link
Author

cepehutu commented Feb 2, 2016

Yeap. Core generated with strace only:

Feb 2 11:49:17 front--voice-4 opensips_log[10457]: INFO:core:handle_sigs: child process 10473 exited by a signal 11
Feb 2 11:49:17 front--voice-4 opensips_log[10457]: INFO:core:handle_sigs: core was generated
Feb 2 11:49:17 front--voice-4 opensips_log[10457]: INFO:core:handle_sigs: terminating due to SIGCHLD

Without strace:
Feb 2 11:48:45 front--voice-4 opensips_log[10426]: INFO:core:handle_sigs: child process 10433 exited by a signal 11
Feb 2 11:48:45 front--voice-4 opensips_log[10426]: INFO:core:handle_sigs: core was not generated
Feb 2 11:48:45 front--voice-4 opensips_log[10426]: INFO:core:handle_sigs: child process 10434 exited by a signal 11
Feb 2 11:48:45 front--voice-4 opensips_log[10426]: INFO:core:handle_sigs: core was not generated
Feb 2 11:48:45 front--voice-4 opensips_log[10426]: INFO:core:handle_sigs: child process 10443 exited by a signal 11
Feb 2 11:48:45 front--voice-4 opensips_log[10426]: INFO:core:handle_sigs: core was not generated
Feb 2 11:48:45 front--voice-4 opensips_log[10426]: INFO:core:handle_sigs: terminating due to SIGCHLD

Core tar at url:
https://icombats.ru/core.tar

@bogdan-iancu
Copy link
Member

@cepehutu , hard to get info from a core file generated on another machine. Please extract the backtrace on your server by using gdb /path/to/opensips /path/to/core and run "bt full"

@cepehutu
Copy link
Author

cepehutu commented Feb 2, 2016

like this?

13:52+00 root@front--voice-4:/home/ilhovsky]# gdb /-voice/sbin/opensips /core.* bt full
Excess command line arguments ignored. (/core.10465 ...)
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-83.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/...
Reading symbols from /-voice/sbin/opensips...done.
[New Thread 10464]
Missing separate debuginfo for /-voice/lib64/opensips/modules/proto_ws.so
Try: yum --enablerepo='-debug' install /usr/lib/debug/.build-id/de/7b3efe36c260e164548ecd260d9a27cc4ed7a1
Missing separate debuginfo for /-voice/lib64/opensips/modules/proto_tls.so
Try: yum --enablerepo='-debug' install /usr/lib/debug/.build-id/9a/8d11fc26c38d97f19f745d7db69595990d9363
Missing separate debuginfo for /-voice/lib64/opensips/modules/tls_mgm.so
Try: yum --enablerepo='-debug' install /usr/lib/debug/.build-id/c4/1469ee4c76bd9a30463c8c61b4a35335610122
Missing separate debuginfo for /-voice/lib64/opensips/modules/rtpengine.so
Try: yum --enablerepo='-debug' install /usr/lib/debug/.build-id/14/faaaa7002c9f5214c367c789cc25315b30aa39
Missing separate debuginfo for /-voice/lib64/opensips/modules/textops.so
Try: yum --enablerepo='-debug' install /usr/lib/debug/.build-id/92/8a02dd98745726e5c563247397a434254cb552
Missing separate debuginfo for /-voice/lib64/opensips/modules/sipmsgops.so
Try: yum --enablerepo='-debug' install /usr/lib/debug/.build-id/a1/d43773dc27ab65e81c01507dac1571285ecc7c
Missing separate debuginfo for /-voice/lib64/opensips/modules/maxfwd.so
Try: yum --enablerepo='-debug' install /usr/lib/debug/.build-id/df/12fb7b846814186c9594264efaa61c96494440
Missing separate debuginfo for /-voice/lib64/opensips/modules/sl.so
Try: yum --enablerepo='-debug' install /usr/lib/debug/.build-id/91/9baa56a8c9d9f5aaf0ae82523ab4a7fd948ed2
Missing separate debuginfo for /-voice/lib64/opensips/modules/rr.so
Try: yum --enablerepo='-debug' install /usr/lib/debug/.build-id/6e/16219d8e6b3784a68bbe525ca704d47bf1d1fc
Missing separate debuginfo for /-voice/lib64/opensips/modules/tm.so
Try: yum --enablerepo='-debug' install /usr/lib/debug/.build-id/98/1e87317a2bf30c93f9c2e8f3a4963f142c6fa6
Missing separate debuginfo for /-voice/lib64/opensips/modules/avpops.so
Try: yum --enablerepo='-debug' install /usr/lib/debug/.build-id/c7/011efc260e832fecf20914de00a813f96041d6
Missing separate debuginfo for /-voice/lib64/opensips/modules/uri.so
Try: yum --enablerepo='-debug' install /usr/lib/debug/.build-id/0f/ab5ab9b1dd5b3eb6106701163423c1de09ce71
Missing separate debuginfo for /-voice/lib64/opensips/modules/usrloc.so
Try: yum --enablerepo='-debug' install /usr/lib/debug/.build-id/32/775f57ca92ae4f376800188dc446bf7807994d
Missing separate debuginfo for /-voice/lib64/opensips/modules/signaling.so
Try: yum --enablerepo='-debug' install /usr/lib/debug/.build-id/92/b7a60958d8f21caae3853591d9327e18839533
Missing separate debuginfo for /-voice/lib64/opensips/modules/registrar.so
Try: yum --enablerepo='-debug' install /usr/lib/debug/.build-id/30/0e0e41f16467d73c614ea2b91f3abdf48e6ca9
Missing separate debuginfo for /-voice/lib64/opensips/modules/domain.so
Try: yum --enablerepo='-debug' install /usr/lib/debug/.build-id/46/d836105a72cb3cb10fe9e63b141079efdc6bd4
Missing separate debuginfo for /-voice/lib64/opensips/modules/db_postgres.so
Try: yum --enablerepo='-debug' install /usr/lib/debug/.build-id/58/ada3c09eb28d6fb455b025781c8c9a2ce52fa5
Missing separate debuginfo for /-voice/lib64/opensips/modules/mi_fifo.so
Try: yum --enablerepo='-debug' install /usr/lib/debug/.build-id/44/314d7ec3240a13e047dc4e094943aebfa692ed
Missing separate debuginfo for /-voice/lib64/opensips/modules/auth.so
Try: yum --enablerepo='-debug' install /usr/lib/debug/.build-id/61/016e853ad8ed565ea471bf912b00b1da8d4de2
Missing separate debuginfo for /-voice/lib64/opensips/modules/auth_db.so
Try: yum --enablerepo='-debug' install /usr/lib/debug/.build-id/f8/4cf9668a7e4da20ca78d2aa9808e63699433a0
Missing separate debuginfo for /-voice/lib64/opensips/modules/dialog.so
Try: yum --enablerepo='-debug' install /usr/lib/debug/.build-id/0d/af6f24214e0946db6efd39476e32dc1f41cbe0
Missing separate debuginfo for /-voice/lib64/opensips/modules/nat_traversal.so
Try: yum --enablerepo='-debug' install /usr/lib/debug/.build-id/5c/3498304ca79082631c32b0d626228fde3b1255
Missing separate debuginfo for /-voice/lib64/opensips/modules/permissions.so
Try: yum --enablerepo='-debug' install /usr/lib/debug/.build-id/cb/25b7ff60067cffbc347da9a651a7ff33406c97
Missing separate debuginfo for /-voice/lib64/opensips/modules/aaa_radius.so
Try: yum --enablerepo='-debug' install /usr/lib/debug/.build-id/0c/3432ec69adcb06246591048fa7b4b8949845a7
Missing separate debuginfo for /-voice/lib64/opensips/modules/auth_aaa.so
Try: yum --enablerepo='-debug' install /usr/lib/debug/.build-id/37/12aaeb41eb4e20f3f3b0b7d0f28b1d9fa230dc
Missing separate debuginfo for /-voice/lib64/opensips/modules/nathelper.so
Try: yum --enablerepo='-debug' install /usr/lib/debug/.build-id/f8/f7ebdeb1348dd15d860fa6ede89f9c73a1b2c9
Missing separate debuginfo for /-voice/lib64/opensips/modules/pike.so
Try: yum --enablerepo='-debug' install /usr/lib/debug/.build-id/1f/3d56a91e116050d9df12031ee1b5ee7bcd6d53
Missing separate debuginfo for /-voice/lib64/opensips/modules/acc.so
Try: yum --enablerepo='-debug' install /usr/lib/debug/.build-id/ed/8905f783e04ce2421fd66b14fcb00951e74841
Missing separate debuginfo for
Try: yum --enablerepo='-debug' install /usr/lib/debug/.build-id/20/c5cf84b53811b4d8dd34cdde046e34b4d66282
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /-voice/lib64/opensips/modules/proto_ws.so...done.
Loaded symbols for /-voice/lib64/opensips/modules/proto_ws.so
Reading symbols from /-voice/lib64/opensips/modules/proto_tls.so...done.
Loaded symbols for /-voice/lib64/opensips/modules/proto_tls.so
Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libssl.so.10
Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libcrypto.so.10
Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /-voice/lib64/opensips/modules/tls_mgm.so...done.
Loaded symbols for /-voice/lib64/opensips/modules/tls_mgm.so
Reading symbols from /-voice/lib64/opensips/modules/rtpengine.so...done.
Loaded symbols for /-voice/lib64/opensips/modules/rtpengine.so
Reading symbols from /-voice/lib64/opensips/modules/textops.so...done.
Loaded symbols for /-voice/lib64/opensips/modules/textops.so
Reading symbols from /-voice/lib64/opensips/modules/sipmsgops.so...done.
Loaded symbols for /-voice/lib64/opensips/modules/sipmsgops.so
Reading symbols from /-voice/lib64/opensips/modules/maxfwd.so...done.
Loaded symbols for /-voice/lib64/opensips/modules/maxfwd.so
Reading symbols from /-voice/lib64/opensips/modules/sl.so...done.
Loaded symbols for /-voice/lib64/opensips/modules/sl.so
Reading symbols from /-voice/lib64/opensips/modules/rr.so...done.
Loaded symbols for /-voice/lib64/opensips/modules/rr.so
Reading symbols from /-voice/lib64/opensips/modules/tm.so...done.
Loaded symbols for /-voice/lib64/opensips/modules/tm.so
Reading symbols from /-voice/lib64/opensips/modules/avpops.so...done.
Loaded symbols for /-voice/lib64/opensips/modules/avpops.so
Reading symbols from /-voice/lib64/opensips/modules/uri.so...done.
Loaded symbols for /-voice/lib64/opensips/modules/uri.so
Reading symbols from /-voice/lib64/opensips/modules/usrloc.so...done.
Loaded symbols for /-voice/lib64/opensips/modules/usrloc.so
Reading symbols from /-voice/lib64/opensips/modules/signaling.so...done.
Loaded symbols for /-voice/lib64/opensips/modules/signaling.so
Reading symbols from /-voice/lib64/opensips/modules/registrar.so...done.
Loaded symbols for /-voice/lib64/opensips/modules/registrar.so
Reading symbols from /-voice/lib64/opensips/modules/domain.so...done.
Loaded symbols for /-voice/lib64/opensips/modules/domain.so
Reading symbols from /-voice/lib64/opensips/modules/db_postgres.so...done.
Loaded symbols for /-voice/lib64/opensips/modules/db_postgres.so
Reading symbols from /usr/pgsql-9.4/lib/libpq.so.5...(no debugging symbols found)...done.
Loaded symbols for /usr/pgsql-9.4/lib/libpq.so.5
Reading symbols from /lib64/libldap_r-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libldap_r-2.4.so.2
Reading symbols from /lib64/liblber-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/liblber-2.4.so.2
Reading symbols from /usr/lib64/libsasl2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libsasl2.so.2
Reading symbols from /usr/lib64/libssl3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libssl3.so
Reading symbols from /usr/lib64/libsmime3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libsmime3.so
Reading symbols from /usr/lib64/libnss3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnss3.so
Reading symbols from /usr/lib64/libnssutil3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnssutil3.so
Reading symbols from /lib64/libplds4.so...(no debugging symbols found)...done.
Loaded symbols for /lib64/libplds4.so
Reading symbols from /lib64/libplc4.so...(no debugging symbols found)...done.
Loaded symbols for /lib64/libplc4.so
Reading symbols from /lib64/libnspr4.so...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnspr4.so
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libfreebl3.so...(no debugging symbols found)...done.
Loaded symbols for /lib64/libfreebl3.so
Reading symbols from /-voice/lib64/opensips/modules/mi_fifo.so...done.
Loaded symbols for /-voice/lib64/opensips/modules/mi_fifo.so
Reading symbols from /-voice/lib64/opensips/modules/auth.so...done.
Loaded symbols for /-voice/lib64/opensips/modules/auth.so
Reading symbols from /-voice/lib64/opensips/modules/auth_db.so...done.
Loaded symbols for /-voice/lib64/opensips/modules/auth_db.so
Reading symbols from /-voice/lib64/opensips/modules/dialog.so...done.
Loaded symbols for /-voice/lib64/opensips/modules/dialog.so
Reading symbols from /-voice/lib64/opensips/modules/nat_traversal.so...done.
Loaded symbols for /-voice/lib64/opensips/modules/nat_traversal.so
Reading symbols from /-voice/lib64/opensips/modules/permissions.so...done.
Loaded symbols for /-voice/lib64/opensips/modules/permissions.so
Reading symbols from /-voice/lib64/opensips/modules/aaa_radius.so...done.
Loaded symbols for /-voice/lib64/opensips/modules/aaa_radius.so
Reading symbols from /usr/lib64/libradiusclient-ng.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libradiusclient-ng.so.2
Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /-voice/lib64/opensips/modules/auth_aaa.so...done.
Loaded symbols for /-voice/lib64/opensips/modules/auth_aaa.so
Reading symbols from /-voice/lib64/opensips/modules/nathelper.so...done.
Loaded symbols for /-voice/lib64/opensips/modules/nathelper.so
Reading symbols from /-voice/lib64/opensips/modules/pike.so...done.
Loaded symbols for /-voice/lib64/opensips/modules/pike.so
Reading symbols from /-voice/lib64/opensips/modules/acc.so...done.
Loaded symbols for /-voice/lib64/opensips/modules/acc.so
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Core was generated by `/-voice/sbin/opensips -f /etc/opensips/opensips.cfg'.
Program terminated with signal 11, Segmentation fault.
#0 io_watch_add (fd=4, type=1, data=0x0, prio=3, flags=1, h=) at net/../io_wait.h:364
364 if (e->flags & flags){
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 glibc-2.12-1.166.el6_7.3.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 nspr-4.10.8-2.el6_7.x86_64 nss-3.19.1-8.el6_7.x86_64 nss-softokn-freebl-3.14.3-23.el6_7.x86_64 nss-util-3.19.1-2.el6_7.x86_64 openldap-2.4.40-7.el6_7.x86_64 openssl-1.0.1e-42.el6_7.2.x86_64 postgresql94-libs-9.4.5-1PGDG.rhel6.x86_64 radiusclient-ng-0.5.6-5.el6.x86_64 zlib-1.2.3-29.el6.x86_64

@bogdan-iancu
Copy link
Member

@cepehutu , once inside gdb, run "bt full" and paste the output.

@cepehutu
Copy link
Author

cepehutu commented Feb 2, 2016

(gdb) bt full
#0 io_watch_add (fd=4, type=1, data=0x0, prio=3, flags=1, h=) at net/../io_wait.h:364
e = 0x60
already =
ep_event = {events = 0, data = {ptr = 0xffffffff00000000, fd = 0, u32 = 0, u64 = 18446744069414584320}}
ctl_flags =
n =
#1 0x0000000000563d52 in tcp_worker_proc (unix_sock=) at net/net_tcp_proc.c:300
FUNCTION = "tcp_worker_proc"
#2 0x00000000005600e3 in tcp_start_processes (chd_rank=, startup_done=) at net/net_tcp.c:1758
r =
reader_fd = {13, 14}
pid =
si =
load_p = 0x7f2e92561a78
FUNCTION = "tcp_start_processes"
#3 0x0000000000439209 in main_loop (argc=, argv=) at main.c:724
startup_done = 0x0
chd_rank = 1
#4 main (argc=, argv=) at main.c:1258
cfg_log_stderr = 0
cfg_stream =
c =
r =
tmp = 0x4108fb "H\203\304\b\303\377\065\342\350<"
tmp_len =
port = 0
proto = 5784901
options = 0x58b818 "f:cCm:M:b:l:n:N:rRvdDFETSVhw:t:u:g:P:G:W:o:"
seed = 3904535557
rfd =
FUNCTION = "main"

@cepehutu
Copy link
Author

cepehutu commented Feb 2, 2016

2.1 version started good at same config.

@bogdan-iancu
Copy link
Member

@cepehutu , it is really strange, I cannot reproduce it.
First, be sure you have an up to date copy of 2.1 branch (do a git pull). Then recompile everything again, to be sure you do not have mixtures of code (do "make proper ; make all")
If still crashing, is there any chance to get temporary access on that machine to do live troubleshooting ?

@cepehutu
Copy link
Author

cepehutu commented Feb 2, 2016

Problem on 2.2 branch,which marked a dev.version: opensips 2.2-dev (x86_64/linux) wri
on 2.1.rpm everething is okey.

Machine on dmz zone, only local access.
Whether there are still some custom debug modes in opensips?

@bogdan-iancu
Copy link
Member

ok, let's try some remote debugging. In gdb, run "p *h" . Post only the output of that command. I assume you will do it over the same core file as above.

@cepehutu
Copy link
Author

cepehutu commented Feb 2, 2016

(gdb) p *h
value has been optimized out
(gdb)

next?

@bogdan-iancu
Copy link
Member

p _worker_io

@cepehutu
Copy link
Author

cepehutu commented Feb 2, 2016

(gdb) p _worker_io
$1 = {name = 0x5b612a "TCP_worker", max_prio = 4, ep_array = 0x7f2e941eb680, epfd = 6, sset = {__val = {0 <repeats 16 times>}}, signo = 0, master_set = {__fds_bits = {
0 <repeats 16 times>}}, max_fd_select = 0, prio_idx = 0x7f2e941eb630, fd_hash = 0x0, fd_array = 0x7f2e941de928, fd_no = 0, max_fd_no = 6553, poll_method = POLL_EPOLL_LT,
flags = 0}

@bogdan-iancu
Copy link
Member

ups.....fd_hash is NULL there 😲

@cepehutu
Copy link
Author

cepehutu commented Feb 2, 2016

What next?

@bogdan-iancu
Copy link
Member

first, please recompile everything:
make proper exclude_modules=""
make all
if still bogus, I will send you a patch to print some extra stuff.

@cepehutu
Copy link
Author

cepehutu commented Feb 2, 2016

make clean
make proper exclude_modules=""
make all
Done.

Feb 2 16:51:48 front--voice-4 opensips_log[46356]: NOTICE:usrloc:preload_udomain: load start time [1454431908]
Feb 2 16:51:48 front--voice-4 opensips_log[46358]: INFO:rtpengine:rtpe_test: rtp proxy udp:10.1.246.24:2223 found, support for it enabled
Feb 2 16:51:48 front--voice-4 kernel: opensips[46363]: segfault at d0 ip 0000000000550302 sp 00007ffefbf485f0 error 4 in opensips[400000+1df000]
Feb 2 16:51:48 front--voice-4 kernel: opensips[46356]: segfault at 70 ip 000000000056066e sp 00007ffefbf48630 error 4 in opensips[400000+1df000]
Feb 2 16:51:48 front--voice-4 kernel: opensips[46357]: segfault at 70 ip 000000000056066e sp 00007ffefbf48630 error 4 in opensips[400000+1df000]
Feb 2 16:51:48 front--voice-4 kernel: opensips[46358]: segfault at 70 ip 000000000056066e sp 00007ffefbf48630 error 4 in opensips[400000+1df000]
Feb 2 16:51:48 front--voice-4 kernel: opensips[46360]: segfault at 70 ip 000000000056066e sp 00007ffefbf48630 error 4 in opensips[400000+1df000]
Feb 2 16:51:48 front--voice-4 kernel: opensips[46362]: segfault at 70 ip 000000000056066e sp 00007ffefbf48630 error 4 in opensips[400000+1df000]
Feb 2 16:51:48 front--voice-4 opensips: INFO:core:daemonize: pre-daemon process exiting with 0
Feb 2 16:51:48 front-sender-voice-4 opensips_log[46349]: INFO:core:handle_sigs: child process 46358 exited by a signal 11

@cepehutu
Copy link
Author

cepehutu commented Feb 2, 2016

(gdb) bt full
#0  io_watch_add (fd=4, type=1, data=0x0, prio=3, flags=1, h=<value optimized out>) at net/../io_wait.h:364
        e = 0x60
        already = <value optimized out>
        ep_event = {events = 0, data = {ptr = 0xffffffff00000000, fd = 0, u32 = 0, u64 = 18446744069414584320}}
        ctl_flags = <value optimized out>
        n = <value optimized out>
#1  0x0000000000563d52 in tcp_worker_proc (unix_sock=<value optimized out>) at net/net_tcp_proc.c:300
        __FUNCTION__ = "tcp_worker_proc"
#2  0x00000000005600e3 in tcp_start_processes (chd_rank=<value optimized out>, startup_done=<value optimized out>) at net/net_tcp.c:1758
        r = <value optimized out>
        reader_fd = {13, 14}
        pid = <value optimized out>
        si = <value optimized out>
        load_p = 0x7ff2016f7a78
        __FUNCTION__ = "tcp_start_processes"
#3  0x0000000000439209 in main_loop (argc=<value optimized out>, argv=<value optimized out>) at main.c:724
        startup_done = 0x0
        chd_rank = 1
#4  main (argc=<value optimized out>, argv=<value optimized out>) at main.c:1258
        cfg_log_stderr = 0
        cfg_stream = <value optimized out>
        c = <value optimized out>
        r = <value optimized out>
        tmp = 0x4108fb "H\203\304\b\303\377\065\342\350<"
        tmp_len = <value optimized out>
        port = 0
        proto = 5784901
        options = 0x58b818 "f:cCm:M:b:l:n:N:rRvdDFETSVhw:t:u:g:P:G:W:o:"
        seed = 190115992
        rfd = <value optimized out>
        __FUNCTION__ = "main"

@bogdan-iancu
Copy link
Member

ok, could you please apply this patch and provide the logs again:
https://gist.github.com/bogdan-iancu/89b0ff0915bb7dbf8dad
Thanks!

@cepehutu
Copy link
Author

cepehutu commented Feb 8, 2016

applied patch.

Feb 8 13:09:05 front--voice-4 kernel: opensips[28159]: segfault at 70 ip 00000000005669da sp 00007ffc79083430 error 4 in opensips[400000+1df000]
Feb 8 13:09:05 front--voice-4 kernel: opensips[28161]: segfault at 70 ip 00000000005669da sp 00007ffc79083430 error 4 in opensips[400000+1df000]
Feb 8 13:09:05 front--voice-4 kernel: opensips[28165]: segfault at 70 ip 00000000005669da sp 00007ffc79083430 error 4 in opensips[400000+1df000]
Feb 8 13:09:05 front--voice-4 kernel: opensips[28162]: segfault at 70 ip 00000000005669da sp 00007ffc79083430 error 4 in opensips[400000+1df000]
Feb 8 13:09:05 front--voice-4 opensips_log[28169]: INFO:rtpengine:rtpe_test: rtp proxy udp:10.1.246.24:2223 found, support for it enabled
Feb 8 13:09:05 front--voice-4 opensips_log[28167]: INFO:rtpengine:rtpe_test: rtp proxy udp:10.1.246.24:2223 found, support for it enabled
Feb 8 13:09:05 front--voice-4 opensips_log[28168]: INFO:rtpengine:rtpe_test: rtp proxy udp:10.1.246.24:2223 found, support for it enabled
Feb 8 13:09:05 front--voice-4 kernel: opensips[28167]: segfault at 70 ip 00000000005669da sp 00007ffc79083430 error 4 in opensips[400000+1df000]
Feb 8 13:09:05 front--voice-4 kernel: opensips[28164]: segfault at 70 ip 00000000005669da sp 00007ffc79083430 error 4 in opensips[400000+1df000]
Feb 8 13:09:05 front--voice-4 kernel: opensips[28169]: segfault at 70 ip 00000000005669da sp 00007ffc79083430 error 4 in opensips[400000+1df000]
Feb 8 13:09:05 front--voice-4 kernel: opensips[28168]: segfault at 70 ip 00000000005669da sp 00007ffc79083430 error 4 in opensips[400000+1df000]

have no core fles :(

@bogdan-iancu bogdan-iancu changed the title segfault v 2.2 segfault v 2.2 on reactor init Mar 20, 2016
@bogdan-iancu
Copy link
Member

@cepehutu , I do not see the logs I inserted into the code - be sure you run in debug level 3 and that you copy here the whole set of logs from opensips starting attempt.

@bogdan-iancu
Copy link
Member

Hi @cepehutu , if you do not have any update on this, I will have to close the report as invalid (so far, nobody else was reported this really strange error.

@bogdan-iancu
Copy link
Member

Closing as there is no follow up here. Also the reported issue seems to be related to some particularities/local issues of the built machine - there is no other similar report and the C code does not confirm such an crash.
@cepehutu , if you get new data, please feel free to reopen. Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants