Opensips crashed on opensips-2.2.2 version

[varghesepaul]

Hi Team ,

we have seen opensips crashed in our server today and it was running successfully for long time.

Opensips version : 2.2.2

ssl : openssl-1.0.1e-48.el6_8.3.x86_64

Back trace BT full:

Program terminated with signal 11, Segmentation fault.
#0 0x00000000004a3fd4 in fm_remove_free (qm=0x7f26de70d000, size=320) at mem/f_malloc.c:187
187 *pf=n->u.nxt_free;
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 glibc-2.12-1.192.el6.x86_64 gnutls-2.8.5-19.el6_7.x86_64 hiredis-0.10.1-3.el6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-57.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libcurl-7.19.7-52.el6.x86_64 libgcrypt-1.4.5-11.el6_4.x86_64 libgpg-error-1.7-4.el6.x86_64 libidn-1.18-2.el6.x86_64 libmicrohttpd-0.9.33-4.el6.x86_64 libselinux-2.0.94-7.el6.x86_64 libssh2-1.4.2-2.el6_7.1.x86_64 libtasn1-2.3-6.el6_5.x86_64 libxml2-2.7.6-21.el6_8.1.x86_64 mysql-community-libs-compat-5.6.19-2.el6.x86_64 nspr-4.11.0-1.el6.x86_64 nss-3.21.0-8.el6.x86_64 nss-softokn-freebl-3.14.3-23.3.el6_8.x86_64 nss-util-3.21.0-2.el6.x86_64 openldap-2.4.40-12.el6.x86_64 openssl-1.0.1e-48.el6_8.3.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt full
#0 0x00000000004a3fd4 in fm_remove_free (qm=0x7f26de70d000, size=320) at mem/f_malloc.c:187
pf = 0x0
hash = 45
#1 fm_malloc (qm=0x7f26de70d000, size=320) at mem/f_malloc.c:415
frag = 0x7f26e0f00a70
n =
hash =
FUNCTION = "fm_malloc"
#2 0x00007f26de4f5cab in shm_malloc (size=320) at ../../mem/shm_mem.h:392
p =
#3 os_malloc (size=320) at tls.h:90
No locals.
#4 0x00000031f0e6ac2e in CRYPTO_malloc () from /usr/lib64/libcrypto.so.10
No symbol table info available.
#5 0x00000031f0e79f67 in ?? () from /usr/lib64/libcrypto.so.10
No symbol table info available.
#6 0x00000031f0e79fde in ?? () from /usr/lib64/libcrypto.so.10
No symbol table info available.
#7 0x00000031f0ef8b62 in EVP_PKEY_CTX_dup () from /usr/lib64/libcrypto.so.10
No symbol table info available.
#8 0x00000031f0eea95d in EVP_MD_CTX_copy_ex () from /usr/lib64/libcrypto.so.10
No symbol table info available.
#9 0x00000031f0efa0b2 in EVP_DigestSignFinal () from /usr/lib64/libcrypto.so.10
No symbol table info available.
---Type to continue, or q to quit---
#10 0x00000031f32354f8 in ?? () from /usr/lib64/libssl.so.10
No symbol table info available.
#11 0x00000031f3235b1c in tls1_generate_master_secret () from /usr/lib64/libssl.so.10
No symbol table info available.
#12 0x00000031f3220d98 in ssl3_get_client_key_exchange () from /usr/lib64/libssl.so.10
No symbol table info available.
#13 0x00000031f3221ebf in ssl3_accept () from /usr/lib64/libssl.so.10
No symbol table info available.
#14 0x00007f26de2d143e in tls_accept (c=0x7f26e113b5c0, poll_events=0x0) at ../tls_mgm/tls_conn_server.h:155
ret =
err =
ssl = 0x7f26e0d81e78
cert =
FUNCTION = "tls_accept"
#15 0x00007f26de2d6ade in tls_fix_read_conn (con=0x7f26e113b5c0, bytes_read=0x7fff2efff71c) at ../tls_mgm/tls_conn_server.h:336
ret =
#16 tls_read_req (con=0x7f26e113b5c0, bytes_read=0x7fff2efff71c) at proto_tls.c:388
bytes = -1
total_bytes = 0
req = 0x7f275eae2da0
FUNCTION = "tls_read_req"
#17 0x000000000050197e in handle_io (fm=, idx=65, event_type=) at net/net_tcp_proc.c:205
ret = 0
n =
---Type to continue, or q to quit---
con = 0x7f26e113b5c0
s =
rw =
resp =
response = {139804961650112, 1}
FUNCTION = "handle_io"
#18 0x00000000005022a0 in io_wait_loop_epoll (repeat=0, t=2, h=) at ./io_wait_loop.h:221
n = 1
r = 65
e = 0x7f275e793718
ret = 1
#19 0x0000000000503567 in tcp_worker_proc (unix_sock=) at net/net_tcp_proc.c:312
FUNCTION = "tcp_worker_proc"
#20 0x00000000004f2713 in tcp_start_processes (chd_rank=0x77a808, startup_done=0x0) at net/net_tcp.c:1761
r =
reader_fd = {63, 65}
pid =
si =
load_p = 0x7f26dea9cc38
FUNCTION = "tcp_start_processes"
#21 0x000000000043209d in main_loop (argc=, argv=) at main.c:677
startup_done = 0x0
chd_rank = 37
#22 main (argc=, argv=) at main.c:1261
---Type to continue, or q to quit---
cfg_stream =
c =
r =
tmp = 0x7fff2f000f66 ""
tmp_len =
port = 0
proto = 5368389
protos_no =
options = 0x526098 "f:cCm:M:b:l:n:N:rRvdDFETSVhw:t:u:g:P:G:W:o:"
ret = -1
seed = 882618245
rfd =
FUNCTION = "main"
(gdb)

Can you help us to understand what might be issue ?. Happy to share any more details.

[razvancrainea]

Hi, Paul!

Can you please update to the latest OpenSIPS 2.2.3? Some of the fixes committed in 2.2.3 might have fixed this issue.
I will now close the ticket - please open a new one for 2.2.3 if the issue still persists.

Best regards,
Răzvan

[varghesepaul]

Hi @razvancrainea ,

Thanks for the response. While trying to upgrade the RPM we are seeing the redis module is missing for 2.2.3.

RPM Location : http://yum.opensips.org/2.2/releases/el/6/x86_64/

Can you please have a look into this ?.

[nikbyte]

Unfortunately EPEL project for redhat 6 marked as orphaned and removed libhiredis because there is no maintainer for this package now. Building of cachedb_redis module was disabled on redhat 6 after they removed this package. There is no possibility to build cachedb_redis package on redhat 6 yet.

[varghesepaul]

Thanks @nikbyte . We were able to compile it from SRC RPMs and able to deploy. All good now.