Opensips Crash on mi_fifo query (1.11.3 and 1.11.4)

[kneeoh]

I am seeing crashes with both 1.11.3 and 1.11.4 when the system is under load and we perform back to back fifo get_statistics all | grep udp commands (about 10 seconds apart). Opensips will segfault.

Children = 8
pkgmem = 512
Shmem = 2048

modules:
couchbase
maxfwd
mysql
sl
signaling
json
rest_client
avpops
tm
sipmsgops
rr
mi_fifo
options
drouting
uri
benchmark

Apr  7 20:29:14 osips03 /usr/local/sbin/opensips[28988]: CRITICAL:core:fm_status: different free frag. count: 2!=3 for hash 200

/usr/local/sbin/opensips[28988]: CRITICAL:core:fm_status: different free frag. count: 2!=3 for hash 200
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff26795000
Core was generated by `/usr/local/sbin/opensips -P /var/run/opensips/opensips.pid -m 1024 -M 64 -u ope'.
Program terminated with signal 11, Segmentation fault.
#0  0x000000000051bde6 in fm_remove_free (n=0x7f0486fdde68, qm=0x7f0486fba010) at mem/f_malloc.c:172
172             *pf=n->u.nxt_free;

(gdb) bt
#0  0x000000000051bde6 in fm_remove_free (n=0x7f0486fdde68, qm=0x7f0486fba010) at mem/f_malloc.c:172
#1  fm_malloc (qm=0x7f0486fba010, size=72) at mem/f_malloc.c:386
#2  0x00000000005227d7 in parse_headers (msg=0x7f0486fdb7c8, flags=512, next=0) at parser/msg_parser.c:315
#3  0x00007f04843af74c in find_first_route (_m=0x7f0486fdb7c8) at loose.c:120
#4  loose_route (_m=<optimized out>) at loose.c:894
#5  0x00000000004199af in do_action (a=0x7f0486fcad68, msg=0x7f0486fdb7c8) at action.c:1836
#6  0x0000000000420945 in run_action_list (a=<optimized out>, msg=0x7f0486fdb7c8) at action.c:172
#7  0x00000000004a8ab8 in eval_elem (e=0x7f0486fcae48, msg=0x7f0486fdb7c8, val=0x0) at route.c:1618
#8  0x00000000004a85c8 in eval_expr (e=0x7f0486fcae48, msg=0x7f0486fdb7c8, val=0x0) at route.c:1963
#9  0x00000000004a85b5 in eval_expr (e=0x7f0486fcae98, msg=0x7f0486fdb7c8, val=0x0) at route.c:1984
#10 0x0000000000419787 in do_action (a=0x7f0486fcb8f8, msg=0x7f0486fdb7c8) at action.c:1085
#11 0x0000000000420945 in run_action_list (a=<optimized out>, msg=0x7f0486fdb7c8) at action.c:172
#12 0x00000000004210c8 in run_actions (msg=0x7f0486fdb7c8, a=0x7f0486fc58f8) at action.c:137
#13 run_actions (msg=0x7f0486fdb7c8, a=0x7f0486fc58f8) at action.c:191
#14 run_top_route (a=0x7f0486fc58f8, msg=0x7f0486fdb7c8) at action.c:212
#15 0x000000000049192f in receive_msg (buf=0x837c20 "INVITE sip:18608281122@10.1.0.180:5060 SIP/2.0\r\nRecord-Route: <sip:10.1.0.56;lr;ftag=BS57jDtSF3H1p;did=d36.69b4aaa4>\r\nVia: SIP/2.0/UDP 10.1.0.56:5060;branch=z9hG4bKae91.dc208921.0\r\nVia: SI"..., 
    len=<optimized out>, rcv_info=<optimized out>) at receive.c:172
#16 0x00000000005149f1 in udp_rcv_loop () at udp_server.c:422
#17 0x0000000000415577 in main_loop () at main.c:929
#18 main (argc=<optimized out>, argv=<optimized out>) at main.c:1612

I've got a back trace, but would like to send it privately please.

[razvancrainea]

Can you privately send me the backtrace at razvan on opensips.org email? I will send you further instructions over the email.

[kneeoh]

Razvan, I just sent you the core file.

[razvancrainea]

Is this still happening? If so, let me know so we can debug further.