Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

crash on run_rr_callbacks() from dlg_onroute() #1059

Closed
linuxmaniac opened this issue Apr 5, 2017 · 16 comments
Closed

crash on run_rr_callbacks() from dlg_onroute() #1059

linuxmaniac opened this issue Apr 5, 2017 · 16 comments

Comments

@linuxmaniac
Copy link
Member

linuxmaniac commented Apr 5, 2017

It seems that kamailio received two BYE messages at the same time and dlg was gone?

Debugging Data

# gdb `which kamailio` core.kamailio.sig11.19255 
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/kamailio...Reading symbols from /usr/lib/debug/.build-id/ac/30bd0b9cd954ea6a966ccc9d55affb6291beb5.debug...done.
done.
[New LWP 19255]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/kamailio -f /etc/kamailio/proxy/kamailio.cfg -P /var/run/kamailio/kam'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f8c1d999565 in remove_dialog_timer () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so
(gdb) 
(gdb) bt full
#0  0x00007f8c1d999565 in remove_dialog_timer () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so
No symbol table info available.
#1  0x00007f8c1d97b4e5 in dlg_onroute () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dialog.so
No symbol table info available.
#2  0x00007f8c2551728d in run_rr_callbacks (req=0x7f8c28bea120, rr_param=0x7ffc7e4bee40) at rr_cb.c:96
        l_param = {
          s = 0x9aab6a <buf+426> "lr=on;ftag=922229c699;did=793.53a;ice_caller=strip;ice_callee=strip;aset=50;rtpprx=yes;vsf=cTdyd2pyYnZLVHc3VmI0VmQ3aGJxN3J3anJidktUdzdWYjRWXgJYVEE->\r\nProxy-Authorization: Digest username=\"anumber\""..., len = 147}
        cbp = 0x7f8c290558c0
        __FUNCTION__ = "run_rr_callbacks"
#3  0x00007f8c255098a2 in after_loose (_m=0x7f8c28bea120, preloaded=2) at loose.c:928
        puri = {user = {s = 0x0, len = 0}, passwd = {s = 0x0, len = 0}, host = {
            s = 0x9aab5a <buf+410> "Y.Y.Y.Y:5062;lr=on;ftag=922229c699;did=793.53a;ice_caller=strip;ice_callee=strip;aset=50;rtpprx=yes;vsf=cTdyd2pyYnZLVHc3VmI0VmQ3aGJxN3J3anJidktUdzdWYjRWXgJYVEE->\r\nProxy-Authorization: Digest userna"..., len = 10}, port = {
            s = 0x9aab65 <buf+421> "5062;lr=on;ftag=922229c699;did=793.53a;ice_caller=strip;ice_callee=strip;aset=50;rtpprx=yes;vsf=cTdyd2pyYnZLVHc3VmI0VmQ3aGJxN3J3anJidktUdzdWYjRWXgJYVEE->\r\nProxy-Authorization: Digest username=\"3562741"..., len = 4}, params = {
            s = 0x9aab6a <buf+426> "lr=on;ftag=922229c699;did=793.53a;ice_caller=strip;ice_callee=strip;aset=50;rtpprx=yes;vsf=cTdyd2pyYnZLVHc3VmI0VmQ3aGJxN3J3anJidktUdzdWYjRWXgJYVEE->\r\nProxy-Authorization: Digest username=\"anumber\""..., len = 147}, sip_params = {
            s = 0x9aab6a <buf+426> "lr=on;ftag=922229c699;did=793.53a;ice_caller=strip;ice_callee=strip;aset=50;rtpprx=yes;vsf=cTdyd2pyYnZLVHc3VmI0VmQ3aGJxN3J3anJidktUdzdWYjRWXgJYVEE->\r\nProxy-Authorization: Digest username=\"anumber\""..., len = 147}, headers = {s = 0x0, len = 0}, port_no = 5062, proto = 0, type = SIP_URI_T, flags = (unknown: 0), transport = {s = 0x0, len = 0}, ttl = {s = 0x0, 
            len = 0}, user_param = {s = 0x0, len = 0}, maddr = {s = 0x0, len = 0}, method = {s = 0x0, len = 0}, lr = {
            s = 0x9aab6a <buf+426> "lr=on;ftag=922229c699;did=793.53a;ice_caller=strip;ice_callee=strip;aset=50;rtpprx=yes;vsf=cTdyd2pyYnZLVHc3VmI0VmQ3aGJxN3J3anJidktUdzdWYjRWXgJYVEE->\r\nProxy-Authorization: Digest username=\"anumber\""..., len = 5}, r2 = {s = 0x0, len = 0}, gr = {s = 0x0, len = 0}, transport_val = {s = 0x0, len = 0}, ttl_val = {s = 0x0, len = 0}, user_param_val = {s = 0x0, len = 0}, 
          maddr_val = {s = 0x0, len = 0}, method_val = {s = 0x0, len = 0}, lr_val = {
            s = 0x9aab6d <buf+429> "on;ftag=922229c699;did=793.53a;ice_caller=strip;ice_callee=strip;aset=50;rtpprx=yes;vsf=cTdyd2pyYnZLVHc3VmI0VmQ3aGJxN3J3anJidktUdzdWYjRWXgJYVEE->\r\nProxy-Authorization: Digest username=\"anumber\",re"..., len = 2}, r2_val = {s = 0x0, len = 0}, gr_val = {s = 0x0, len = 0}}
        rt = 0x0
        status = 0
        uri = {
          s = 0x9aab56 <buf+406> "sip:Y.Y.Y.Y:5062;lr=on;ftag=922229c699;did=793.53a;ice_caller=strip;ice_callee=strip;aset=50;rtpprx=yes;vsf=cTdyd2pyYnZLVHc3VmI0VmQ3aGJxN3J3anJidktUdzdWYjRWXgJYVEE->\r\nProxy-Authorization: Digest us"..., len = 167}
        use_ob = 0
        rparams = {
          s = 0x9aab6a <buf+426> "lr=on;ftag=922229c699;did=793.53a;ice_caller=strip;ice_callee=strip;aset=50;rtpprx=yes;vsf=cTdyd2pyYnZLVHc3VmI0VmQ3aGJxN3J3anJidktUdzdWYjRWXgJYVEE->\r\nProxy-Authorization: Digest username=\"anumber\""..., len = 147}
        __FUNCTION__ = "after_loose"
#4  0x0000000000446650 in do_action (h=0x7ffc7e4bf560, a=0x7f8c27f994c8, msg=0x7f8c28bea120) at action.c:1060
        ret = -5
        v = 24865014
        dst = {send_sock = 0x17b6aab, to = {s = {sa_family = 52142, sa_data = "B\000\000\000\000\000\253j{\001\000\000\000"}, sin = {sin_family = 52142, sin_port = 66, sin_addr = {s_addr = 0}, 
              sin_zero = "\253j{\001\000\000\000"}, sin6 = {sin6_family = 52142, sin6_port = 66, sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 = "\253j{\001\000\000\000\000\205RD\000\000\000\000", 
                  __u6_addr16 = {27307, 379, 0, 0, 21125, 68, 0, 0}, __u6_addr32 = {24865451, 0, 4477573, 0}}}, sin6_scope_id = 2118906304}}, id = 32764, proto = -16 '\360', send_flags = {f = 241 '\361', 
            blst_imask = 75 'K'}}
        end = 0x7ffc7e4bf560 "\001"
        uri = {user = {s = 0x7ffc7e4c0830 "\003", len = 938514842}, passwd = {s = 0xa00000000 <error: Cannot access memory at address 0xa00000000>, len = 22844112}, host = {
            s = 0x100000003 <error: Cannot access memory at address 0x100000003>, len = 681881128}, port = {s = 0x3 <error: Cannot access memory at address 0x3>, len = 0}, params = {
            s = 0x200000001 <error: Cannot access memory at address 0x200000001>, len = 2118906304}, sip_params = {s = 0x15c91c0 "", len = -1642256896}, headers = {
            s = 0x9aa9c8 <buf+8> "Y.Y.Y.Y:5080;prxroute=1 SIP/2.0\r\nRecord-Route: <sip:x.x.x.x;r2=on;lr=on;ftag=922229c699;ngcplb=yes;socket=udp:y.y.y.y:5060>\r\nRecord-Route: <sip:y.y.y.y;r2=on;lr=on;ftag=922229c699;ngcplb=y"..., len = 671147880}, port_no = 65535, proto = 65535, type = ERROR_URI_T, flags = (unknown: 683581728), transport = {s = 0x7ffc7e4c0830 "\003", len = 2118906688}, ttl = {
            s = 0x17b6aea "{\001", len = 4377518}, user_param = {s = 0x7f8c27f6ae20 "\001", len = 671137416}, maddr = {s = 0x7f8c28bea120 "+-\002", len = 671137528}, method = {
            s = 0x7e4bf280 <error: Cannot access memory at address 0x7e4bf280>, len = 623674360}, lr = {s = 0x7ffc7e4c0670 "\277\001", len = 2118906724}, r2 = {s = 0x7f8c00000000 "", len = 0}, gr = {
            s = 0x7ffc7e4bf4c4 "\214\177", len = -1642256896}, transport_val = {s = 0xd0de87bdb0391519 <error: Cannot access memory at address 0xd0de87bdb0391519>, len = 671137536}, ttl_val = {s = 0x0, 
            len = 671137416}, user_param_val = {s = 0x7f8c28bea120 "+-\002", len = 671137528}, maddr_val = {s = 0x7ffc7e4bf280 "", len = 4589069}, method_val = {s = 0x0, len = 519504614}, lr_val = {
            s = 0x10000000 <error: Cannot access memory at address 0x10000000>, len = 250335224}, r2_val = {s = 0xe9a188 <error: Cannot access memory at address 0xe9a188>, len = 18100232}, gr_val = {
            s = 0x11993f0 <error: Cannot access memory at address 0x11993f0>, len = 8}}
        next_hop = {user = {s = 0x0, len = 0}, passwd = {s = 0x1800000000 <error: Cannot access memory at address 0x1800000000>, len = 6243476}, host = {
            s = 0xe98ed8 <error: Cannot access memory at address 0xe98ed8>, len = -1642256896}, port = {s = 0x6afc59 <__FUNCTION__.8262> "rval_new_empty", len = 671154536}, params = {
            s = 0x7ffc7e4c0830 "\003", len = 683581728}, sip_params = {s = 0x7f8c2800e768 "\017", len = 683581728}, headers = {s = 0x17b6aea "{\001", len = 4479757}, port_no = 0, proto = 4096, 
          type = ERROR_URI_T, flags = (unknown: 936), transport = {s = 0x19 <error: Cannot access memory at address 0x19>, len = 683581728}, ttl = {s = 0x7ffc7e4c0830 "\003", len = 2118908048}, 
---Type <return> to continue, or q <return> to quit---
          user_param = {s = 0x7f8c287c7320 "\002", len = 4349792}, maddr = {s = 0x7ffc7e4bf350 "\224\363K~\374\177", len = 4349855}, method = {s = 0x7f8c00000000 "", len = 4362395}, lr = {
            s = 0x7ffc7e4bf394 "6", len = 1}, r2 = {s = 0x0, len = -1642256896}, gr = {s = 0x1141e58 <error: Cannot access memory at address 0x1141e58>, len = -1642256896}, transport_val = {
            s = 0x1d <error: Cannot access memory at address 0x1d>, len = 0}, ttl_val = {s = 0x3600000064 <error: Cannot access memory at address 0x3600000064>, len = 532}, user_param_val = {s = 0x0, 
            len = 1}, maddr_val = {s = 0x7f8c28bea120 "+-\002", len = 939161967}, method_val = {s = 0x0, len = 5665286}, lr_val = {s = 0x10000000 <error: Cannot access memory at address 0x10000000>, 
            len = 250337280}, r2_val = {s = 0xe99938 <error: Cannot access memory at address 0xe99938>, len = -1642256896}, gr_val = {s = 0x11993f0 <error: Cannot access memory at address 0x11993f0>, 
            len = 497}}
        u = 0x7ffc7e4bf560
        flags = 0
        st = {flags = 0, id = 0, name = {n = 0, s = {s = 0x0, len = 0}, re = 0x0}, avp = 0x5f4494 <fm_malloc+3044>}
        sct = 0x7ffc7e4bf560
        sjt = 0x7ffc7e4bf560
        mct = 0x17b6aea
        rv = 0x7f8c28a9d728
        rv1 = 0x7f8c28bea120
        c1 = {cache_type = 681970264, val_type = RV_NONE, c = {avp_val = {n = 683581728, s = {s = 0x7f8c28bea120 "+-\002", len = 0}, re = 0x7f8c28bea120}, pval = {rs = {s = 0x7f8c28bea120 "+-\002", 
                len = 0}, ri = 939035407, flags = 32652}}, i2s = '\000' <repeats 16 times>, "\270\361K~\374\177"}
        s = {s = 0x1ffffffff <error: Cannot access memory at address 0x1ffffffff>, len = 683581728}
        srevp = {0x7ffc7e4c0670, 0x7ffc7e4bf214}
        mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NUMBER_ST, u = {number = 2, 
              string = 0x2 <error: Cannot access memory at address 0x2>, str = {s = 0x2 <error: Cannot access memory at address 0x2>, len = 0}, data = 0x2, attr = 0x2, select = 0x2}}, {
            type = STRING_RVE_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 15}, data = 0x0, attr = 0x0, select = 0x0}}, {type = STRING_RVE_ST, u = {number = 0, string = 0x0, str = {s = 0x0, 
                len = 1}, data = 0x0, attr = 0x0, select = 0x0}}, {type = STRING_RVE_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 36}, data = 0x0, attr = 0x0, select = 0x0}}, {
            type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, 
              data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}}
        __FUNCTION__ = "do_action"
#5  0x0000000000445115 in run_actions (h=0x7f8b5fb23500, h@entry=0x7ffc7e4bf560, a=0x6, msg=0x1) at action.c:1549
        ret = 2118912048
        mod = 0x7f8c27f994c8
        __FUNCTION__ = "run_actions"
#6  0x0000000000451ae7 in run_actions_safe (h=0x7ffc7e4c0830, a=<optimized out>, msg=<optimized out>) at action.c:1614
        ctx = {rec_lev = 1, run_flags = 0, last_retcode = 0, jmp_env = {{__jmpbuf = {140722427393376, -3396127869322586855, 140239955730720, 140722427398192, 140722427394192, 24865514, 
                -3396127820052097767, 3397538908824868121}, __mask_was_saved = 0, __saved_mask = {__val = {5665521, 140722427393848, 140239955730720, 140722427393848, 7568605033954878976, 140722427393812, 
                  497, 140239942814231, 140239942814231, 140722427393744, 0, 99, 5666971, 15310176, 18099168, 18453488}}}}}
        ret = <optimized out>
        ign_on_break = 0
#7  0x0000000000426c70 in rval_get_int (h=0x7ffc7e4c0830, msg=0x7f8b5fb23500, i=0x7ffc7e4bf890, rv=0x0, cache=0x9d6460 <tcp_main_ltimer+112672>) at rvalue.c:912
        r_avp = 0x0
        avp_val = {n = 2118907856, s = {s = 0x7ffc7e4bf7d0 "", len = 5668051}, re = 0x7ffc7e4bf7d0}
        pval = {rs = {s = 0x6909162d0000001f <error: Cannot access memory at address 0x6909162d0000001f>, len = 683540480}, ri = 1, flags = 32764}
        tmp = {s = 0x7ffc00000001 <error: Cannot access memory at address 0x7ffc00000001>, len = 670665240}
        s = 0x7f8c28bea120
        r = 0
        ret = 0
        destroy_pval = 2118912048
        __FUNCTION__ = "rval_get_int"
#8  0x000000000042c9fc in rval_expr_eval_int (h=0x7ffc7e4c0830, msg=0x7f8c28bea120, res=0x7ffc7e4bf890, rve=0x7f8c27f995f8) at rvalue.c:1910
        i1 = 32652
        i2 = 1
        ret = -1
        c1 = {cache_type = 670665240, val_type = 32652, c = {avp_val = {n = 31, s = {s = 0x6909162d0000001f <error: Cannot access memory at address 0x6909162d0000001f>, len = 670994688}, 
              re = 0x6909162d0000001f}, pval = {rs = {s = 0x6909162d0000001f <error: Cannot access memory at address 0x6909162d0000001f>, len = 670994688}, ri = -1642256896, flags = 1762203181}}, 
          i2s = "\000\000\000\000\004\000\000\000ȋ\371'\214\177\000\000\000\000\000\000\000"}
        c2 = {cache_type = 683581728, val_type = 32652, c = {avp_val = {n = 670665664, s = {s = 0x7f8c27f98bc0 "\001", len = 670665672}, re = 0x7f8c27f98bc0}, pval = {rs = {s = 0x7f8c27f98bc0 "\001", 
                len = 670665672}, ri = 4589069, flags = 0}}, i2s = "\304\023\000\000\001\000\000\000\260\211\371'\214\177\000\000\000\000\000\000\000"}
        rv1 = 0x7f8c252c7f66 <pv_set_avp+1270>
        rv2 = 0x10000017b6aea
        __FUNCTION__ = "rval_expr_eval_int"
#9  0x0000000000446b27 in do_action (h=0x7ffc7e4c0830, a=0x7f8c27fe6830, msg=0x7f8c28bea120) at action.c:1030
        ret = -5
---Type <return> to continue, or q <return> to quit---
        v = 0
        dst = {send_sock = 0x7f8c28bea120, to = {s = {sa_family = 7, sa_data = "\000\000\000\000\000\000\260\376K~\374\177\000"}, sin = {sin_family = 7, sin_port = 0, sin_addr = {s_addr = 0}, 
              sin_zero = "\260\376K~\374\177\000"}, sin6 = {sin6_family = 7, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 = "\260\376K~\374\177\000\000 \241\276(\214\177\000", 
                  __u6_addr16 = {65200, 32331, 32764, 0, 41248, 10430, 32652, 0}, __u6_addr32 = {2118909616, 32764, 683581728, 32652}}}, sin6_scope_id = 24865514}}, id = 0, proto = 91 '[', send_flags = {
            f = 119 'w', blst_imask = 87 'W'}}
        end = 0x7f8c27f995f8 "\001"
        uri = {user = {s = 0x7ffc7e4bfeb0 " \241\276(\214\177", len = 683581728}, passwd = {s = 0x7ffc7e4bfeb0 " \241\276(\214\177", len = 683581728}, host = {s = 0x17b6aea "{\001", len = 4482640}, 
          port = {s = 0x7f8c2818fe58 "\220\001", len = 687376112}, params = {s = 0x7ffc7e4c0830 "\003", len = -5}, sip_params = {s = 0x7ffc7e4bfab0 "", len = 683581728}, headers = {
            s = 0x1666830 "\220m\177\001", len = 2118909840}, port_no = 13540, proto = 14338, type = 32652, flags = (URI_USER_NORMALIZE | URI_SIP_USER_PHONE | unknown: 939668708), transport = {s = 0x0, 
            len = 938508137}, ttl = {s = 0x100000000 <error: Cannot access memory at address 0x100000000>, len = 4349792}, user_param = {s = 0x7ffc7e4bfb20 "", len = 4349855}, maddr = {
            s = 0x7f8c287c7320 "\002", len = 4362395}, method = {s = 0x7f8b5fb95036 "udp", len = 3}, lr = {s = 0x7f8c37f0999a <vfprintf+22490> "\200\275(\373\377\377", len = 2118909824}, r2 = {
            s = 0x7f8c287c7320 "\002", len = 540189421}, gr = {s = 0x7f8c20547260 <log_flag> "\377\377\377\377\001", len = 4349855}, transport_val = {s = 0x7f8c285b4f00 "\002", len = 4362395}, ttl_val = {
            s = 0x7ffc00000000 <error: Cannot access memory at address 0x7ffc00000000>, len = -5}, user_param_val = {s = 0x4b37 <error: Cannot access memory at address 0x4b37>, len = 10}, maddr_val = {
            s = 0x10000020 <error: Cannot access memory at address 0x10000020>, len = 938689182}, method_val = {s = 0x1666830 "\220m\177\001", len = 687260760}, lr_val = {s = 0x7ffc7e4c0000 "", 
            len = 683581728}, r2_val = {s = 0x7ffc7e4c0000 "", len = 683581728}, gr_val = {s = 0x17b6aea "{\001", len = 4482640}}
        next_hop = {user = {s = 0x7ffc7e4bfc00 "Y\374j", len = -1642256896}, passwd = {s = 0x0, len = 687213120}, host = {s = 0x7ffc7e4c0830 "\003", len = -5}, port = {s = 0x7ffc7e4bfc10 "", 
            len = 683581728}, params = {s = 0x17b6aea "{\001", len = 4477573}, sip_params = {s = 0x7ffc7e4bfc10 "", len = 2118908992}, headers = {s = 0x7f8c28f60a40 "\334\001", len = 683581728}, 
          port_no = 0, proto = 0, type = SIP_URI_T, flags = (unknown: 0), transport = {s = 0x7ffc00000000 <error: Cannot access memory at address 0x7ffc00000000>, len = 0}, ttl = {
            s = 0x7ffc7e4c0020 " \241\276(\214\177", len = 0}, user_param = {s = 0x0, len = 0}, maddr = {s = 0x0, len = 938514842}, method = {s = 0x7f8c37f0999a <vfprintf+22490> "\200\275(\373\377\377", 
            len = 0}, lr = {s = 0x0, len = 0}, r2 = {s = 0xffffffffffffffff <error: Cannot access memory at address 0xffffffffffffffff>, len = 519504614}, gr = {
            s = 0x10000000 <error: Cannot access memory at address 0x10000000>, len = 250336336}, transport_val = {s = 0xe99d30 <error: Cannot access memory at address 0xe99d30>, len = 18099120}, 
          ttl_val = {s = 0x11993f0 <error: Cannot access memory at address 0x11993f0>, len = 8}, user_param_val = {s = 0x19b <error: Cannot access memory at address 0x19b>, len = -1642256896}, 
          maddr_val = {s = 0x1142bb0 <error: Cannot access memory at address 0x1142bb0>, len = 6243476}, method_val = {s = 0x7ffc7e4c0830 "\003", len = 6978301}, lr_val = {
            s = 0x6afc59 <__FUNCTION__.8262> "rval_new_empty", len = -1642256896}, r2_val = {s = 0x0, len = 687224008}, gr_val = {s = 0x7ffc7e4c0830 "\003", len = -5}}
        u = 0x7f8c27f995f8
        flags = 0
        st = {flags = 2118908928, id = 32764, name = {n = -1642256896, s = {s = 0x6909162d9e1d2600 <error: Cannot access memory at address 0x6909162d9e1d2600>, len = 0}, re = 0x6909162d9e1d2600}, 
          avp = 0x7f8c28f60a40}
        sct = 0x7f8c27f995f8
        sjt = 0x7f8c27f995f8
        mct = 0x17b6aea
        rv = 0x0
        rv1 = 0x7f8c28bea120
        c1 = {cache_type = 687185704, val_type = 32652, c = {avp_val = {n = -1642256896, s = {s = 0x6909162d9e1d2600 <error: Cannot access memory at address 0x6909162d9e1d2600>, len = 683581728}, 
              re = 0x6909162d9e1d2600}, pval = {rs = {s = 0x6909162d9e1d2600 <error: Cannot access memory at address 0x6909162d9e1d2600>, len = 683581728}, ri = 542603232, flags = 32652}}, 
          i2s = "\360\223\031\001F\000\000\000\000&\035\236-\026\ti\243\001\000\000\000"}
        s = {s = 0x186b4f57011424d8 <error: Cannot access memory at address 0x186b4f57011424d8>, len = -1642256896}
        srevp = {0x9aad1a <buf+858>, 0x5e7}
        mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NUMBER_ST, u = {number = 2, 
              string = 0x2 <error: Cannot access memory at address 0x2>, str = {s = 0x2 <error: Cannot access memory at address 0x2>, len = 0}, data = 0x2, attr = 0x2, select = 0x2}}, {
            type = STRING_RVE_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 15}, data = 0x0, attr = 0x0, select = 0x0}}, {type = STRING_RVE_ST, u = {number = 0, string = 0x0, str = {s = 0x0, 
                len = 1}, data = 0x0, attr = 0x0, select = 0x0}}, {type = STRING_RVE_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 36}, data = 0x0, attr = 0x0, select = 0x0}}, {
            type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, 
              data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}}
        __FUNCTION__ = "do_action"
#10 0x0000000000445115 in run_actions (h=0x7f8b5fb23500, a=0x6, msg=0x1) at action.c:1549
        ret = 1
        mod = 0x7f8c27fe6830
        __FUNCTION__ = "run_actions"
#11 0x0000000000446d40 in do_action (h=0x7ffc7e4c0830, a=0x7f8c288d33a8, msg=0x7f8c28bea120) at action.c:678
        ret = -5
        v = 0
        dst = {send_sock = 0x17b6aab, to = {s = {sa_family = 21125, sa_data = "D\000\000\000\000\000\300y\020(\214\177\000"}, sin = {sin_family = 21125, sin_port = 68, sin_addr = {s_addr = 0}, 
              sin_zero = "\300y\020(\214\177\000"}, sin6 = {sin6_family = 21125, sin6_port = 68, sin6_flowinfo = 0, sin6_addr = {__in6_u = {
                  __u6_addr8 = "\300y\020(\214\177\000\000/\000\000\000\000\000\000", __u6_addr16 = {31168, 10256, 32652, 0, 47, 0, 0, 0}, __u6_addr32 = {672168384, 32652, 47, 0}}}, 
              sin6_scope_id = 980445555}}, id = 32652, proto = 0 '\000', send_flags = {f = 0 '\000', blst_imask = 0 '\000'}}
        end = 0x6 <error: Cannot access memory at address 0x6>
        uri = {user = {s = 0x7ffc7e4c0064 "", len = 6404846}, passwd = {s = 0x7ffc7e4c0830 "\003", len = 2118910388}, host = {s = 0x17b6aea "{\001", len = -1642256896}, port = {
            s = 0x7f8c28bea120 "+-\002", len = 687260760}, params = {s = 0x7ffc7e4c0830 "\003", len = -5}, sip_params = {s = 0x7ffc7e4c0000 "", len = 683581728}, headers = {s = 0x17b6aea "{\001", 
---Type <return> to continue, or q <return> to quit---
            len = 4477573}, port_no = 0, proto = 32332, type = 32764, flags = (URI_USER_NORMALIZE | unknown: 4212331800), transport = {s = 0x7f8c28f6c458 ")\002", len = 683581728}, ttl = {
            s = 0x100000000 <error: Cannot access memory at address 0x100000000>, len = 683581728}, user_param = {s = 0x7ffc7e4c0088 "", len = 2118910056}, maddr = {s = 0x7ffc7e4c0064 "", len = 5800526}, 
          method = {s = 0x7f8c28f68630 "\001", len = 683581728}, lr = {s = 0x7ffc7e4c0830 "\003", len = 519504614}, r2 = {s = 0x10000000 <error: Cannot access memory at address 0x10000000>, 
            len = 250336336}, gr = {s = 0xe99d30 <error: Cannot access memory at address 0xe99d30>, len = 18099120}, transport_val = {s = 0x11993f0 <error: Cannot access memory at address 0x11993f0>, 
            len = 8}, ttl_val = {s = 0x19b <error: Cannot access memory at address 0x19b>, len = -1642256896}, user_param_val = {s = 0x7f8c287c7320 "\002", len = 6233684}, maddr_val = {
            s = 0x7ffc7e4c0080 "", len = 679244504}, method_val = {s = 0x7f8c38263620 "", len = 0}, lr_val = {s = 0x7ffc7e4c0830 "\003", len = 683581728}, r2_val = {s = 0x7ffc7e4c0830 "\003", 
            len = 2118910768}, gr_val = {s = 0x17b6aea "{\001", len = 4528871}}
        next_hop = {user = {s = 0x0, len = 1}, passwd = {s = 0x7ffc7e4c0000 "", len = 1094259993}, host = {s = 0x7f8c28bea120 "+-\002", len = 2118912048}, port = {s = 0x7ffc7e4c0330 "\001", 
            len = 24865514}, params = {s = 0xd0de87bdacb91519 <error: Cannot access memory at address 0xd0de87bdacb91519>, len = -82635495}, sip_params = {s = 0x0, len = 544780589}, headers = {
            s = 0x7e4c0168 <error: Cannot access memory at address 0x7e4c0168>, len = 687244856}, port_no = 36328, proto = 10486, type = 32652, 
          flags = (URI_USER_NORMALIZE | URI_SIP_USER_PHONE | unknown: 939139724), transport = {s = 0x0, len = 0}, ttl = {s = 0x0, len = 0}, user_param = {s = 0x7ffc7e4c0140 "h\345\230 \214\177", 
            len = 2118910240}, maddr = {s = 0x7f8c3a706973 <error: Cannot access memory at address 0x7f8c3a706973>, len = 683581728}, method = {s = 0x1816f60 "\340hm\001", len = 0}, lr = {
            s = 0x40 <error: Cannot access memory at address 0x40>, len = -1642256896}, r2 = {s = 0x300000039 <error: Cannot access memory at address 0x300000039>, len = 2118910768}, gr = {
            s = 0x7ffc7e4c0830 "\003", len = 4353136}, transport_val = {s = 0x1000000000000 <error: Cannot access memory at address 0x1000000000000>, len = 7200}, ttl_val = {s = 0x1541c40 "CEST", 
            len = 623629966}, user_param_val = {s = 0x7f8b5fc10115 "5060", len = 10498376}, maddr_val = {s = 0x9a9790 <_km_log_func> "\340jA", len = 0}, method_val = {
            s = 0x7f8c2098e568 <_xlog_prefix> "\242\263x \214\177", len = -1642256896}, lr_val = {s = 0x416ae0 <syslog@plt> "\377%\222\310X", len = 687262272}, r2_val = {
            s = 0xffffffff <error: Cannot access memory at address 0xffffffff>, len = 683581728}, gr_val = {s = 0x7ffc7e4c0830 "\003", len = 2118910768}}
        u = 0x6
        flags = 32652
        st = {flags = 0, id = 0, name = {n = 1, s = {s = 0x7f8c00000001 "", len = 2118909952}, re = 0x7f8c00000001}, avp = 0xd0de87b241391519}
        sct = 0x6
        sjt = 0x6
        mct = 0x17b6aea
        rv = 0x7f8c28f79ba8
        rv1 = 0x7f8c28bea120
        c1 = {cache_type = RV_CACHE_EMPTY, val_type = RV_NONE, c = {avp_val = {n = 5060, s = {s = 0x29000013c4 <error: Cannot access memory at address 0x29000013c4>, len = 2118912048}, re = 0x29000013c4}, 
            pval = {rs = {s = 0x29000013c4 <error: Cannot access memory at address 0x29000013c4>, len = 2118912048}, ri = 672168304, flags = 32652}}, 
          i2s = "\270y\020(\214\177\000\000\210\000L~\374\177\000\000\210\000L~\374\177"}
        s = {s = 0x7f8c287c766e "4", len = 3}
        srevp = {0x7ffc7e4c0670, 0x7f8c28bea120}
        mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NUMBER_ST, u = {number = 2, 
              string = 0x2 <error: Cannot access memory at address 0x2>, str = {s = 0x2 <error: Cannot access memory at address 0x2>, len = 0}, data = 0x2, attr = 0x2, select = 0x2}}, {
            type = STRING_RVE_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 15}, data = 0x0, attr = 0x0, select = 0x0}}, {type = STRING_RVE_ST, u = {number = 0, string = 0x0, str = {s = 0x0, 
                len = 1}, data = 0x0, attr = 0x0, select = 0x0}}, {type = STRING_RVE_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 36}, data = 0x0, attr = 0x0, select = 0x0}}, {
            type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, 
              data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}}
        __FUNCTION__ = "do_action"
#12 0x0000000000445115 in run_actions (h=0x7f8b5fb23500, a=0x6, msg=0x1) at action.c:1549
        ret = 2118912048
        mod = 0x7f8c28f6c808
        __FUNCTION__ = "run_actions"
#13 0x0000000000446b78 in do_action (h=0x7ffc7e4c0830, a=0x7f8c28f6ee18, msg=0x7f8c28bea120) at action.c:1049
        ret = 1
        v = 1
        dst = {send_sock = 0x7f8c28bea120, to = {s = {sa_family = 51646, sa_data = "\334%\214\177\000\000\264\000\000\000\374\177\000"}, sin = {sin_family = 51646, sin_port = 9692, sin_addr = {
                s_addr = 32652}, sin_zero = "\264\000\000\000\374\177\000"}, sin6 = {sin6_family = 51646, sin6_port = 9692, sin6_flowinfo = 32652, sin6_addr = {__in6_u = {
                  __u6_addr8 = "\264\000\000\000\374\177\000\000\000\000\000\000\000\000\000", __u6_addr16 = {180, 0, 32764, 0, 0, 0, 0, 0}, __u6_addr32 = {180, 32764, 0, 0}}}, 
              sin6_scope_id = 674766208}}, id = 32652, proto = 32 ' ', send_flags = {f = 161 '\241', blst_imask = 190 '\276'}}
        end = 0x7f8c28f6ca40 "\001"
        uri = {user = {s = 0x7ffc7e4c05dc "", len = 635512766}, passwd = {s = 0x7f8b5fdb1890 "\224*\002", len = 683581728}, host = {s = 0x5fcdb5d8 <error: Cannot access memory at address 0x5fcdb5d8>, 
            len = 0}, port = {s = 0x0, len = 0}, params = {s = 0x0, len = 0}, sip_params = {s = 0x4881 <error: Cannot access memory at address 0x4881>, len = -1642256896}, headers = {
            s = 0x6 <error: Cannot access memory at address 0x6>, len = 634972939}, port_no = 180, proto = 0, type = 32651, flags = (unknown: 2118911600), transport = {s = 0x0, len = 681887176}, ttl = {
            s = 0x7f8c28bea120 "+-\002", len = 1610866240}, user_param = {s = 0x7f8b6003dc38 "\240\377h_\213\177", len = 4529039}, maddr = {s = 0x0, len = 0}, method = {
            s = 0x4 <error: Cannot access memory at address 0x4>, len = 635512766}, lr = {s = 0x0, len = 683581728}, r2 = {s = 0x7f8b5ff2fff8 "", len = 200}, gr = {s = 0x0, len = 683581728}, 
          transport_val = {s = 0x7f8c28837df0 <incomplete sequence \350>, len = 638037828}, ttl_val = {s = 0x7ffc7e4c05dc "", len = -1642256896}, user_param_val = {
            s = 0x7f8c260671f0 <tm_cfg> "h\371\220_\213\177", len = 634977214}, maddr_val = {s = 0x7f8b00000004 <error: Cannot access memory at address 0x7f8b00000004>, len = 1609884936}, method_val = {
            s = 0x4 <error: Cannot access memory at address 0x4>, len = 683581728}, lr_val = {s = 0xc8 <error: Cannot access memory at address 0xc8>, len = 1609885456}, r2_val = {s = 0x7f8b5ff4e508 "", 
            len = 634984442}, gr_val = {s = 0x0, len = -1642256896}}
        next_hop = {user = {s = 0x7f8c28bea120 "+-\002", len = 2118911456}, passwd = {s = 0x5 <error: Cannot access memory at address 0x5>, len = 683581728}, host = {
---Type <return> to continue, or q <return> to quit---
            s = 0xb4 <error: Cannot access memory at address 0xb4>, len = 1610866240}, port = {s = 0x7f8b6003dc38 "\240\377h_\213\177", len = 635245418}, params = {
            s = 0x7ffc000000b7 <error: Cannot access memory at address 0x7ffc000000b7>, len = 10493856}, sip_params = {s = 0xa01fa8 <def_list+8> "", len = 10493872}, headers = {
            s = 0xa01fb8 <def_list+24> "", len = 10493896}, port_no = 7448, proto = 160, type = ERROR_URI_T, flags = (URI_SIP_USER_PHONE | unknown: 519504612), transport = {
            s = 0x10000000 <error: Cannot access memory at address 0x10000000>, len = 519504614}, ttl = {s = 0x0, len = 250281984}, user_param = {s = 0x0, len = 519504614}, maddr = {
            s = 0x10000000 <error: Cannot access memory at address 0x10000000>, len = 250338176}, method = {s = 0xe98c70 <error: Cannot access memory at address 0xe98c70>, len = 18097280}, lr = {
            s = 0x11993f0 <error: Cannot access memory at address 0x11993f0>, len = 8}, r2 = {s = 0x1dc <error: Cannot access memory at address 0x1dc>, len = 519504614}, gr = {
            s = 0x10000000 <error: Cannot access memory at address 0x10000000>, len = 250341208}, transport_val = {s = 0xe989e0 <error: Cannot access memory at address 0xe989e0>, len = 18094248}, 
          ttl_val = {s = 0x11993f0 <error: Cannot access memory at address 0x11993f0>, len = 8}, user_param_val = {s = 0x1bf <error: Cannot access memory at address 0x1bf>, len = -1642256896}, 
          maddr_val = {s = 0x11418a8 <error: Cannot access memory at address 0x11418a8>, len = 6243476}, method_val = {s = 0x7f8c28a4c5c8 "\231\034", len = 6978301}, lr_val = {
            s = 0x71c670 <__FUNCTION__.7459> "parse_via", len = 1610865720}, r2_val = {s = 0xd0de87b257b91519 <error: Cannot access memory at address 0xd0de87b257b91519>, len = -82635495}, gr_val = {
            s = 0x0, len = -1642256896}}
        u = 0x7f8c28f6ca40
        flags = 32651
        st = {flags = 683581728, id = 32652, name = {n = 2118911456, s = {s = 0x7ffc7e4c05e0 "", len = 5}, re = 0x7ffc7e4c05e0}, avp = 0x7f8c28bea120}
        sct = 0x7f8c28f6ca40
        sjt = 0x7f8c28f6ca40
        mct = 0x17b6aea
        rv = 0x1
        rv1 = 0x7f8c28bea120
        c1 = {cache_type = 180, val_type = RV_NONE, c = {avp_val = {n = 1610865912, s = {s = 0x7f8b6003dcf8 "\264", len = 10482636}, re = 0x7f8b6003dcf8}, pval = {rs = {s = 0x7f8b6003dcf8 "\264", 
                len = 10482636}, ri = 5, flags = 1}}, i2s = "\340\005L~\374\177\000\000\070\334\003`\213\177\000\000\240\377h_\213\177"}
        s = {s = 0xa48980 <_pkg_root> "\344\016p", len = 808857653}
        srevp = {0x7f8c27ede430, 0x7f8b6003de40}
        mod_f_params = {{type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NUMBER_ST, u = {number = 2, 
              string = 0x2 <error: Cannot access memory at address 0x2>, str = {s = 0x2 <error: Cannot access memory at address 0x2>, len = 0}, data = 0x2, attr = 0x2, select = 0x2}}, {
            type = STRING_RVE_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 15}, data = 0x0, attr = 0x0, select = 0x0}}, {type = STRING_RVE_ST, u = {number = 0, string = 0x0, str = {s = 0x0, 
                len = 1}, data = 0x0, attr = 0x0, select = 0x0}}, {type = STRING_RVE_ST, u = {number = 0, string = 0x0, str = {s = 0x0, len = 36}, data = 0x0, attr = 0x0, select = 0x0}}, {
            type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, 
              data = 0x0, attr = 0x0, select = 0x0}}, {type = NOSUBTYPE, u = {number = 0, string = 0x0, str = {s = 0x0, len = 0}, data = 0x0, attr = 0x0, select = 0x0}}}
        __FUNCTION__ = "do_action"
#14 0x0000000000445115 in run_actions (h=0x7f8b5fb23500, h@entry=0x7ffc7e4c0830, a=0x6, a@entry=0x7f8c28f1b930, msg=0x1, msg@entry=0x7f8c28bea120) at action.c:1549
        ret = 1
        mod = 0x7f8c28f6ee18
        __FUNCTION__ = "run_actions"
#15 0x0000000000451b85 in run_top_route (a=0x7f8c28f1b930, msg=0x7f8c28bea120, c=<optimized out>) at action.c:1635
        ctx = {rec_lev = 3, run_flags = 0, last_retcode = 1, jmp_env = {{__jmpbuf = {140722427398192, 3397538390581253401, 140239959079216, 140239955730720, 0, 0, -3396127868821367527, 
                3397538908824868121}, __mask_was_saved = 0, __saved_mask = {__val = {10136323, 1354, 7568605033954878976, 10136001, 6243476, 140239948066720, 140239955730720, 10136044, 10136000, 1354, 
                  7568605033954878976, 0, 140239960257184, 1073741825, 140239907703544, 140239960247216}}}}}
        p = <optimized out>
        ret = <optimized out>
        sfbk = 0
#16 0x0000000000556f9e in receive_msg (buf=0x0, len=683581728, rcv_info=0x7ffc7e4c0b10) at receive.c:240
        msg = 0x7f8c28bea120
        ctx = {rec_lev = 23098816, run_flags = 0, last_retcode = 23098816, jmp_env = {{__jmpbuf = {0, 6, 140236580472680, 140236549023960, 140722427398688, 140239791653606, 268435456, 250568360}, 
              __mask_was_saved = 15165008, __saved_mask = {__val = {10105736, 0, 140722427398950, 5, 1, 140239942050864, 140240218561991, 5, 0, 7063492, 140240210401128, 140239955456248, 140239942050864, 
                  140240220669584, 140240218587877, 140240220669584}}}}}
        inb = {
          s = 0x9aa9c0 <buf> "BYE sip:Y.Y.Y.Y:5080;prxroute=1 SIP/2.0\r\nRecord-Route: <sip:x.x.x.x;r2=on;lr=on;ftag=922229c699;ngcplb=yes;socket=udp:y.y.y.y:5060>\r\nRecord-Route: <sip:y.y.y.y;r2=on;lr=on;ftag=922229c699;"..., len = 1354}
        netinfo = {data = {s = 0xffffffff <error: Cannot access memory at address 0xffffffff>, len = 1571855534}, dst = 0x7f8c37ec9768, rcv = 0x7f8c38895000}
        __FUNCTION__ = "receive_msg"
#17 0x0000000000479f10 in udp_rcv_loop () at udp_server.c:495
        len = 1354
        buf = "BYE sip:Y.Y.Y.Y:5080;prxroute=1 SIP/2.0\r\nRecord-Route: <sip:x.x.x.x;r2=on;lr=on;ftag=922229c699;ngcplb=yes;socket=udp:y.y.y.y:5060>\r\nRecord-Route: <sip:y.y.y.y;r2=on;lr=on;ftag=922229c699;"...
        from = 0x7f8c28ba70f8
        fromlen = 16
        ri = {src_ip = {af = 2, len = 4, u = {addrl = {234897418, 140236580472680}, addr32 = {234897418, 0, 1603290984, 32651}, addr16 = {16394, 3584, 0, 0, 18280, 24464, 32651, 0}, 
              addr = "\n@\000\016\000\000\000\000hG\220_\213\177\000"}}, dst_ip = {af = 2, len = 4, u = {addrl = {335560714, 0}, addr32 = {335560714, 0, 0, 0}, addr16 = {16394, 5120, 0, 0, 0, 0, 0, 0}, 
---Type <return> to continue, or q <return> to quit---
              addr = "\n@\000\024", '\000' <repeats 11 times>}}, src_port = 5060, dst_port = 5062, proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {sa_family = 2, 
              sa_data = "\023\304\n@\000\016\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 234897418}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {
              sin6_family = 2, sin6_port = 50195, sin6_flowinfo = 234897418, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 
                    0}}}, sin6_scope_id = 0}}, bind_address = 0x7f8c27ede430, proto = 1 '\001'}
        __FUNCTION__ = "udp_rcv_loop"
#18 0x00000000004ffe39 in main_loop () at main.c:1614
        si_desc = "udp receiver child=5 sock=Y.Y.Y.Y:5062\000\000\000\001\000\000\000pX\300(\214\177\000\000\000\000\000\000\000\000\000\000n\000\000\000\000\000\000\000\315f_\000\000\000\000\000\003\000\000\000\000\000\000\000\000&\035\236-\026\ti\000\000\000\000\000\000\000\000\000&\035\236-\026\ti\310\002\245\000\000\000\000\000\005\000\000\000\000\000\000"
        woneinit = 1
        __FUNCTION__ = "main_loop"
#19 0x000000000041cd6c in main (argc=0, argv=0x0) at main.c:2631
        cfg_stream = 0x0
        r = 0
        tmp = 0x7ffc7e4c2ee4 ""
        tmp_len = 0
        port = 0
        proto = 948524496
        ret = 0
        seed = 1296618944
        debug_save = 0
        debug_flag = 4
        dont_fork_cnt = 0
        st = {st_dev = 15, st_ino = 122623671, st_nlink = 2, st_mode = 16877, st_uid = 106, st_gid = 108, __pad0 = 0, st_rdev = 0, st_size = 60, st_blksize = 4096, st_blocks = 0, st_atim = {
            tv_sec = 1488564228, tv_nsec = 817944387}, st_mtim = {tv_sec = 1490749503, tv_nsec = 811780252}, st_ctim = {tv_sec = 1490749503, tv_nsec = 811780252}, __glibc_reserved = {0, 0, 0}}
        __FUNCTION__ = "main"
(gdb) 

Log Messages

Mar 29 09:02:14 sp2 proxy[19254]: NOTICE: <script>: New request on proxy - M=ACK R=sip:Y.Y.Y.Y:5080;prxroute=1 F=sip:anumber@sipwise.hello.local:5060 T=sip:bnumber@sipwise.hello.local:5060 IP=X.X.X.X:5060 (x.x.x.x:5060) ID=bf8e1a679ee462a4 UA='EPC2425-110715'
Mar 29 09:03:57 sp2 proxy[19255]: NOTICE: <script>: New request on proxy - M=BYE R=sip:Y.Y.Y.Y:5080;prxroute=1 F=sip:anumber@sipwise.hello.local:5060 T=sip:bnumber@sipwise.hello.local:5060 IP=X.X.X.X:5060 (x.x.x.x:5060) ID=bf8e1a679ee462a4 UA='EPC2425-110715'
Mar 29 09:03:57 sp2 proxy[19257]: NOTICE: <script>: New request on proxy - M=BYE R=sip:Y.Y.Y.Y:5080;prxroute=1 F=sip:anumber@sipwise.hello.local:5060 T=sip:bnumber@sipwise.hello.local:5060 IP=X.X.X.X:5060 (x.x.x.x:5060) ID=bf8e1a679ee462a4 UA='EPC2425-110715'
Mar 29 09:03:57 sp2 proxy[19257]: NOTICE: <script>: NAT-Reply - S=200 - OK M=BYE IP=X.X.X.X:5060 (Y.Y.Y.Y:5080) ID=bf8e1a679ee462a4 UA='<null>'

Additional Information

  • Kamailio Version - output of kamailio -v
version: kamailio 4.4.4 (x86_64/linux) 
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown 
compiled with gcc 4.9.2

This is NGCP based on 4.4.4 with tm deep cloning edb9df8 backported

@linuxmaniac linuxmaniac added the 4.4 label Apr 5, 2017
@miconda
Copy link
Member

miconda commented Apr 5, 2017

Any idea why frames 1 and 2 don't have the symbols? Was there a single core file, presuming the options for core files per pid was set?

@linuxmaniac
Copy link
Member Author

Any idea why frames 1 and 2 don't have the symbols? Was there a single core file, presuming the options for core files per pid was set?

No idea, per-process core file is enabled too.

# cat /proc/sys/kernel/core_uses_pid
1

@linuxmaniac
Copy link
Member Author

linuxmaniac commented Apr 6, 2017

same thing in another customer

#0  0x00007fae3f1f7e14 in strnlen () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1  0x00007fae3f1c0b19 in vfprintf () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#2  0x00007fae3f25a216 in __vsyslog_chk () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#3  0x00007fae3f25a36f in syslog () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#4  0x00007fae32c3457e in dlg_onroute (req=0x7fae3def7ac8, route_params=0x7fad006e9b1e, 
    param=0x7fff717dab10) at dlg_handlers.c:1350
#0  0x00007fae3f1f7e14 in strnlen () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fae3f1c0b19 in vfprintf () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007fae3f25a216 in __vsyslog_chk () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007fae3f25a36f in syslog () from /lib/x86_64-linux-gnu/libc.so.6
#4  0x00007fae32c3457e in dlg_onroute (req=0x7fae3def7ac8, route_params=0x7fad006e9b1e, 
    param=0x7fff717dab10) at dlg_handlers.c:1350
#5  0x00007fae3a7ce28d in run_rr_callbacks (req=0x7fae3def7ac8, rr_param=0x7fff717daec0)
    at rr_cb.c:96
#6  0x00007fae3a7c08a2 in after_loose (_m=0x7fae3def7ac8, preloaded=2) at loose.c:928

@miconda
Copy link
Member

miconda commented Apr 6, 2017

Can you do:

  • frame 4
  • list
  • info locals

The code in branch 4.4 for dlg_handlers.c:1350 is:

			LM_WARN("inconsitent dlg timer data on dlg %p [%u:%u] "
				"with clid '%.*s' and tags '%.*s' '%.*s'\n",
				dlg, dlg->h_entry, dlg->h_id,
				dlg->callid.len, dlg->callid.s,
				dlg->tag[DLG_CALLER_LEG].len, dlg->tag[DLG_CALLER_LEG].s,
				dlg->tag[DLG_CALLEE_LEG].len, dlg->tag[DLG_CALLEE_LEG].s);

So no strlen() to be used there.

@linuxmaniac
Copy link
Member Author

This is the NGCP version of 4.4.5
sources at https://github.com/sipwise/kamailio/tree/mr4.5.4 but be aware that list of patches are at https://github.com/sipwise/kamailio/blob/mr4.5.4/debian/patches/series
and I changed the sensible data from the paste

In order you to see the source easily I pasted dlg_handlers.c already patched at https://paste.debian.net/926240/

(gdb) f 4
#4  0x00007fae32c3457e in dlg_onroute (req=0x7fae3def7ac8, route_params=0x7fad006e9b1e, 
    param=0x7fff717dab10) at dlg_handlers.c:1350
1350	dlg_handlers.c: No such file or directory.
(gdb) info locals
val = {
  s = 0x9aaf79 <buf+1465> "c68.0d81;ice_caller=strip;ice_callee=strip;aset=50;rtpprx=yes;vsf=Y3Z5enJeQmp8bVRNYGQtEwxdZHR2LHRwc1RAaX0hWEFyMw1mZg-->\r\nP-NGCP-Src-Ip: 172.17.65.99\r\nP-NGCP-Src-Port: 5060\r\nP-NGCP-Src-Proto: udp\r\nP-NG"..., len = 8}
ftag = {
  s = 0x9aabaa <buf+490> "C03675F3E457BC60\r\nTo: <sip:00000000000@sip.local:5060>;tag=0FC4E4CA-58E4A7D4000A8034-123E3700\r\nCall-ID: 3A477FD70434C083@10.10.10.18\r\nCSeq: 5709 BYE\r\nProxy-Authorization: Digest username=\"2233"..., len = 16}
old_state = 4
dir = 1
ret = 0
callid = {
  s = 0x9aac1a <buf+602> "3A477FD70434C083@10.10.10.18\r\nCSeq: 5709 BYE\r\nProxy-Authorization: Digest username=\"1111111111\", realm=\"sip.local\", nonce=\"WOUBrFjkp9Qj4YjEwa9jvelH7ty68fD/\", uri=\"sip:00000000000@172.17.66.118"..., len = 28}
new_state = 5
unref = 1
event = 10498440
dlg = 0x7fad74f20258
ttag = {
  s = 0x9aabed <buf+557> "0FC4E4CA-58E4A7D4000A8034-123E3700\r\nCall-ID: 3A477FD70434C083@10.10.10.18\r\nCSeq: 5709 BYE\r\nProxy-Authorization: Digest username=\"1111111111\", realm=\"sip.local\", nonce=\"WOUBrFjkp9Qj4YjEwa9jvelH"..., len = 34}
h_id = 854147520
dlg = 0x7fad00000000

So relevant code is too

			LM_WARN("inconsitent dlg timer data on dlg %p [%u:%u] "
				"with clid '%.*s' and tags '%.*s' '%.*s'\n",
				dlg, dlg->h_entry, dlg->h_id,
				dlg->callid.len, dlg->callid.s,
				dlg->tag[DLG_CALLER_LEG].len, dlg->tag[DLG_CALLER_LEG].s,
				dlg->tag[DLG_CALLEE_LEG].len, dlg->tag[DLG_CALLEE_LEG].s);

So no strlen() to be used there.

but the strlen() is coming from syslog()

I would say the problem is dlg is gone from memory when trying to use the string

@miconda
Copy link
Member

miconda commented Apr 6, 2017

Did you put in pastebin the wrong file (like dlg_db_handlers.c)?

All the tokens provided by kamailio in that log message are by length and pointer (%.*s), so there should be no strlen done on the pointers we set, so not sure on what syslog can do the strlen and crash.

@miconda
Copy link
Member

miconda commented Apr 6, 2017

I wonder if there is a copy/paste issue or the dlg is printed twice by info locals:

event = 10498440
dlg = 0x7fad74f20258
ttag = {
  s = 0x9aabed <buf+557> "0FC4E4CA-58E4A7D4000A8034-123E3700\r\nCall-ID: 3A477FD70434C083@10.10.10.18\r\nCSeq: 5709 BYE\r\nProxy-Authorization: Digest username=\"1111111111\", realm=\"sip.local\", nonce=\"WOUBrFjkp9Qj4YjEwa9jvelH"..., len = 34}
h_id = 854147520
dlg = 0x7fad00000000

After event and at the end. First looks like a valid value, second a bit more strange.

Can you get in the same frame:

p dlg
p *dlg

@linuxmaniac
Copy link
Member Author

Did you put in pastebin the wrong file (like dlg_db_handlers.c)?

Yes, I did, sorry. The right file at https://paste.debian.net/926247/

@linuxmaniac
Copy link
Member Author

(gdb) f 4
#4  0x00007fae32c3457e in dlg_onroute (req=0x7fae3def7ac8, route_params=0x7fad006e9b1e, 
    param=0x7fff717dab10) at dlg_handlers.c:1350
1350	dlg_handlers.c: No such file or directory.
(gdb) p dlg
$1 = (dlg_cell_t *) 0x7fad74f20258
(gdb) p *dlg
$2 = {ref = 1264713769, next = 0x7fad74e7aa78, prev = 0x7fad74f20288, h_id = 20, h_entry = 0, 
  state = 1962017437, lifetime = 32685, init_ts = 11, start_ts = 0, end_ts = 1734963823, 
  dflags = 1667457375, iflags = 1818321759, sflags = 1601332588, toroute = 1919251317, 
  toroute_name = {
    s = 0x3633323631393631 <error: Cannot access memory at address 0x3633323631393631>, 
    len = 1768189440}, from_rr_nb = 115, tl = {next = 0x38, prev = 0x0, timeout = 0}, callid = {
    s = 0x7fad00000000 <error: Cannot access memory at address 0x7fad00000000>, len = 890716088}, 
  from_uri = {s = 0x7fae3517bf90 <__FUNCTION__.15256> "rtpp_function_call", len = 890714464}, 
  to_uri = {s = 0x981 <error: Cannot access memory at address 0x981>, len = -252645136}, req_uri = {
    s = 0x7fad74f20380 "SDj02te01-da0287f0f142f9dac6bda3fd889f1592-06a7f50", len = 50}, tag = {{
      s = 0x7fad757740f8 "z9hG4bK41d6.cdcb6121b68837c41a117213d8d3d9a4.0", len = 46}, {
      s = 0x7fad74aa0688 "", len = 6742548}}, cseq = {{s = 0x0, len = 56}, {s = 0x0, len = 0}}, 
  route_set = {{s = 0x7fad00000000 <error: Cannot access memory at address 0x7fad00000000>, 
      len = 890720288}, {s = 0x7fae3517beeb <__FUNCTION__.6323> "shm_str_dup", len = 890714464}}, 
  contact = {{s = 0x2d3 <error: Cannot access memory at address 0x2d3>, len = -252645136}, {
      s = 0x30657432306a4453 <error: Cannot access memory at address 0x30657432306a4453>, 
      len = 1633955121}}, bind_addr = {0x3966323431663066, 0x3361646236636164}, cbs = {
    first = 0x3531663938386466, types = 808268345}, profile_links = 0x3035, vars = 0x48}

@linuxmaniac
Copy link
Member Author

for the relevant part of code

			LM_WARN("inconsitent dlg timer data on dlg %p [%u:%u] "
				"with clid '%.*s' and tags '%.*s' '%.*s'\n",
				dlg, dlg->h_entry, dlg->h_id,
				dlg->callid.len, dlg->callid.s,
				dlg->tag[DLG_CALLER_LEG].len, dlg->tag[DLG_CALLER_LEG].s,
				dlg->tag[DLG_CALLEE_LEG].len, dlg->tag[DLG_CALLEE_LEG].s);
(gdb) p dlg->h_entry
$3 = 0
(gdb) p dlg->h_id
$4 = 20
(gdb) p dlg->callid.len
$5 = 890716088
(gdb) p dlg->callid.s
$6 = 0x7fad00000000 <error: Cannot access memory at address 0x7fad00000000>
(gdb) p dlg->tag[0].len
$7 = 46
(gdb) p dlg->tag[0].s
$8 = 0x7fad757740f8 "z9hG4bK41d6.cdcb6121b68837c41a117213d8d3d9a4.0"
(gdb) p dlg->tag[1].len
$9 = 6742548
(gdb) p dlg->tag[1].s
$10 = 0x7fad74aa0688 ""

@linuxmaniac
Copy link
Member Author

and we have this at the log at the time of the core, 11064 is another process.

Apr  5 10:19:36 sipwise1-prx03a proxy[11064]: WARNING: dialog [dlg_handlers.c:1220]: dlg_onroute(): unable to find dialog for BYE with route param 'c68.0d81' [2156:6352]

@miconda
Copy link
Member

miconda commented Apr 6, 2017

Is it possible that there is a time consuming operations for BYE (eg, a slow database operation -- accounting or something else) that can take more than 5 seconds?

@linuxmaniac
Copy link
Member Author

more than 5 secs? I don't think so. @apogrebennyk can you confirm?

@apogrebennyk
Copy link
Contributor

We have latency logging enabled but there were no abnormally long delays reported. The last lines before the crash are like this:

Apr  5 10:19:39 sipwise1 proxy[11071]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [375 ms] for [select `received`,`contact`,`socket`
,`cflags`,`pat]
Apr  5 10:19:39 sipwise1 proxy[11072]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [312 ms] for [select `received`,`contact`,`socket`
,`cflags`,`pat]
Apr  5 10:19:39 sipwise1 proxy[11073]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [375 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat]
Apr  5 10:19:39 sipwise1 proxy[11069]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [313 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat]
Apr  5 10:19:39 sipwise1 proxy[11070]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [312 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat]
Apr  5 10:19:40 sipwise1 proxy[11075]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [375 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat]
Apr  5 10:19:40 sipwise1 proxy[11074]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [438 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat]
Apr  5 10:19:40 sipwise1 proxy[11071]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [375 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat]
Apr  5 10:19:40 sipwise1 proxy[11072]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [313 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat]
Apr  5 10:19:40 sipwise1 proxy[11073]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [375 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat]
Apr  5 10:19:41 sipwise1 proxy[11069]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [312 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat]
Apr  5 10:19:41 sipwise1 proxy[11070]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [313 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat]
Apr  5 10:19:41 sipwise1 proxy[11075]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [375 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat]
Apr  5 10:19:42 sipwise1 proxy[11072]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [312 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat]
Apr  5 10:19:42 sipwise1 proxy[11071]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [312 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat]
Apr  5 10:19:42 sipwise1 proxy[11074]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [438 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat]
Apr  5 10:19:42 sipwise1 proxy[11073]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [375 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat]
Apr  5 10:19:42 sipwise1 proxy[11069]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [313 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat]
Apr  5 10:19:42 sipwise1 proxy[11070]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [312 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat]
Apr  5 10:19:43 sipwise1 proxy[11075]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [375 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat]
Apr  5 10:19:43 sipwise1 proxy[11072]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [313 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat]
Apr  5 10:19:43 sipwise1 proxy[11071]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [312 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat]
Apr  5 10:19:43 sipwise1 proxy[11073]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [375 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat]
Apr  5 10:19:43 sipwise1 proxy[11074]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [438 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat]
Apr  5 10:19:44 sipwise1 proxy[11069]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [375 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat]
Apr  5 10:19:44 sipwise1 proxy[11070]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [313 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat]
Apr  5 10:19:44 sipwise1 proxy[11075]: NOTICE: <core> [db_query.c:60]: db_do_submit_query(): alert - query execution too long [375 ms] for [select `received`,`contact`,`socket`,`cflags`,`pat]
Apr  5 10:19:44 sipwise1 proxy[11056]: ALERT: <core> [main.c:740]: handle_sigs(): child process 11061 exited by a signal 11
Apr  5 10:19:44 sipwise1 proxy[11056]: ALERT: <core> [main.c:743]: handle_sigs(): core was generated

miconda added a commit that referenced this issue Apr 6, 2017
- done for dlg_onroute(), as an extra safety for early detection of
  races, related to GH #1059
sipwise-jenkins pushed a commit to sipwise/kamailio that referenced this issue Apr 6, 2017
kamailio/kamailio#1059

Change-Id: I3dcad249537f92c2ef75708a60f53b5b5383264a
sipwise-jenkins pushed a commit to sipwise/kamailio that referenced this issue Apr 7, 2017
kamailio/kamailio#1059

Change-Id: I3dcad249537f92c2ef75708a60f53b5b5383264a
(cherry picked from commit 1e2115a)
sipwise-jenkins pushed a commit to sipwise/kamailio that referenced this issue Apr 7, 2017
kamailio/kamailio#1059

Change-Id: I3dcad249537f92c2ef75708a60f53b5b5383264a
(cherry picked from commit 1e2115a)
sipwise-jenkins pushed a commit to sipwise/kamailio that referenced this issue Apr 7, 2017
kamailio/kamailio#1059

Change-Id: Ic0b3e5b84d92c0a2348a839fa7915597ea501605
sipwise-jenkins pushed a commit to sipwise/kamailio that referenced this issue Apr 7, 2017
kamailio/kamailio#1059

Change-Id: Ic0b3e5b84d92c0a2348a839fa7915597ea501605
(cherry picked from commit f81cba7)
sipwise-jenkins pushed a commit to sipwise/kamailio that referenced this issue Apr 7, 2017
kamailio/kamailio#1059

Change-Id: I3dcad249537f92c2ef75708a60f53b5b5383264a
(cherry picked from commit 1e2115a)
(cherry picked from commit 026743a)
sipwise-jenkins pushed a commit to sipwise/kamailio that referenced this issue Apr 7, 2017
kamailio/kamailio#1059

Change-Id: I3dcad249537f92c2ef75708a60f53b5b5383264a
(cherry picked from commit 1e2115a)
miconda added a commit that referenced this issue Apr 21, 2017
…ute execution

- avoid execution of script callbacks for event route
- related to GH #1059 and #1069
miconda added a commit that referenced this issue Apr 26, 2017
…ute execution

- avoid execution of script callbacks for event route
- related to GH #1059 and #1069
- backport of a8b7f94
miconda added a commit that referenced this issue Apr 26, 2017
…ute execution

- avoid execution of script callbacks for event route
- related to GH #1059 and #1069
- backport of a8b7f94

(cherry picked from commit d95f096)
@miconda
Copy link
Member

miconda commented Apr 26, 2017

If you haven't ported the patches referenced above yourself, I just did it. Can you try with latest 4.4 branch and see the results?

sergey-vb pushed a commit to sergey-vb/kamailio that referenced this issue Apr 27, 2017
…ute execution

- avoid execution of script callbacks for event route
- related to GH kamailio#1059 and kamailio#1069
@miconda
Copy link
Member

miconda commented May 15, 2017

Closing this one, eventually fixed with the patches from the few weeks ago. If issue persists, re-open.

@miconda miconda closed this as completed May 15, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants