Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OpenSIPS 1.10 segfault on fm_remove_free #126

Closed
jpyle490 opened this issue Nov 13, 2013 · 6 comments
Closed

OpenSIPS 1.10 segfault on fm_remove_free #126

jpyle490 opened this issue Nov 13, 2013 · 6 comments
Assignees
@bogdan-iancu
Labels
bug
Milestone
1.10

Comments

@jpyle490
Copy link

I've been having some stability problems with dcb87bc from Nov 8. It has crashed in a variety of ways (mostly down and hard), but all appear to have fm_remove_free in common. Two backtraces are below - telephone numbers changed to protect the guilty.

==> backtrace1.log

Program terminated with signal 11, Segmentation fault.
#0  0x00000000004afd60 in fm_remove_free (n=0x7f35d55157a8, qm=0x7f35d54e7010) at mem/f_malloc.c:172
172 mem/f_malloc.c: No such file or directory.
(gdb) bt full
#0  0x00000000004afd60 in fm_remove_free (n=0x7f35d55157a8, qm=0x7f35d54e7010) at mem/f_malloc.c:172
        pf = 0x0
        hash = 3
#1  fm_malloc (qm=0x7f35d54e7010, size=24) at mem/f_malloc.c:383
        frag = 0x7f35d55157a8
        n = <optimized out>
        hash = <optimized out>
        __FUNCTION__ = "fm_malloc"
#2  0x00007f35d43c6174 in build_rr (_inbound=1, params=0x0, tag=0x7f35d5515b20, user=0x7fffe1ebe160, _l2=0x7f35d5515fe8, 
    _l=0x7f35d5515f98) at record.c:158
        prefix = <optimized out>
        term = <optimized out>
        r2 = <optimized out>
        prefix_len = 19
        suffix = <optimized out>
        suffix_len = 41
        p = <optimized out>
#3  record_route (_m=_m@entry=0x7f35d5514950, params=0x0) at record.c:310
        l = 0x7f35d5515f98
        l2 = 0x7f35d5515fe8
        user = {s = 0x7fffe1ebe1b0 "x\326O\325\065\177", len = 0}
        from = <optimized out>
        tag = 0x7f35d5515b20
        __FUNCTION__ = "record_route"
#4  0x00007f35d43c869b in w_record_route (msg=0x7f35d5514950, key=<optimized out>, bar=<optimized out>) at rr_mod.c:275
        s = {s = 0x5a3 <Address 0x5a3 out of bounds>, len = -716093104}
        __FUNCTION__ = "w_record_route"
#5  0x0000000000419930 in do_action (a=0x7f35d54fd880, msg=0x7f35d5514950) at action.c:1715
        val_s = {s = 0x7b24a0 "Tue Nov 12 20:45:47", len = -504634336}
        aux_counter = <optimized out>
        i = <optimized out>
        avp_val = {n = 8070304, s = {s = 0x7b24a0 "Tue Nov 12 20:45:47", len = 8070304}}
        j = <optimized out>
        key_number = <optimized out>
        adv_addr = {s = 0x7b24a0 "Tue Nov 12 20:45:47", len = -504634336}
        increment = <optimized out>
        decrement = <optimized out>
        avp_type = <optimized out>
        aux = {s = 0x7b24a0 "Tue Nov 12 20:45:47", len = -504634336}
        cdb_reply = <optimized out>
        val_number = <optimized out>
        it = <optimized out>
        avp_name = {n = 24, s = {s = 0x3000000018 <Address 0x3000000018 out of bounds>, len = -504633872}}
        ret = -5
        v = <optimized out>
        to = <optimized out>
        p = <optimized out>
        tmp = <optimized out>
        new_uri = <optimized out>
        end = <optimized out>
        crt = <optimized out>
        len = <optimized out>
        i = <optimized out>
        user = 0
        expires = 0
        vals = {{s = 0x7b24a0 "Tue Nov 12 20:45:47", len = -504634336}, {
            s = 0x7f35d4d82380 "%.3s %.3s%3d %.2d:%.2d:%.2d %d\n", len = -504634224}, {s = 0x19 <Address 0x19 out of bounds>, 
            len = 7935638}, {s = 0x7b24a0 "Tue Nov 12 20:45:47", len = -724978014}, {
            s = 0x33313032fbad8001 <Address 0x33313032fbad8001 out of bounds>, len = 8070304}}
        result = {s = 0x19 <Address 0x19 out of bounds>, len = 0}
        uri = {user = {s = 0x7b24a0 "Tue Nov 12 20:45:47", len = 8070304}, passwd = {s = 0x7b24a0 "Tue Nov 12 20:45:47", 
            len = 8070329}, host = {s = 0x3000000018 <Address 0x3000000018 out of bounds>, len = -504633664}, port = {
            s = 0x7fffe1ebe600 "\377\377\377\377", len = 0}, params = {s = 0x0, len = 0}, headers = {s = 0x0, len = 0}, 
          port_no = 9376, proto = 123, type = ERROR_URI_T, transport = {s = 0x7f35d4c9b6a2 "H9\\$8t\bH\213T$(\306\002", 
            len = -72548352}, ttl = {s = 0x0, len = 24}, user_param = {s = 0x7fffe1ebe720 "PIQ\325\065\177", 
            len = -504633760}, maddr = {s = 0x0, len = 24}, method = {s = 0x7fffe1ebe740 "\020\350\353\341\377\177", 
            len = -1}, lr = {s = 0x0, len = 0}, r2 = {s = 0x7f35d4fb1fc0 "", len = 0}, gr = {s = 0x0, len = 24}, 
          transport_val = {s = 0x7fffe1ebe780 "", len = -504633664}, ttl_val = {
            s = 0x7fffe1ebe480 "\001\200\255\373\065\177", len = 8070304}, user_param_val = {s = 0x7fffe1ebe570 "\271${", 
            len = -724032640}, maddr_val = {s = 0x7fffe1ebe5e0 "0", len = 25}, method_val = {s = 0x7f35d5514950 "\004", 
            len = 8070304}, lr_val = {s = 0x7f35d4c9b6a2 "H9\\$8t\bH\213T$(\306\002", len = -72515583}, r2_val = {
            s = 0x7b24a0 "Tue Nov 12 20:45:47", len = 8070304}, gr_val = {s = 0x7b24a0 "Tue Nov 12 20:45:47", len = 8070304}, 
          u_name = {{s = 0x7b24b9 "", len = 8070329}, {s = 0x7b24a0 "Tue Nov 12 20:45:47", len = 8070329}, {s = 0x0, 
              len = 0}, {s = 0x0, len = 0}, {s = 0x0, len = -716091216}}, u_val = {{
              s = 0x7fffe1ebe540 "\001\200\255", <incomplete sequence \373>, len = 8070304}, {s = 0x7fffe1ebe630 "\005\026y", 
              len = -724032640}, {s = 0x7fffe1ebe6a0 "0", len = 25}, {s = 0x5a3 <Address 0x5a3 out of bounds>, 
              len = 8070304}, {s = 0x7f35d4c9b6a2 "H9\\$8t\bH\213T$(\306\002", len = -72515583}}, u_params_no = 9376}
        next_hop = {user = {s = 0x7b24a0 "Tue Nov 12 20:45:47", len = 8070304}, passwd = {s = 0x7b24a0 "Tue Nov 12 20:45:47", 
            len = 8070329}, host = {s = 0x7b24b9 "", len = 8070304}, port = {s = 0x7b24b9 "", len = 0}, params = {s = 0x0, 
            len = 0}, headers = {s = 0x0, len = 0}, port_no = 9376, proto = 123, type = ERROR_URI_T, transport = {
            s = 0x1a <Address 0x1a out of bounds>, len = 7929856}, ttl = {s = 0x0, len = -716089616}, user_param = {
            s = 0x7f35d4c7d513 "H\201\304", <incomplete sequence \330>, len = 48}, maddr = {s = 0x0, len = -504633856}, 
          method = {s = 0x4d842f "D\017\266\\$0A\200\373\b\017\207\t\t", len = -1}, lr = {
            s = 0x15 <Address 0x15 out of bounds>, len = 7935615}, r2 = {s = 0x7f35d4fb1fc0 "", len = 0}, gr = {
            s = 0xc <Address 0xc out of bounds>, len = 7935493}, transport_val = {s = 0x2 <Address 0x2 out of bounds>, 
            len = 7936476}, ttl_val = {s = 0x0, len = -721733504}, user_param_val = {
            s = 0x7f35d4c970da "\203\370\001\031\322f\203;", len = 2}, maddr_val = {
            s = 0x4295c2 "H\201\304", <incomplete sequence \330>, len = 8070304}, method_val = {
            s = 0x1a <Address 0x1a out of bounds>, len = 14671}, lr_val = {s = 0x3bc <Address 0x3bc out of bounds>, len = 2}, 
          r2_val = {s = 0x7f35d4c7d513 "H\201\304", <incomplete sequence \330>, len = 48}, gr_val = {
            s = 0x7fffe1ebe7a0 "\240${", len = -504633664}, u_name = {{s = 0x1 <Address 0x1 out of bounds>, len = 20}, {
              s = 0x7f350000002d <Address 0x7f350000002d out of bounds>, len = 47}, {s = 0x7f35d4d7992d "Tue", 
              len = -724067934}, {s = 0xc <Address 0xc out of bounds>, len = 14671}, {s = 0x7f35d5514950 "\004", 
              len = 14671}}, u_val = {{s = 0x3bc <Address 0x3bc out of bounds>, len = -721733504}, {
              s = 0x7f35d4c970da "\203\370\001\031\322f\203;", len = -716093104}, {
              s = 0x4295c2 "H\201\304", <incomplete sequence \330>, len = 10556224}, {
              s = 0x3000000020 <Address 0x3000000020 out of bounds>, len = -504633328}, {s = 0x7fffe1ebe750 "\226\026y", 
              len = 7935638}}, u_params_no = 9380}
        u = <optimized out>
        port = <optimized out>
        cmatch = <optimized out>
        aitem = <optimized out>
        adefault = <optimized out>
        spec = <optimized out>
        model = <optimized out>
        val = {rs = {s = 0x7b24a0 "Tue Nov 12 20:45:47", len = -504634368}, ri = -724032640, flags = 32565}
        pve = <optimized out>
        name_s = {s = 0x33313032007b24a0 <Address 0x33313032007b24a0 out of bounds>, len = -724978014}
        start = {tv_sec = 139869832445953, tv_usec = 8070304}
        route_params_bak = <optimized out>
        route_params_number_bak = <optimized out>
        __FUNCTION__ = "do_action"
#6  0x000000000041dca0 in run_action_list (a=0x0, a@entry=0x7f35d54f4360, msg=0x7f35d5514950) at action.c:171
        ret = 24
        t = 0x7f35d54fd880
#7  0x000000000041dfd0 in run_actions (msg=0x7f35d5514950, a=0x7f35d54f4360) at action.c:136
No locals.
#8  run_actions (msg=0x7f35d5514950, a=0x7f35d54f4360) at action.c:190
No locals.
#9  run_top_route (a=0x7f35d54f4360, msg=msg@entry=0x7f35d5514950) at action.c:211
        bl_last_msg_id = 4
        bk_action_flags = 0
        bk_rec_lev = 0
        ret = <optimized out>
#10 0x00000000004597bf in receive_msg (buf=<optimized out>, len=956, rcv_info=rcv_info@entry=0x7fffe1ebe8f0) at receive.c:171
        msg = 0x7f35d5514950
        start = {tv_sec = 139869188839336, tv_usec = 139869189740200}
        tmp = 0x394f <Address 0x394f out of bounds>
        __FUNCTION__ = "receive_msg"
#11 0x00000000004ab1ea in udp_rcv_loop () at udp_server.c:424
        len = <optimized out>
        buf = "INVITE sip:2165551212@192.168.54.238:5060 SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.54.238:1443;branch=z9hG4bK55c9.7776a54.0\r\nTo: sip:2165551212@192.168.54.238:5060\r\nFrom: \"Test\" <sip:2169311212@172.21.254.10"...
        tmp = 0x3bc <Address 0x3bc out of bounds>
        from = 0x7f35d550d7a8
        fromlen = 16
        ri = {src_ip = {af = 2, len = 4, u = {addrl = {55536167104, 485331304458}, addr32 = {3996559552, 12, 10, 113}, 
              addr16 = {43200, 60982, 12, 0, 10, 0, 113, 0}, 
              addr = "\300\250\066\356\f\000\000\000\n\000\000\000q\000\000"}}, dst_ip = {af = 2, len = 4, u = {addrl = {
                3996559552, 0}, addr32 = {3996559552, 0, 0, 0}, addr16 = {43200, 60982, 0, 0, 0, 0, 0, 0}, 
              addr = "\300\250\066\356", '\000' <repeats 11 times>}}, src_port = 1443, dst_port = 1442, proto = 1, 
          proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {sa_family = 2, 
              sa_data = "\005\243\300\250\066\356\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 41733, 
              sin_addr = {s_addr = 3996559552}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, 
              sin6_port = 41733, sin6_flowinfo = 3996559552, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, 
                  __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, 
          bind_address = 0x7f35d54efcb0}
        p = <optimized out>
        msg = {
          s = 0x791620 "INVITE sip:2165551212@192.168.54.238:5060 SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.54.238:1443;branch=z9hG4bK55c9.7776a54.0\r\nTo: sip:2165551212@192.168.54.238:5060\r\nFrom: \"Test\" <sip:2169311212@172.21.254.10"..., len = 956}
        __FUNCTION__ = "udp_rcv_loop"
#12 0x00000000004162cc in main_loop () at main.c:917
        i = 0
        pid = <optimized out>
        si = <optimized out>
        startup_done = <optimized out>
        chd_rank = 1
        rc = <optimized out>
        load_p = 0x7f35d0289ef0
#13 main (argc=<optimized out>, argv=<optimized out>) at main.c:1598
        cfg_log_stderr = <optimized out>
        cfg_stream = 0xa0d010
        c = <optimized out>
        r = <optimized out>
        tmp = 0x7fffe1ebee3a ""
        tmp_len = <optimized out>
        port = <optimized out>
        proto = <optimized out>
        options = 0x510290 "f:cCm:M:b:l:n:N:rRvdDFETSVhw:t:u:g:P:G:W:o:"
        ret = -1
        seed = 415923319
        rfd = <optimized out>
        __FUNCTION__ = "main"

==> backtrace2.log

Program terminated with signal 11, Segmentation fault.
#0  0x00000000004afd60 in fm_remove_free (n=0x7f9e037941a0, qm=0x7f9e03766010) at mem/f_malloc.c:172
172 mem/f_malloc.c: No such file or directory.
(gdb) bt full
#0  0x00000000004afd60 in fm_remove_free (n=0x7f9e037941a0, qm=0x7f9e03766010) at mem/f_malloc.c:172
        pf = 0x0
        hash = 9
#1  fm_malloc (qm=0x7f9e03766010, size=size@entry=56) at mem/f_malloc.c:383
        frag = 0x7f9e037941a0
        n = <optimized out>
        hash = <optimized out>
        __FUNCTION__ = "fm_malloc"
#2  0x0000000000428821 in anchor_lump (msg=msg@entry=0x7f9e03793950, offset=82, len=len@entry=0, type=type@entry=HDR_VIA_T)
    at data_lump.c:398
        tmp = <optimized out>
        prev = <optimized out>
        t = <optimized out>
        list = <optimized out>
        __FUNCTION__ = "anchor_lump"
#3  0x0000000000440dba in build_req_buf_from_sip_req (msg=msg@entry=0x7f9e03793950, 
    returned_len=returned_len@entry=0x7fffd6133e84, send_sock=0x7f9e0376ecb0, proto=<optimized out>, flags=flags@entry=1)
    at msg_translator.c:1438
        len = 1055
        new_len = <optimized out>
        received_len = 0
        rport_len = 0
        uri_len = 0
        via_len = <optimized out>
        body_delta = 15
        line_buf = 0x7f9e03794268 "Via: SIP/2.0/UDP 192.168.54.238:1442;branch=z9hG4bK8106.46070c5.0\r\n"
        received_buf = 0x0
        rport_buf = 0x0
        new_buf = 0x0
        buf = 0x791620 "INVITE sip:2165551212@192.168.54.238:5060;transport=udp;sipxecs-lineid=9 SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.54.238:1443;branch=z9hG4bK8106.4d014a42.0\r\nTo: sip:2165551212@192.168.54.238:5060\r\nFrom: \"Jef"...
        id_buf = <optimized out>
        offset = <optimized out>
        s_offset = <optimized out>
        size = <optimized out>
        id_len = 0
        anchor = <optimized out>
        via_insert_param = 0x0
        branch = {s = 0x7f9e03794008 "z9hG4bK8106.46070c5.0", len = 21}
        extra_params = {s = 0x0, len = 0}
        hp = {host = 0x0, port = 0x0}
        via1_deleted = <optimized out>
        __FUNCTION__ = "build_req_buf_from_sip_req"
#4  0x00007f9e028684a5 in print_uac_request (proto=<optimized out>, send_sock=<optimized out>, len=0x7fffd6133e84, 
    i_req=0x7f9e03793950) at t_fwd.c:192
        buf = <optimized out>
#5  update_uac_dst (uac=0x7f9dfe5107a0, request=0x7f9e03793950) at t_fwd.c:327
        send_sock = <optimized out>
        shbuf = 0x7f9dfe5104b0 ""
        len = <optimized out>
#6  add_uac (t=t@entry=0x7f9dfe5105c8, request=request@entry=0x7f9e03793950, uri=uri@entry=0x7fffd6133f80, 
    next_hop=next_hop@entry=0x7fffd6133f70, path=path@entry=0x7f9e03794080, proxy=0x7f9e037941b8, proxy@entry=0x0)
    at t_fwd.c:434
        branch = <optimized out>
        do_free_proxy = 1
        ret = <optimized out>
        __FUNCTION__ = "add_uac"
#7  0x00007f9e0286a115 in t_forward_nonack (t=t@entry=0x7f9dfe5105c8, p_msg=p_msg@entry=0x7f9e03793950, proxy=proxy@entry=0x0)
    at t_fwd.c:658
        backup_uri = {s = 0x7f9e03795a68 "sip:2165551212@external.fvdstuff.com;transport=udp;sipxecs-lineid=9", 
          len = 74}
        backup_dst = {s = 0x7f9e037955a8 "sip:192.168.54.238:1441", len = 23}
        branch_ret = <optimized out>
        lowest_ret = -5
        current_uri = {s = 0x7f9e03795a68 "sip:2165551212@external.fvdstuff.com;transport=udp;sipxecs-lineid=9", 
          len = 74}
        added_branches = 0
        i = <optimized out>
        q = 0
        t_invite = <optimized out>
        success_branch = <optimized out>
        dst_uri = {s = 0x7f9e03793950 "\v", len = 42407995}
        bk_sock = 0x0
        br_flags = <optimized out>
        bk_br_flags = 0
        idx = <optimized out>
        path = {s = 0x710000000a <Address 0x710000000a out of bounds>, len = 0}
        bk_path = {s = 0x0, len = 0}
        __FUNCTION__ = "t_forward_nonack"
#8  0x00007f9e02866db8 in t_relay_to (p_msg=p_msg@entry=0x7f9e03793950, proxy=proxy@entry=0x0, flags=flags@entry=2)
    at t_funcs.c:248
        ret = 0
        new_tran = <optimized out>
        reply_ret = 72
        t = 0x7f9dfe5105c8
        __FUNCTION__ = "t_relay_to"
#9  0x00007f9e02875cff in w_t_relay (p_msg=0x7f9e03793950, proxy=0x0, flags=0x2 <Address 0x2 out of bounds>) at tm.c:1138
        t = <optimized out>
        ret = <optimized out>
#10 0x0000000000419930 in do_action (a=0x7f9e03789388, msg=0x7f9e03793950) at action.c:1715
        val_s = {s = 0x7f9e037956b8 "Content-Length: 437\r\n", len = 18}
        aux_counter = <optimized out>
        i = <optimized out>
        avp_val = {n = -703379992, s = {s = 0x7fffd61345e8 "", len = 0}}
        j = <optimized out>
        key_number = <optimized out>
        adv_addr = {s = 0x7f9e037956b8 "Content-Length: 437\r\n", len = 18}
        increment = <optimized out>
        decrement = <optimized out>
        avp_type = <optimized out>
        aux = {s = 0x7f9e037956b8 "Content-Length: 437\r\n", len = 18}
        cdb_reply = <optimized out>
        val_number = <optimized out>
        it = <optimized out>
        avp_name = {n = 50336640, s = {s = 0x7f9e03001380 "%.3s %.3s%3d %.2d:%.2d:%.2d %d\n", len = -703389156}}
        ret = -5
        v = <optimized out>
        to = <optimized out>
        p = <optimized out>
        tmp = <optimized out>
        new_uri = <optimized out>
        end = <optimized out>
        crt = <optimized out>
        len = <optimized out>
        i = <optimized out>
        user = 0
        expires = 0
        vals = {{s = 0x7f9e037956b8 "Content-Length: 437\r\n", len = 18}, {s = 0x7f9e02f05930 "H\213\227\210", 
            len = 52635776}, {s = 0x400000000 <Address 0x400000000 out of bounds>, len = 0}, {
            s = 0x6d6134920 <Address 0x6d6134920 out of bounds>, len = 52635776}, {s = 0x7fffd6134910 "x\002Q\376\235\177", 
            len = -72537977}}
        result = {s = 0x7fffd61345e4 "2013", len = 15912208}
        uri = {user = {s = 0x7fffd6134928 "(", len = 49223870}, passwd = {
            s = 0x7f9e03232880 "\207(\255", <incomplete sequence \373>, len = 1}, host = {s = 0x7fffd61346f4 "", 
            len = -703379624}, port = {s = 0x7f9e02f05930 "H\213\227\210", len = 52635776}, params = {s = 0x0, 
            len = 4914759}, headers = {s = 0x7fffd61346f8 " H\023\326\377\177", len = 52635776}, port_no = 1024, proto = 0, 
          type = ERROR_URI_T, transport = {s = 0x2 <Address 0x2 out of bounds>, len = 18}, ttl = {
            s = 0x4ac329 "HcC\030H\213S\020\306\004\002", len = -703379628}, user_param = {
            s = 0x2000000000000 <Address 0x2000000000000 out of bounds>, len = 58283432}, maddr = {
            s = 0x7f9e00000012 <Address 0x7f9e00000012 out of bounds>, len = 0}, method = {s = 0x7f9e03230bc0 "", 
            len = 52635776}, lr = {s = 0x7f9e03787b80 "", len = 58276176}, r2 = {s = 0x7fffd6134470 "", len = 54}, gr = {
            s = 0x36 <Address 0x36 out of bounds>, len = 1443}, transport_val = {
            s = 0x4ac435 "H\205\300t\036H\213\025\207 +", len = 58283432}, ttl_val = {
            s = 0x7f9e00000012 <Address 0x7f9e00000012 out of bounds>, len = 58276176}, user_param_val = {
            s = 0x7fffd6134470 "", len = 58227584}, maddr_val = {
            s = 0x4504c2 "\205\300\017\211\b\376\377\377H\213\005\267\266\060", len = 50336669}, method_val = {
            s = 0x500000001 <Address 0x500000001 out of bounds>, len = 0}, lr_val = {s = 0x3 <Address 0x3 out of bounds>, 
            len = 58283432}, r2_val = {s = 0x7f9d00000012 <Address 0x7f9d00000012 out of bounds>, len = 58276176}, gr_val = {
            s = 0x203793950 <Address 0x203793950 out of bounds>, len = 58276176}, u_name = {{
              s = 0x4b014c "H\203\304\030[]\303\017\037D", len = 58227560}, {s = 0x44cddc "H\203\304\070[]\303\017\037D", 
              len = 1443}, {s = 0x7f9e03787dc8 "6", len = -703380368}, {s = 0x45205f "\213E\024\250@t\265L\213-K\366\064", 
              len = 58228168}, {s = 0x7f9e03787dc8 "6", len = 58276176}}, u_val = {{s = 0x7fffd6134470 "", len = -703379848}, 
            {s = 0x7fffd61346f8 " H\023\326\377\177", len = 0}, {s = 0x13c400000000 <Address 0x13c400000000 out of bounds>, 
              len = 0}, {s = 0x0, len = 0}, {s = 0x4afe47 "H\215C\030H\203\304\020[]A\\\303\017\037@", len = 65}}, 
          u_params_no = 49146}
        next_hop = {user = {s = 0x7f9e03793950 "\v", len = 42415403}, passwd = {s = 0xc <Address 0xc out of bounds>, 
            len = 4299056}, host = {s = 0x0, len = 14}, port = {s = 0x0, len = 58276176}, params = {s = 0x0, len = 4314057}, 
          headers = {s = 0x7f9e03795e68 "8", len = 52635776}, port_no = 0, proto = 0, type = ERROR_URI_T, transport = {
            s = 0x0, len = -1}, ttl = {s = 0x7f9e03232880 "\207(\255", <incomplete sequence \373>, len = -703378512}, 
          user_param = {s = 0x7f9e03230bc0 "", len = 52635776}, maddr = {s = 0x7f9e02ef18be "A\211\304\353\304\017\037D", 
            len = 52635776}, method = {s = 0x7f9e02f160da "\203\370\001\031\322f\203;", len = 49305904}, lr = {
            s = 0x7f9e03232880 "\207(\255", <incomplete sequence \373>, len = 0}, r2 = {s = 0x0, len = 7935528}, gr = {
            s = 0x7f9e03232880 "\207(\255", <incomplete sequence \373>, len = -703378416}, transport_val = {
            s = 0xfbad2887 <Address 0xfbad2887 out of bounds>, len = -703378392}, ttl_val = {
            s = 0x7f9e02ef18be "A\211\304\353\304\017\037D", len = 1442}, user_param_val = {
            s = 0x5a1 <Address 0x5a1 out of bounds>, len = 20}, maddr_val = {s = 0x7f9e02ff892d "Tue", len = 50301346}, 
          method_val = {s = 0xc <Address 0xc out of bounds>, len = 8070304}, lr_val = {
            s = 0x398a <Address 0x398a out of bounds>, len = 0}, r2_val = {s = 0x0, len = 52635776}, gr_val = {
            s = 0x202f160da <Address 0x202f160da out of bounds>, len = 14730}, u_name = {{
              s = 0x4295c2 "H\201\304", <incomplete sequence \330>, len = 24}, {s = 0x7fffd6134970 "", len = -703379280}, {
              s = 0x0, len = 7935542}, {s = 0xe <Address 0xe out of bounds>, len = 7935557}, {
              s = 0x4 <Address 0x4 out of bounds>, len = 7935562}}, u_val = {{s = 0x1e <Address 0x1e out of bounds>, 
              len = 0}, {s = 0x0, len = 70596}, {
              s = 0x79164a "transport=udp;sipxecs-lineid=9 SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.54.238:1443;branch=z9hG4bK8106.4d014a42.0\r\nTo: sip:2165551212@192.168.54.238:5060\r\nFrom: \"Jeff Pyle\" <sip:2169311212@internal.fvdstuff."..., len = 13}, 
            {s = 0x0, len = 0}, {s = 0x0, len = 0}}, u_params_no = 0}
        u = <optimized out>
        port = <optimized out>
        cmatch = <optimized out>
        aitem = <optimized out>
        adefault = <optimized out>
        spec = <optimized out>
        model = <optimized out>
        val = {rs = {s = 0x7f9e0378e9b0 "osxbc-uc", len = 52628416}, ri = 52635776, flags = 32670}
        pve = <optimized out>
        name_s = {s = 0x0, len = 49241660}
        start = {tv_sec = 50336669, tv_usec = 140316639838820}
        route_params_bak = <optimized out>
        route_params_number_bak = <optimized out>
        __FUNCTION__ = "do_action"
#11 0x000000000041dca0 in run_action_list (a=0x0, msg=0x7f9e03793950) at action.c:171
        ret = 72
        t = 0x7f9e03789388
#12 0x0000000000469ee3 in eval_elem (e=0x7f9e03789460, msg=0x7f9e03793950, val=0x0) at route.c:1499
        uri = {user = {s = 0x0, len = 0}, passwd = {s = 0x0, len = 0}, host = {s = 0x0, len = 0}, port = {s = 0x0, 
            len = 7935591}, params = {s = 0x1 <Address 0x1 out of bounds>, len = 0}, headers = {s = 0x0, len = 0}, 
          port_no = 65535, proto = 65535, type = ERROR_URI_T, transport = {s = 0x0, len = 0}, ttl = {s = 0x7f9e03230fc0 "", 
            len = 0}, user_param = {s = 0x1 <Address 0x1 out of bounds>, len = 0}, maddr = {s = 0x0, len = 8070304}, 
          method = {s = 0x7f9e02f1a6a2 "H9\\$8t\bH\213T$(\306\002", len = -72515583}, lr = {s = 0x0, len = 8070304}, r2 = {
            s = 0x7b24a0 "Tue Nov 12 20:50:59", len = 8070304}, gr = {s = 0x1a <Address 0x1a out of bounds>, len = 52}, 
          transport_val = {s = 0xd <Address 0xd out of bounds>, len = 14730}, ttl_val = {
            s = 0x7f9e02efc513 "H\201\304", <incomplete sequence \330>, len = 48}, user_param_val = {
            s = 0x7fffd6134990 "\350\233P\376\235\177", len = -703379280}, maddr_val = {s = 0x0, len = 8070304}, 
          method_val = {s = 0x7b24a0 "Tue Nov 12 20:50:59", len = 8060928}, lr_val = {s = 0x7f9e02ff892d "Tue", 
            len = 50301346}, r2_val = {s = 0xc <Address 0xc out of bounds>, len = -703378620}, gr_val = {
            s = 0x7f9dfe510278 "\003", len = 52}, u_name = {{s = 0xd <Address 0xd out of bounds>, len = 52635776}, {
              s = 0x7f9e02f160da "\203\370\001\031\322f\203;", len = -28245384}, {
              s = 0x4295c2 "H\201\304", <incomplete sequence \330>, len = 0}, {
              s = 0x3000000028 <Address 0x3000000028 out of bounds>, len = -703378944}, {
              s = 0x7fffd6134940 "\242\211\377\002\236\177", len = 50301346}}, u_val = {{s = 0x7b24a4 "Nov 12 20:50:59", 
              len = 14730}, {s = 0x4afe47 "H\215C\030H\203\304\020[]A\\\303\017\037@", len = 13}, {
              s = 0x1 <Address 0x1 out of bounds>, len = 0}, {s = 0x0, len = 8}, {
              s = 0x43260d "H\213\f$L\213D$\bI\211H\030E\211` \351\017\376\377\377\017\037D", len = -28271640}}, 
          u_params_no = 1088}
        ret = -5
        ival = <optimized out>
        lval = {rs = {s = 0x7b24a0 "Tue Nov 12 20:50:59", len = 8070329}, ri = 8070329, flags = 0}
        rval = {rs = {s = 0x7b24b9 "", len = 0}, ri = 0, flags = 0}
        p = <optimized out>
        n = <optimized out>
        __FUNCTION__ = "eval_elem"
#13 0x00000000004695b5 in eval_expr (e=0x7f9e03789460, msg=0x7f9e03793950, val=0x0) at route.c:1844
        rec_lev = 3
        ret = <optimized out>
        __FUNCTION__ = "eval_expr"
#14 0x00000000004696a1 in eval_expr (e=0x7f9e037894b0, msg=0x7f9e03793950, val=0x0) at route.c:1860
        rec_lev = 3
        ret = <optimized out>
        __FUNCTION__ = "eval_expr"
#15 0x00000000004695a1 in eval_expr (e=0x7f9e03789500, msg=msg@entry=0x7f9e03793950, val=val@entry=0x0) at route.c:1865
        rec_lev = 3
        ret = <optimized out>
        __FUNCTION__ = "eval_expr"
#16 0x000000000041962d in do_action (a=0x7f9e037896c0, msg=0x7f9e03793950) at action.c:993
        val_s = {s = 0x0, len = 0}
        aux_counter = <optimized out>
        i = <optimized out>
        avp_val = {n = 0, s = {s = 0x0, len = 0}}
        j = <optimized out>
        key_number = <optimized out>
        adv_addr = {s = 0x0, len = 0}
        increment = <optimized out>
        decrement = <optimized out>
        avp_type = <optimized out>
        aux = {s = 0x0, len = 0}
        cdb_reply = <optimized out>
        val_number = <optimized out>
        it = <optimized out>
        avp_name = {n = 0, s = {s = 0x600000000 <Address 0x600000000 out of bounds>, len = 0}}
        ret = -5
        v = <optimized out>
        to = <optimized out>
        p = <optimized out>
        tmp = <optimized out>
        new_uri = <optimized out>
        end = <optimized out>
        crt = <optimized out>
        len = <optimized out>
        i = <optimized out>
        user = 0
        expires = 0
        vals = {{s = 0x0, len = 0}, {s = 0x0, len = 1}, {s = 0x0, len = 0}, {s = 0xc0b180 "p\262\300", len = 0}, {
            s = 0x791899 "v=0\r\no=- 3593296261 3593296261 IN IP4 172.21.201.25\r\ns=Blink Pro 3.3.1 (MacOSX)\r\nc=IN IP4 172.21.201.25\r\nt=0 0\r\nm=audio 16396 RTP/AVP 108 9 99 98 0 104 96\r\na=rtcp:16397\r\na=rtpmap:108 opus/48000\r\na=fmt"..., 
            len = 422}}
        result = {s = 0x0, len = 0}
        uri = {user = {s = 0x0, len = 0}, passwd = {s = 0x0, len = 0}, host = {s = 0x0, len = 0}, port = {
            s = 0x7f9e03232880 "\207(\255", <incomplete sequence \373>, len = 58188312}, params = {s = 0xc0b780 "p\270\300", 
            len = -703378208}, headers = {
            s = 0x79190b "audio 16396 RTP/AVP 108 9 99 98 0 104 96\r\na=rtcp:16397\r\na=rtpmap:108 opus/48000\r\na=fmtp:108 useinbandfec=1\r\na=rtpmap:9 G722/8000\r\na=rtpmap:99 speex/32000\r\na=rtpmap:98 speex/16000\r\na=rtpmap:0 PCMU/8000"..., 
            len = 1}, port_no = 32, proto = 0, type = ERROR_URI_T, transport = {s = 0x7f9e02f72553 "\211\302\203=\260P,", 
            len = 1}, ttl = {s = 0x7fffd6134ce0 "P9y\003\236\177", len = 0}, user_param = {s = 0x7f9e0377df38 "\200\261\300", 
            len = 12628352}, maddr = {s = 0x7f9e00000000 <Address 0x7f9e00000000 out of bounds>, len = 58282768}, method = {
            s = 0x0, len = 58188312}, lr = {s = 0x7f9e03795278 "", len = 58276176}, r2 = {
            s = 0x7f9e01140832 "HcS8H\213K0\205\300D\210<\021u\266L\213%\277\331 ", len = 0}, gr = {
            s = 0x7fffd6132d12 "ookup_request: no transaction found\n=24600, isACK=0\n", len = -703378272}, transport_val = {
            s = 0x0, len = 58276176}, ttl_val = {s = 0xf <Address 0xf out of bounds>, len = 58188680}, user_param_val = {
            s = 0x7f9e03793950 "\v", len = 15}, maddr_val = {s = 0x7f9e0377e5e0 "\002", len = 58276176}, method_val = {
            s = 0x7f9e0377e508 "\017", len = 58276176}, lr_val = {
            s = 0x419930 "A\213.A\211\307\351\245\337\377\377A\213n\b\205\355\017\205\251\v", len = 0}, r2_val = {
            s = 0x3232880 <Address 0x3232880 out of bounds>, len = 0}, gr_val = {
            s = 0x2fbad2887 <Address 0x2fbad2887 out of bounds>, len = -703376408}, u_name = {{
              s = 0x7f9e021e14db "H\205\300t\r#X4\203\373\001\031\300\203\310\001[ø\377\377\377\377[\303fff.\017\037\204", 
              len = 58276176}, {s = 0x4179ea "A\213.A\211\307\351\353\376\377\377A\213\216\260", len = 0}, {
              s = 0x7f9e03230bc0 "", len = 52635776}, {s = 0x7f9e02ef18be "A\211\304\353\304\017\037D", len = 0}, {
              s = 0x2fbad2887 <Address 0x2fbad2887 out of bounds>, len = 49305904}}, u_val = {{
              s = 0x7f9e03232880 "\207(\255", <incomplete sequence \373>, len = 0}, {s = 0x0, len = -703376296}, {
              s = 0x203232880 <Address 0x203232880 out of bounds>, len = -703376288}, {
              s = 0x2fbad2887 <Address 0x2fbad2887 out of bounds>, len = -703376264}, {
              s = 0x7f9e02ef18be "A\211\304\353\304\017\037D", len = 49305904}}, u_params_no = 10368}
        next_hop = {user = {s = 0x0, len = 0}, passwd = {s = 0x1 <Address 0x1 out of bounds>, len = 52635776}, host = {
            s = 0x7fffd61354b0 "", len = -72537977}, port = {s = 0x7fffd61354c8 "\001", len = 49223870}, params = {
            s = 0x7fffd61354d0 "\001", len = 49223689}, headers = {
            s = 0x7f9e02ff4490 ' ' <repeats 16 times>, '0' <repeats 16 times>, len = 1}, port_no = 24, proto = 0, type = 48, 
          transport = {s = 0x7fffd61351c0 "\200\023", len = -703377152}, ttl = {s = 0x7fffd6135530 "\001", len = -703376080}, 
          user_param = {s = 0x2d6135520 <Address 0x2d6135520 out of bounds>, len = 24}, maddr = {
            s = 0xd6135690 <Address 0xd6135690 out of bounds>, len = -703376080}, method = {
            s = 0x202ef1809 <Address 0x202ef1809 out of bounds>, len = -703376064}, lr = {s = 0x0, len = 1}, r2 = {
            s = 0x200000000 <Address 0x200000000 out of bounds>, len = 0}, gr = {
            s = 0x200000020 <Address 0x200000020 out of bounds>, len = 50336640}, transport_val = {
            s = 0xd6135388 <Address 0xd6135388 out of bounds>, len = -72515580}, ttl_val = {
            s = 0x2000007dd <Address 0x2000007dd out of bounds>, len = 0}, user_param_val = {
            s = 0xd61353e4d6135720 <Address 0xd61353e4d6135720 out of bounds>, len = -703385872}, maddr_val = {
            s = 0x2d6132f3f <Address 0x2d6132f3f out of bounds>, len = -703377680}, method_val = {s = 0x0, len = 0}, 
          lr_val = {s = 0x2d6134f80 <Address 0x2d6134f80 out of bounds>, len = 8070304}, r2_val = {s = 0x7fffd6135070 "0", 
            len = 50336640}, gr_val = {s = 0x7fffd61350e0 "\200\023", len = 25}, u_name = {{s = 0x7f9e03793950 "\v", 
              len = 8070304}, {s = 0x4 <Address 0x4 out of bounds>, len = 58166376}, {
              s = 0x200000002 <Address 0x200000002 out of bounds>, len = 58276176}, {s = 0x0, len = 1443}, {
              s = 0x4695b5 "\213\035Em1", len = 8070329}}, u_val = {{s = 0x7b24a0 "Tue Nov 12 20:50:59", len = 3}, {
              s = 0x7f9e03778de0 "\001", len = 1}, {s = 0x7f9e03793950 "\v", len = 24}, {s = 0x7fffd6135340 "\r", len = 24}, {
              s = 0x7fffd6135350 "\024", len = -703376752}}, u_params_no = 21344}
        u = <optimized out>
        port = <optimized out>
        cmatch = <optimized out>
        aitem = <optimized out>
        adefault = <optimized out>
        spec = <optimized out>
        model = <optimized out>
        val = {rs = {s = 0xc0b780 "p\270\300", len = 0}, ri = 0, flags = 0}
        pve = <optimized out>
        name_s = {
          s = 0x79190b "audio 16396 RTP/AVP 108 9 99 98 0 104 96\r\na=rtcp:16397\r\na=rtpmap:108 opus/48000\r\na=fmtp:108 useinbandfec=1\r\na=rtpmap:9 G722/8000\r\na=rtpmap:99 speex/32000\r\na=rtpmap:98 speex/16000\r\na=rtpmap:0 PCMU/8000"..., 
          len = 12647552}
        start = {tv_sec = 0, tv_usec = 0}
        route_params_bak = <optimized out>
        route_params_number_bak = <optimized out>
        __FUNCTION__ = "do_action"
#17 0x000000000041dca0 in run_action_list (a=0x0, msg=0x7f9e03793950) at action.c:171
        ret = 72
        t = 0x7f9e037896c0
#18 0x000000000041ddb2 in run_actions (msg=<optimized out>, a=<optimized out>) at action.c:136
        ret = <optimized out>
#19 run_actions (a=<optimized out>, msg=<optimized out>) at action.c:117
        ret = <optimized out>
#20 0x000000000041baf3 in do_action (a=0x7f9e0377f138, msg=0x7f9e03793950) at action.c:649
        val_s = {s = 0x7b0000 "", len = 0}
        aux_counter = <optimized out>
        i = <optimized out>
        avp_val = {n = 8070329, s = {s = 0x7b24b9 "", len = 8070304}}
        j = <optimized out>
        key_number = <optimized out>
        adv_addr = {s = 0x7b0000 "", len = 0}
        increment = <optimized out>
        decrement = <optimized out>
        avp_type = <optimized out>
        aux = {s = 0x7b0000 "", len = 0}
        cdb_reply = <optimized out>
        val_number = <optimized out>
        it = <optimized out>
        avp_name = {n = 8070329, s = {s = 0x7b24b9 "", len = 0}}
        ret = -5
        v = <optimized out>
        to = <optimized out>
        p = <optimized out>
        tmp = <optimized out>
        new_uri = <optimized out>
        end = <optimized out>
        crt = <optimized out>
        len = <optimized out>
        i = <optimized out>
        user = 0
        expires = 0
        vals = {{s = 0x7b0000 "", len = 0}, {s = 0x7f9e03001380 "%.3s %.3s%3d %.2d:%.2d:%.2d %d\n", len = 0}, {
            s = 0xffffffff <Address 0xffffffff out of bounds>, len = 0}, {s = 0xffffffff <Address 0xffffffff out of bounds>, 
            len = 52629440}, {s = 0x0, len = 52629440}}
        result = {s = 0xfbad8001 <Address 0xfbad8001 out of bounds>, len = 8070304}
        uri = {user = {s = 0x0, len = 52629440}, passwd = {s = 0x0, len = 8070329}, host = {s = 0x7b24b9 "", len = 8070304}, 
          port = {s = 0x7b24b9 "", len = -703376768}, params = {s = 0x7b24a0 "Tue Nov 12 20:50:59", len = 26}, headers = {
            s = 0x7f9e03793b28 "\002", len = 14730}, port_no = 14, proto = 0, type = ERROR_URI_T, transport = {
            s = 0x7f9e02efc513 "H\201\304", <incomplete sequence \330>, len = 48}, ttl = {s = 0x7fffd6135370 "\240${", 
            len = -703376752}, user_param = {s = 0x7fffd6135380 "", len = -703376736}, maddr = {
            s = 0x7b24a0 "Tue Nov 12 20:50:59", len = 8070304}, method = {s = 0x7f9e0377e508 "\017", len = 58276176}, lr = {
            s = 0x7f9e0377e5e0 "\002", len = 58276176}, r2 = {s = 0x0, len = 58276176}, gr = {
            s = 0x41dca0 "\205\300u\a\203\ru\r4", len = -703376480}, transport_val = {s = 0x7f9e0377efa0 "\023", 
            len = 58276176}, ttl_val = {s = 0x7f9e0377f078 "\002", len = 58276176}, user_param_val = {s = 0x0, 
            len = 58276176}, maddr_val = {s = 0x41dca0 "\205\300u\a\203\ru\r4", len = 0}, method_val = {s = 0x0, 
            len = -703376480}, lr_val = {s = 0x0, len = 13}, r2_val = {s = 0x469ee3 "H\205\333t\n\307C\024\030", len = 20}, 
          gr_val = {s = 0x7f9e00000032 <Address 0x7f9e00000032 out of bounds>, len = 59}, u_name = {{
              s = 0x7dd <Address 0x7dd out of bounds>, len = 8070304}, {s = 0x7b24a0 "Tue Nov 12 20:50:59", len = 0}, {
              s = 0x7f9e02f4a56b "H\203\304@[Ð\220\220\220\220\220\220\220\220\220\220\220\220\220\220H\211|$\360\337l$\360H\211t$\360\337l$\360\336\351\335\\$\370\362\017\020D$\370ÐH\215\025\371\315.", len = 59}, {
              s = 0xc00000014 <Address 0xc00000014 out of bounds>, len = 0}, {s = 0x0, len = 0}}, u_val = {{
              s = 0xffffffffffffb9b0 <Address 0xffffffffffffb9b0 out of bounds>, len = 12555072}, {
              s = 0x7f9e02f4af5d "H\203\304\bÐ\220\220\220\220\220\220\220\220\220\220\220\220\220H\203\354\bH\213\005]\302.", len = 58278136}, {s = 0x460fd1 "E1Ʌ\300\017\204d\375\377\377f\203|$\034", len = 7935619}, {s = 0x0, len = -703376384}, {
              s = 0x5a300000001 <Address 0x5a300000001 out of bounds>, len = 32}}, u_params_no = 56067}
        next_hop = {user = {s = 0x1 <Address 0x1 out of bounds>, len = 4914759}, passwd = {s = 0x7f9e03793950 "\v", 
            len = -703375984}, host = {s = 0x0, len = 58276176}, port = {s = 0x140 <Address 0x140 out of bounds>, 
            len = 4455054}, params = {s = 0x7f9e03232880 "\207(\255", <incomplete sequence \373>, len = 0}, headers = {
            s = 0x76a <Address 0x76a out of bounds>, len = 4363714}, port_no = 0, proto = 0, type = ERROR_URI_T, transport = {
            s = 0x3000000030 <Address 0x3000000030 out of bounds>, len = -703376032}, ttl = {s = 0x0, len = -80}, 
          user_param = {s = 0x4abb90 "\203\370\377\211\305u~H\213\005\352\377*", len = 0}, maddr = {
            s = 0x7c <Address 0x7c out of bounds>, len = 0}, method = {s = 0x12 <Address 0x12 out of bounds>, len = 18}, 
          lr = {s = 0x1 <Address 0x1 out of bounds>, len = 1}, r2 = {s = 0x0, len = 58286288}, gr = {
            s = 0x47f248 "H\203\304([]A\\A]\303\017\037D", len = 0}, transport_val = {
            s = 0x7f9e02f4a56b "H\203\304@[Ð\220\220\220\220\220\220\220\220\220\220\220\220\220\220H\211|$\360\337l$\360H\211t$\360\337l$\360\336\351\335\\$\370\362\017\020D$\370ÐH\215\025\371\315.", len = 7866194}, ttl_val = {
            s = 0x7 <Address 0x7 out of bounds>, len = 100}, user_param_val = {s = 0x7f9dfe24d820 "8", len = 100}, 
          maddr_val = {s = 0x4afe47 "H\215C\030H\203\304\020[]A\\\303\017\037@", len = 1}, method_val = {
            s = 0x4b014c "H\203\304\030[]\303\017\037D", len = 1055}, lr_val = {
            s = 0x4afe47 "H\215C\030H\203\304\020[]A\\\303\017\037@", len = 58276176}, r2_val = {
            s = 0x7f9e0377eac8 "\020\353w\003\236\177", len = -703375952}, gr_val = {s = 0x472cd8 "H\307C\030", len = 0}, 
          u_name = {{s = 0x42772a "H\205\300H\211\303\017\204", <incomplete sequence \312>, len = 58190432}, {
              s = 0x7f9e03793950 "\v", len = -703375808}, {s = 0x44a777 "1\322H\205\300\017\204\236", len = 58189280}, {
              s = 0x2 <Address 0x2 out of bounds>, len = 0}, {s = 0x1 <Address 0x1 out of bounds>, len = 58276176}}, u_val = {
            {s = 0x7f9e03793950 "\v", len = -703375808}, {s = 0x2 <Address 0x2 out of bounds>, len = 58191992}, {
              s = 0x2 <Address 0x2 out of bounds>, len = 58276176}, {s = 0x0, len = 1443}, {s = 0x4695b5 "\213\035Em1", 
              len = 58276176}}, u_params_no = 22080}
        u = <optimized out>
        port = <optimized out>
        cmatch = <optimized out>
        aitem = <optimized out>
        adefault = <optimized out>
        spec = <optimized out>
        model = <optimized out>
        val = {rs = {s = 0x0, len = 0}, ri = 8060928, flags = 0}
        pve = <optimized out>
        name_s = {s = 0x7b24a0 "Tue Nov 12 20:50:59", len = 8070304}
        start = {tv_sec = 8070304, tv_usec = 8070329}
        route_params_bak = <optimized out>
        route_params_number_bak = <optimized out>
        __FUNCTION__ = "do_action"
#21 0x000000000041dca0 in run_action_list (a=0x0, msg=0x7f9e03793950) at action.c:171
        ret = 72
        t = 0x7f9e0377f138
#22 0x000000000041c221 in do_action (a=0x7f9e0377f210, msg=0x7f9e03793950) at action.c:1010
        val_s = {s = 0x7b24a0 "Tue Nov 12 20:50:59", len = -703375168}
        aux_counter = <optimized out>
        i = <optimized out>
        avp_val = {n = 52, s = {s = 0x3331303200000034 <Address 0x3331303200000034 out of bounds>, len = 2013}}
        j = <optimized out>
        key_number = <optimized out>
        adv_addr = {s = 0x7b24a0 "Tue Nov 12 20:50:59", len = -703375168}
        increment = <optimized out>
        decrement = <optimized out>
        avp_type = <optimized out>
        aux = {s = 0x7b24a0 "Tue Nov 12 20:50:59", len = -703375168}
        cdb_reply = <optimized out>
        val_number = <optimized out>
        it = <optimized out>
        avp_name = {n = 24, s = {s = 0x3000000018 <Address 0x3000000018 out of bounds>, len = -703374704}}
        ret = 1
        v = 1
        to = <optimized out>
        p = <optimized out>
        tmp = <optimized out>
        new_uri = <optimized out>
        end = <optimized out>
        crt = <optimized out>
        len = <optimized out>
        i = <optimized out>
        user = 0
        expires = 0
        vals = {{s = 0x7b24a0 "Tue Nov 12 20:50:59", len = -703375168}, {
            s = 0x7f9e03001380 "%.3s %.3s%3d %.2d:%.2d:%.2d %d\n", len = -703375056}, {s = 0x19 <Address 0x19 out of bounds>, 
            len = 7935670}, {s = 0x7b24a0 "Tue Nov 12 20:50:59", len = 49391266}, {
            s = 0x33313032fbad8001 <Address 0x33313032fbad8001 out of bounds>, len = 8070304}}
        result = {s = 0x398a <Address 0x398a out of bounds>, len = 35797166}
        uri = {user = {s = 0x7b24a0 "Tue Nov 12 20:50:59", len = 8070304}, passwd = {s = 0x7b24a0 "Tue Nov 12 20:50:59", 
            len = 8070329}, host = {s = 0x3000000018 <Address 0x3000000018 out of bounds>, len = -703374496}, port = {
            s = 0x7fffd6135aa0 "\377\377\377\377", len = 0}, params = {s = 0x0, len = 0}, headers = {s = 0x0, len = 0}, 
          port_no = 9376, proto = 123, type = ERROR_URI_T, transport = {s = 0x7f9e02f1a6a2 "H9\\$8t\bH\213T$(\306\002", 
            len = 0}, ttl = {s = 0x0, len = 24}, user_param = {s = 0x7fffd6135bc0 "P9y\003\236\177", len = -703374592}, 
          maddr = {s = 0x0, len = 24}, method = {s = 0x7fffd6135be0 "\260\\\023\326\377\177", len = -1}, lr = {
            s = 0x7fffd61359c0 "\031", len = 50336640}, r2 = {s = 0x7f9e03230fc0 "", len = 0}, gr = {
            s = 0x2 <Address 0x2 out of bounds>, len = 24}, transport_val = {s = 0x7fffd6135c20 "", len = -703374496}, 
          ttl_val = {s = 0x7fffd6135920 "\001\200\255\373\236\177", len = 8070304}, user_param_val = {
            s = 0x7fffd6135a10 "\271${", len = 50336640}, maddr_val = {s = 0x7fffd6135a80 "0", len = 25}, method_val = {
            s = 0x7f9e03793950 "\v", len = 8070304}, lr_val = {s = 0x7f9e02f1a6a2 "H9\\$8t\bH\213T$(\306\002", 
            len = -72515583}, r2_val = {s = 0x7b24a0 "Tue Nov 12 20:50:59", len = 8070304}, gr_val = {
            s = 0x7b24a0 "Tue Nov 12 20:50:59", len = 8070304}, u_name = {{s = 0x7b24b9 "", len = 8070329}, {
              s = 0x7b24a0 "Tue Nov 12 20:50:59", len = 8070329}, {s = 0x0, len = 0}, {s = 0x0, len = 0}, {s = 0x0, 
              len = 58278064}}, u_val = {{s = 0x7fffd61359e0 "\001\200\255", <incomplete sequence \373>, len = 8070304}, {
              s = 0x7fffd6135ad0 "\005\026y", len = 50336640}, {s = 0x7fffd6135b40 "0", len = 25}, {
              s = 0x5a3 <Address 0x5a3 out of bounds>, len = 8070304}, {s = 0x7f9e02f1a6a2 "H9\\$8t\bH\213T$(\306\002", 
              len = -72515583}}, u_params_no = 9376}
        next_hop = {user = {s = 0x7b24a0 "Tue Nov 12 20:50:59", len = 8070304}, passwd = {s = 0x7b24a0 "Tue Nov 12 20:50:59", 
            len = 8070329}, host = {s = 0x7b24b9 "", len = 8070304}, port = {s = 0x7b24b9 "", len = 0}, params = {s = 0x0, 
            len = 0}, headers = {s = 0x0, len = 0}, port_no = 9376, proto = 123, type = ERROR_URI_T, transport = {
            s = 0x1a <Address 0x1a out of bounds>, len = 7929856}, ttl = {s = 0x0, len = 58280568}, user_param = {
            s = 0x7f9e02efc513 "H\201\304", <incomplete sequence \330>, len = 48}, maddr = {s = 0x0, len = -703374688}, 
          method = {s = 0x4d842f "D\017\266\\$0A\200\373\b\017\207\t\t", len = -1}, lr = {
            s = 0x7f9e00000016 <Address 0x7f9e00000016 out of bounds>, len = 7935646}, r2 = {s = 0x7f9e03230fc0 "", len = 0}, 
          gr = {s = 0xc <Address 0xc out of bounds>, len = 7935493}, transport_val = {s = 0x2 <Address 0x2 out of bounds>, 
            len = 7936575}, ttl_val = {s = 0x0, len = 52635776}, user_param_val = {
            s = 0x7f9e02f160da "\203\370\001\031\322f\203;", len = 2}, maddr_val = {
            s = 0x4295c2 "H\201\304", <incomplete sequence \330>, len = 8070304}, method_val = {
            s = 0x1a <Address 0x1a out of bounds>, len = 14730}, lr_val = {s = 0x41f <Address 0x41f out of bounds>, len = 2}, 
          r2_val = {s = 0x7f9e02efc513 "H\201\304", <incomplete sequence \330>, len = 48}, gr_val = {
            s = 0x7fffd6135c40 "\240${", len = -703374496}, u_name = {{s = 0x1 <Address 0x1 out of bounds>, len = 20}, {
              s = 0x7f9e00000032 <Address 0x7f9e00000032 out of bounds>, len = 59}, {s = 0x7f9e02ff892d "Tue", 
              len = 50301346}, {s = 0xc <Address 0xc out of bounds>, len = 14730}, {s = 0x7f9e03793950 "\v", len = 14730}}, 
          u_val = {{s = 0x41f <Address 0x41f out of bounds>, len = 52635776}, {
              s = 0x7f9e02f160da "\203\370\001\031\322f\203;", len = 58276176}, {
              s = 0x4295c2 "H\201\304", <incomplete sequence \330>, len = 12555072}, {
              s = 0x3000000020 <Address 0x3000000020 out of bounds>, len = -703374160}, {s = 0x7fffd6135bf0 "\266\026y", 
              len = 7935670}}, u_params_no = 9380}
        u = <optimized out>
        port = <optimized out>
        cmatch = <optimized out>
        aitem = <optimized out>
        adefault = <optimized out>
        spec = <optimized out>
        model = <optimized out>
        val = {rs = {s = 0x7b24a0 "Tue Nov 12 20:50:59", len = -703375200}, ri = 50336640, flags = 858861618}
        pve = <optimized out>
        name_s = {s = 0x33313032fe4fc7a8 <Address 0x33313032fe4fc7a8 out of bounds>, len = 1}
        start = {tv_sec = 140312286593044, tv_usec = 140316581560370}
        route_params_bak = <optimized out>
        route_params_number_bak = <optimized out>
        __FUNCTION__ = "do_action"
#23 0x000000000041dca0 in run_action_list (a=0x0, a@entry=0x7f9e03773360, msg=0x7f9e03793950) at action.c:171
        ret = 72
        t = 0x7f9e0377f210
#24 0x000000000041dfd0 in run_actions (msg=0x7f9e03793950, a=0x7f9e03773360) at action.c:136
No locals.
#25 run_actions (msg=0x7f9e03793950, a=0x7f9e03773360) at action.c:190
No locals.
#26 run_top_route (a=0x7f9e03773360, msg=msg@entry=0x7f9e03793950) at action.c:211
        bl_last_msg_id = 11
        bk_action_flags = 0
        bk_rec_lev = 0
        ret = <optimized out>
#27 0x00000000004597bf in receive_msg (buf=<optimized out>, len=1055, rcv_info=rcv_info@entry=0x7fffd6135d90) at receive.c:171
        msg = 0x7f9e03793950
        start = {tv_sec = 140316639807400, tv_usec = 140316640708264}
        tmp = 0x398a <Address 0x398a out of bounds>
        __FUNCTION__ = "receive_msg"
#28 0x00000000004ab1ea in udp_rcv_loop () at udp_server.c:424
        len = <optimized out>
        buf = "INVITE sip:2165551212@192.168.54.238:5060;transport=udp;sipxecs-lineid=9 SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.54.238:1443;branch=z9hG4bK8106.4d014a42.0\r\nTo: sip:2165551212@192.168.54.238:5060\r\nFrom: \"Jeff"...
        tmp = 0x41f <Address 0x41f out of bounds>
        from = 0x7f9e0378c7a8
        fromlen = 16
        ri = {src_ip = {af = 2, len = 4, u = {addrl = {55536167104, 485331304458}, addr32 = {3996559552, 12, 10, 113}, 
              addr16 = {43200, 60982, 12, 0, 10, 0, 113, 0}, 
              addr = "\300\250\066\356\f\000\000\000\n\000\000\000q\000\000"}}, dst_ip = {af = 2, len = 4, u = {addrl = {
                3996559552, 0}, addr32 = {3996559552, 0, 0, 0}, addr16 = {43200, 60982, 0, 0, 0, 0, 0, 0}, 
              addr = "\300\250\066\356", '\000' <repeats 11 times>}}, src_port = 1443, dst_port = 1442, proto = 1, 
          proto_reserved1 = 0, proto_reserved2 = 0, src_su = {s = {sa_family = 2, 
              sa_data = "\005\243\300\250\066\356\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 41733, 
              sin_addr = {s_addr = 3996559552}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, 
              sin6_port = 41733, sin6_flowinfo = 3996559552, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, 
                  __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, 
          bind_address = 0x7f9e0376ecb0}
        p = <optimized out>
        msg = {
          s = 0x791620 "INVITE sip:2165551212@192.168.54.238:5060;transport=udp;sipxecs-lineid=9 SIP/2.0\r\nVia: SIP/2.0/UDP 192.168.54.238:1443;branch=z9hG4bK8106.4d014a42.0\r\nTo: sip:2165551212@192.168.54.238:5060\r\nFrom: \"Jef"..., len = 1055}
        __FUNCTION__ = "udp_rcv_loop"
#29 0x00000000004162cc in main_loop () at main.c:917
        i = 1
        pid = <optimized out>
        si = <optimized out>
        startup_done = <optimized out>
        chd_rank = 2
        rc = <optimized out>
        load_p = 0x7f9dfe508ef0
#30 main (argc=<optimized out>, argv=<optimized out>) at main.c:1598
        cfg_log_stderr = <optimized out>
        cfg_stream = 0xbf5010
        c = <optimized out>
        r = <optimized out>
        tmp = 0x7fffd6137e3a ""
        tmp_len = <optimized out>
        port = <optimized out>
        proto = <optimized out>
        options = 0x510290 "f:cCm:M:b:l:n:N:rRvdDFETSVhw:t:u:g:P:G:W:o:"
        ret = -1
        seed = 2866523381
        rfd = <optimized out>
        __FUNCTION__ = "main"
@jpyle490
Copy link
Author

Others have had success with e6f4a51. I did not.

Program terminated with signal 11, Segmentation fault.
#0 0x00000000004af780 in fm_malloc ()
(gdb) bt full
#0 0x00000000004af780 in fm_malloc ()
#1 0x0000000000440f1c in build_req_buf_from_sip_req ()
#2 0x000000000042cb71 in forward_request ()
#3 0x00007f0c8ebaee68 in t_relay_to () from /usr/lib/opensips/modules/tm.so
#4 0x00007f0c8ebbdcff in ?? () from /usr/lib/opensips/modules/tm.so
#5 0x0000000000419930 in do_action ()
#6 0x000000000041dca0 in run_action_list ()
#7 0x0000000000469903 in ?? ()
#8 0x0000000000468fd5 in eval_expr ()
#9 0x00000000004690c1 in eval_expr ()
#10 0x0000000000468fc1 in eval_expr ()
#11 0x000000000041962d in do_action ()
#12 0x000000000041dca0 in run_action_list ()
#13 0x000000000041c221 in do_action ()
#14 0x000000000041dca0 in run_action_list ()
#15 0x000000000041c221 in do_action ()
#16 0x000000000041dca0 in run_action_list ()
#17 0x000000000041dfd0 in run_top_route ()
#18 0x0000000000459927 in receive_msg ()
#19 0x00000000004aac0a in udp_rcv_loop ()
#20 0x00000000004162cc in main ()

@jpyle490
Copy link
Author

Issue #83 reads similar to this. So,
http://storageb.fidelityvoice.net/~jpyle/logs-1.zip

It has the memlog (1.3M) and the full backtrace. The first line could be telling:
CRITICAL:core:qm_free: freeing already freed pointer, first free: auth.c: uac_auth(187) - aborting

@bogdan-iancu
Copy link
Member

Hi Jeff,

Thanks for your report- I noted the also your email. The log and backtrace confirm that the original fix done by Ovidiu has some side-effect (double free in some cases). I will do some digging to find out the problem.

Regards,
Bogdan

@ghost ghost assigned bogdan-iancu Nov 20, 2013
bogdan-iancu added a commit that referenced this issue Nov 20, 2013
@bogdan-iancu
Do not free the header string as it is part of the lumps (and freed t…
b279e17 
…ogether with the lumps).

Reported by Jeff Pyle.
Fixing bug #126
bogdan-iancu added a commit that referenced this issue Nov 20, 2013
@bogdan-iancu
Do not free the header string as it is part of the lumps (and freed t…
d5f92dd 
…ogether with the lumps).

Reported by Jeff Pyle.
Fixing bug #126
bogdan-iancu added a commit that referenced this issue Nov 20, 2013
@bogdan-iancu
Do not free the header string as it is part of the lumps (and freed t…
e2b9332 
…ogether with the lumps).

Reported by Jeff Pyle.
Fixing bug #126
bogdan-iancu added a commit that referenced this issue Nov 20, 2013
@bogdan-iancu
Do not free the header string as it is part of the lumps (and freed t…
e3afb35 
…ogether with the lumps).

Reported by Jeff Pyle.
Fixing bug #126
@bogdan-iancu
Copy link
Member

Jeff, please update from GIT - all supported version were fix. Confirmation on the fix will be appreciated.

@jpyle490
Copy link
Author

Bogdan, the fix prevents the crash. Yay! However, I see the following warning from modules/uac_auth/auth.c:

    if (auth_hdr.s || auth_hdr.len)
            LM_WARN("potential memory leak at addr: %p\n", auth_hdr.s);

Memory use is stable for all the processes after a few dozen calls utilizing uac_auth(). False alarm?

bogdan-iancu added a commit that referenced this issue Nov 22, 2013
@bogdan-iancu
Fixed bogus warning because of variables not being reset.
cfb8457 
Reported by Jeff Pyle, part of fixing issue #126.
bogdan-iancu added a commit that referenced this issue Nov 22, 2013
@bogdan-iancu
Fixed bogus warning because of variables not being reset.
68976ca 
Reported by Jeff Pyle, part of fixing issue #126.
bogdan-iancu added a commit that referenced this issue Nov 22, 2013
@bogdan-iancu
Fixed bogus warning because of variables not being reset.
a48232c 
Reported by Jeff Pyle, part of fixing issue #126.
bogdan-iancu added a commit that referenced this issue Nov 22, 2013
@bogdan-iancu
Fixed bogus warning because of variables not being reset.
8143866 
Reported by Jeff Pyle, part of fixing issue #126.
@bogdan-iancu
Copy link
Member

Warnings fixed also, thank you, closing this report.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bogdan-iancu bogdan-iancu

Labels
bug
Projects
None yet
Milestone
1.10
Development

No branches or pull requests
2 participants
@bogdan-iancu @jpyle490