Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crash on qm_debug_check_frag() on INVITE #2503

Closed
rnatella opened this issue Oct 9, 2020 · 7 comments
Closed

Crash on qm_debug_check_frag() on INVITE #2503

rnatella opened this issue Oct 9, 2020 · 7 comments

Comments

@rnatella
Copy link

rnatella commented Oct 9, 2020

Description

I am experimenting with fuzzing on Kamailio SIP. A malformed INVITE (with a long tag) crashes the server, raised by qm_debug_check_frag().

Troubleshooting

The error message:

qm_debug_check_frag(): BUG: qm: fragm. 0x7ffff03642e8 (address 0x7ffff0364320) end overwritten (9191919191919191, 9191919191919191)! Memory allocator was called from tm: t_reply.c:2410. Fragment marked by tm: t_msgbuilder.c:327. Exec from core/mem/q_malloc.c:511.

Output from GDB:

(gdb) watch *0x7ffff0364320
Hardware watchpoint 1: *0x7ffff0364320
(gdb) run -f kamailio-basic.cfg -L src/modules -Y runtime_dir/ -n 1 -D -E
Starting program: /home/rnatella/workdir-sip/kamailio/src/kamailio -f kamailio-basic.cfg -L src/modules -Y runtime_dir/ -n 1 -D -E
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
 0(29450) INFO: <core> [core/sctp_core.c:75]: sctp_core_check_support(): SCTP API not enabled - if you want to use it, load sctp module
Listening on
             udp: 127.0.0.1 [127.0.0.1]:5060
Aliases:

WARNING: no fork mode
 0(29450) INFO: rr [./../outbound/api.h:52]: ob_load_api(): unable to import bind_ob - maybe module is not loaded
 0(29450) INFO: rr [rr_mod.c:185]: mod_init(): outbound module not available
 0(29450) INFO: <core> [main.c:2841]: main(): processes (at least): 4 - shm size: 67108864 - pkg size: 8388608
 0(29450) INFO: <core> [core/udp_server.c:154]: probe_max_receive_buffer(): SO_RCVBUF is initially 212992
 0(29450) INFO: <core> [core/udp_server.c:206]: probe_max_receive_buffer(): SO_RCVBUF is finally 425984
 0(29450) ERROR: {1 2 INVITE 1-670@127.0.0.1} <core> [core/parser/parse_rr.c:78]: do_parse_rr_body(): Failed parsing name-addr (<sip:127.0"0tttttttttttttttttttttttttttK-670-1-7)
 0(29450) ERROR: {1 2 INVITE 1-670@127.0.0.1} <core> [core/parser/parse_rr.c:140]: do_parse_rr_body(): Failed parsing rr header body [<sip:127.0"0tttttttttttttttttttttttttttK-670-1-7]
 0(29450) ERROR: {1 2 INVITE 1-670@127.0.0.1} rr [loose.c:468]: find_rem_target(): failed to parse last Route HF
 0(29450) ERROR: {1 2 INVITE 1-670@127.0.0.1} rr [loose.c:700]: after_strict(): searching for last Route URI failed

Hardware watchpoint 1: *0x7ffff0364320

Old value = <unreadable>
New value = 4932352
__memmove_sse2_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:316
316     ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S: No such file or directory.
(gdb) bt
#0  __memmove_sse2_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:316
#1  0x00007ffff602e4ff in build_local_reparse (Trans=<optimized out>, branch=<optimized out>, len=<optimized out>, method=<optimized out>,
    method_len=<optimized out>, to=<optimized out>, reason=<optimized out>) at t_msgbuilder.c:336
#2  0x00007ffff607eaff in build_ack (rpl=<optimized out>, trans=<optimized out>, branch=0, ret_len=<optimized out>) at t_reply.c:360
#3  reply_received (p_msg=<optimized out>) at t_reply.c:2398
#4  0x00000000005a0af3 in do_forward_reply (msg=0x7ffff68eed28, mode=<optimized out>) at core/forward.c:757
#5  0x0000000000688cae in receive_msg (buf=<optimized out>, len=<optimized out>, rcv_info=<optimized out>) at core/receive.c:509
#6  0x00000000004a6b39 in udp_rcv_loop () at core/udp_server.c:543
#7  0x000000000042c938 in main_loop () at main.c:1480
#8  0x000000000043c574 in main (argc=<optimized out>, argv=<optimized out>) at main.c:2863
(gdb) frame 1
#1  0x00007ffff602e4ff in build_local_reparse (Trans=<optimized out>, branch=<optimized out>, len=<optimized out>, method=<optimized out>,
    method_len=<optimized out>, to=<optimized out>, reason=<optimized out>) at t_msgbuilder.c:336
336             append_str(d, method, method_len);
(gdb) print d
$1 = 0x7ffff0364320 ""
(gdb) c
Continuing.

Hardware watchpoint 1: *0x7ffff0364320

Old value = 4932352
New value = 4932417
__memmove_sse2_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:317
317     ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S: No such file or directory.
(gdb) bt
#0  __memmove_sse2_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:317
#1  0x00007ffff602e4ff in build_local_reparse (Trans=<optimized out>, branch=<optimized out>, len=<optimized out>, method=<optimized out>,
    method_len=<optimized out>, to=<optimized out>, reason=<optimized out>) at t_msgbuilder.c:336
#2  0x00007ffff607eaff in build_ack (rpl=<optimized out>, trans=<optimized out>, branch=0, ret_len=<optimized out>) at t_reply.c:360
#3  reply_received (p_msg=<optimized out>) at t_reply.c:2398
#4  0x00000000005a0af3 in do_forward_reply (msg=0x7ffff68eed28, mode=<optimized out>) at core/forward.c:757
#5  0x0000000000688cae in receive_msg (buf=<optimized out>, len=<optimized out>, rcv_info=<optimized out>) at core/receive.c:509
#6  0x00000000004a6b39 in udp_rcv_loop () at core/udp_server.c:543
#7  0x000000000042c938 in main_loop () at main.c:1480
#8  0x000000000043c574 in main (argc=<optimized out>, argv=<optimized out>) at main.c:2863
(gdb) c
Continuing.

Hardware watchpoint 1: *0x7ffff0364320

Old value = 4932417
New value = 541803329
0x00007ffff602e504 in build_local_reparse (Trans=<optimized out>, branch=<optimized out>, len=<optimized out>, method=<optimized out>,
    method_len=<optimized out>, to=<optimized out>, reason=<optimized out>) at t_msgbuilder.c:337
337             *d = ' ';
(gdb) bt
#0  0x00007ffff602e504 in build_local_reparse (Trans=<optimized out>, branch=<optimized out>, len=<optimized out>, method=<optimized out>,
    method_len=<optimized out>, to=<optimized out>, reason=<optimized out>) at t_msgbuilder.c:337
#1  0x00007ffff607eaff in build_ack (rpl=<optimized out>, trans=<optimized out>, branch=0, ret_len=<optimized out>) at t_reply.c:360
#2  reply_received (p_msg=<optimized out>) at t_reply.c:2398
#3  0x00000000005a0af3 in do_forward_reply (msg=0x7ffff68eed28, mode=<optimized out>) at core/forward.c:757
#4  0x0000000000688cae in receive_msg (buf=<optimized out>, len=<optimized out>, rcv_info=<optimized out>) at core/receive.c:509
#5  0x00000000004a6b39 in udp_rcv_loop () at core/udp_server.c:543
#6  0x000000000042c938 in main_loop () at main.c:1480
#7  0x000000000043c574 in main (argc=<optimized out>, argv=<optimized out>) at main.c:2863
(gdb) c
Continuing.
 0(29450) CRITICAL: {2 2 INVITE 1-670@127.0.0.1} <core> [core/mem/q_malloc.c:138]: qm_debug_check_frag(): BUG: qm: fragm. 0x7ffff03642e8 (address 0x7ffff0364320) end overwritten (9191919191919191, 9191919191919191)! Memory allocator was called from tm: t_reply.c:2410. Fragment marked by tm: t_msgbuilder.c:327. Exec from core/mem/q_malloc.c:511.

Program received signal SIGSEGV, Segmentation fault.
0x000000000082f45f in qm_status (qmp=<optimized out>) at core/mem/q_malloc.c:902
902                                     f!=&(qm->free_hash[h].head); f=f->u.nxt_free, i++, j++){

Reproduction

I am running the server with a basic configuration (attached kamailio-basic.cfg), using the command:

./src/kamailio -f kamailio-basic.cfg -L src/modules -Y runtime_dir/ -n 1 -D -E

kamailio-basic.cfg.txt

On the same machine, I am sending the malformed message (also attached):

cat sip-crash.txt | nc -4u -w1 localhost 5060

sip-crash.txt

You can find more information about my fuzzing setup at: https://github.com/rnatella/aflnet-kamailio-sip

Debugging Data

See previous section

Log Messages

See previous section

SIP Traffic

See previos section

Possible Solutions

Additional Information

  • Kamailio Version - output of kamailio -v
version: kamailio 5.5.0-dev2 (x86_64/linux) 6049a1-dirty
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, DBG_QM_MALLOC, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: 6049a1 -dirty
compiled on 10:12:13 Oct  9 2020 with /home/rnatella/aflnet/afl-clang-fast 6.0
  • Operating System:
Ubuntu 18.04.2 LTS

Linux dockertest1 4.15.0-109-generic #110-Ubuntu SMP Tue Jun 23 02:39:32 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
@henningw
Copy link
Contributor

henningw commented Oct 9, 2020

Thanks for the report. If you wanted to include some description about the fuzzing setup, it is missing.
In case you find more crashes with your fuzzing, please send a report of them to the e-mail address mentioned in our wiki instead of opening an issue on the public tracker.

@rnatella
Copy link
Author

rnatella commented Oct 9, 2020

Fixed the missing link about the fuzzing setup. Thanks for the pointer!

@henningw henningw changed the title Crash on qm_debug_check_frag() Crash on qm_debug_check_frag() on INVITE Oct 9, 2020
@miconda
Copy link
Member

miconda commented Oct 9, 2020

This happens when the INVITE is forwarded and not answered with 200ok, isn't it? Does it happen every time? If you get a core file, can you get the gdb output for bt full?

@rnatella
Copy link
Author

rnatella commented Oct 9, 2020

The crash happens every time, regardless that another SIP client ("33" in my setup) is registered or not. You can find attached bt full. Despite I compiled with "make mode=debug all", most variables are optimized out.
gdb.txt

@miconda
Copy link
Member

miconda commented Oct 9, 2020

I pushed some commits to master branch to catch it, can you try and see if it is fixed in your tests?

@rnatella
Copy link
Author

Seems fixed, no more crashes!

The output:

Call-I+: 1-670@127.0.0.1
CSeq: 2 INVITE
Contact.0.1:5061
Max-Forward]
 0(26164) WARNING: <core> [core/receive.c:317]: receive_msg(): parsing relevant headers failed
 0(26164) ERROR: <core> [core/parser/msg_parser.c:397]: parse_headers(): duplicate From header field [From: <sip:30@127.0.0.1>;tag=:
To: <sip:33@127.0.0.1>;tag=gggggggg���������������������������������]
 0(26164) ERROR: pv [pv_core.c:1965]: pv_get_hdr(): error parsing headers
 0(26164) ERROR: <core> [core/parser/msg_parser.c:397]: parse_headers(): duplicate From header field [From: <sip:30@127.0.0.1>;tag=:
To: <sip:33@127.0.0.1>;tag=gggggggg���������������������������������]
 0(26164) ERROR: pv [pv_core.c:731]: pv_get_callid(): cannot parse Call-Id header
 0(26164) ERROR: {1 <null> <null>} <core> [core/parser/msg_parser.c:397]: parse_headers(): duplicate From header field [From: <sip:30@127.0.0.1>;tag=:
To: <sip:33@127.0.0.1>;tag=gggggggg���������������������������������]
 0(26164) ERROR: {1 <null> <null>} maxfwd [mf_funcs.c:51]: is_maxfwd_present(): parsing MAX_FORWARD header failed!
 0(26164) ERROR: {1 <null> <null>} <core> [core/parser/msg_parser.c:397]: parse_headers(): duplicate From header field [From: <sip:30@127.0.0.1>;tag=:
To: <sip:33@127.0.0.1>;tag=gggggggg���������������������������������]
 0(26164) ERROR: {1 <null> <null>} <core> [core/msg_translator.c:2394]: build_res_buf_from_sip_req(): alas, parse_headers failed

@miconda
Copy link
Member

miconda commented Oct 12, 2020

Thanks for testing and feedback. Related commits were backported to stable branches.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants