Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

usrloc: core dump #1900

Closed
btriller opened this issue Mar 21, 2019 · 2 comments

Comments

Projects
None yet
2 participants
@btriller
Copy link
Contributor

commented Mar 21, 2019

Description

Kamailio crashes while querying for unexpired contacts in a location table while it is saving contacts to db.
Seems to be a regression, since this didn't happened with Kamailio 4.4.

Reproduction

Constantly flush many locations and simultaneously get location statistics from usrloc module via kamcmd ul.db_users:

shell1 % while kamcmd ul.flush ; do : ; done
shell2 % curl -H content-type:\ application/json -s -d '{"jsonrpc":"2.0", "method":"ul.db_contacts", "params": ["location"]}' localhost:5060/JSONRPC

Debugging Data

#0  0x00007f600e3345ad in ul_rpc_db_contacts (rpc=0x7f6004f8b960 <func_param>, ctx=0x7f6004f8b8e0 <_jsonrpc_ctx>) at ul_rpc.c:774
#1  0x00007f6004d6c15c in jsonrpc_dispatch (msg=0x7ffc7fc4e250, s1=0x0, s2=0x0) at jsonrpcs_mod.c:1251
#2  0x000055c674d60eb9 in do_action (h=0x7ffc7fc4e170, a=0x7f6012607828, msg=0x7ffc7fc4e250) at core/action.c:1054
#3  0x000055c674d6ea4d in run_actions (h=0x7ffc7fc4e170, a=0x7f6012607828, msg=0x7ffc7fc4e250) at core/action.c:1552
#4  0x000055c674d60e25 in do_action (h=0x7ffc7fc4e170, a=0x7f6012607ac8, msg=0x7ffc7fc4e250) at core/action.c:1045
#5  0x000055c674d6ea4d in run_actions (h=0x7ffc7fc4e170, a=0x7f6012600b28, msg=0x7ffc7fc4e250) at core/action.c:1552
#6  0x000055c674d60e25 in do_action (h=0x7ffc7fc4e170, a=0x7f6012607d00, msg=0x7ffc7fc4e250) at core/action.c:1045
#7  0x000055c674d6ea4d in run_actions (h=0x7ffc7fc4e170, a=0x7f60125fecf8, msg=0x7ffc7fc4e250) at core/action.c:1552
#8  0x00007f6004f913b4 in xhttp_process_request (orig_msg=0x7f6012622410, new_buf=0x7f6012638ad0 "POST /JSONRPC HTTP/1.1\r\nVia: SIP/2.0/TCP 127.0.0.1:58590\r\nHost: localhost:5060\r\nUser-Agent: curl/7.52.1\r\nAccept: */*\r\nContent-Length: 67\r\nContent-Type: application/x-www-form-urlencoded\r\n\r\n{\"jsonrpc\": \"2.0\", \"method\":\"ul.db_contacts\",\"params\":[\"location\"]}", new_len=256) at xhttp_mod.c:282
#9  0x00007f6004f9253b in xhttp_handler (msg=0x7f6012622410) at xhttp_mod.c:357
#10 0x000055c674e4be83 in nonsip_msg_run_hooks (msg=0x7f6012622410) at core/nonsip_hooks.c:111
#11 0x000055c674e944d7 in receive_msg (buf=0x7f5feac686a8 "POST /JSONRPC HTTP/1.1\r\nHost: localhost:5060\r\nUser-Agent: curl/7.52.1\r\nAccept: */*\r\nContent-Length: 67\r\nContent-Type: application/x-www-form-urlencoded\r\n\r\n{\"jsonrpc\": \"2.0\", \"method\":\"ul.db_contacts\",\"params\":[\"location\"]}", len=222, rcv_info=0x7f5feac683c8) at core/receive.c:202
#12 0x000055c674f2e43c in receive_tcp_msg (tcpbuf=0x7f5feac686a8 "POST /JSONRPC HTTP/1.1\r\nHost: localhost:5060\r\nUser-Agent: curl/7.52.1\r\nAccept: */*\r\nContent-Length: 67\r\nContent-Type: application/x-www-form-urlencoded\r\n\r\n{\"jsonrpc\": \"2.0\", \"method\":\"ul.db_contacts\",\"params\":[\"location\"]}", len=222, rcv_info=0x7f5feac683c8, con=0x7f5feac683b0) at core/tcp_read.c:1263
#13 0x000055c674f30a70 in tcp_read_req (con=0x7f5feac683b0, bytes_read=0x7ffc7fc4ee68, read_flags=0x7ffc7fc4ee6c) at core/tcp_read.c:1478
#14 0x000055c674f339bb in handle_io (fm=0x7f60126506b0, events=1, idx=-1) at core/tcp_read.c:1652
#15 0x000055c674f23d9b in io_wait_loop_epoll (h=0x55c6753e0fa0 <io_w>, t=2, repeat=0) at core/io_wait.h:1065
#16 0x000055c674f359b0 in tcp_receive_loop (unix_sock=54) at core/tcp_read.c:1822
#17 0x000055c674e14095 in tcp_init_children () at core/tcp_main.c:4802
#18 0x000055c674d216e1 in main_loop () at main.c:1713
#19 0x000055c674d286a9 in main (argc=13, argv=0x7ffc7fc4f4f8) at main.c:2642

Log Messages

ERROR: usrloc [urecord.c:431]: wb_timer(): updating contact in db failed (aor: 1234@example.com)
ERROR: db_mysql [km_dbase.c:127]: db_mysql_submit_query(): driver error on query: Commands out of sync; you can't run this command now (2014)
ERROR: <core> [db_query.c:339]: db_do_update(): error while submitting query
ERROR: usrloc [ucontact.c:1378]: db_update_ucontact_instance(): updating database failed

Additional Information

  • Kamailio Version - output of kamailio -v
version: kamailio 5.0.7 (x86_64/linux)
flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown
compiled with gcc 6.3.0
  • Operating System:
Debian 9.5

miconda added a commit that referenced this issue Mar 21, 2019

@miconda

This comment has been minimized.

Copy link
Member

commented Mar 21, 2019

Can you try with the patch referenced above? For now it is only in master branch, if solves it, then I will backport.

The patch is with safety checks for db result to avoid a crash. Code in 4.4, 5.0 and even master is pretty much the same for this rpc function, so I guess is rather a matter of getting mysql out of sync, than the version of kamailio. You are sending rpc commands at high rate in a loop in shell, many of the rpc commands were designed more for human/admin interaction, not considering high rate processing. I haven't developed the db rpc function in usrloc, never used them before, so I am not sure about their purpose. They are also restricted to sql backends, using raw query internally, something that is not recommended to be done in C code.

@miconda

This comment has been minimized.

Copy link
Member

commented Apr 1, 2019

Closing here. Reopen if still an issue after the patch pushed for it.

@miconda miconda closed this Apr 1, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.