Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sql_cacher crashes if table/view is not found #1734

Closed
clifjones opened this issue Jun 18, 2019 · 0 comments
Closed

sql_cacher crashes if table/view is not found #1734

clifjones opened this issue Jun 18, 2019 · 0 comments
Assignees
@rvlad-patrascu

Comments

@clifjones
Copy link

clifjones commented Jun 18, 2019 •
edited

OpenSIPS version you are running

$ opensips -V
version: opensips 2.4.5 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: 6009742
main.c compiled on 22:49:28 Jun 13 2019 with gcc 7

Crash Core Dump

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f10262dcbbd in mi_reload (root=<optimized out>,
    param=<optimized out>) at sql_cacher.c:1064
1064			lock_start_write(db_hdls->c_entry->ref_lock);
Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.26-23.amzn2.x86_64 glibc-2.26-32.amzn2.0.1.x86_64 keyutils-libs-1.5.8-3.amzn2.0.2.x86_64 krb5-libs-1.15.1-20.amzn2.0.1.x86_64 libcom_err-1.42.9-12.amzn2.0.2.x86_64 libcrypt-2.26-32.amzn2.0.1.x86_64 libcurl-7.61.1-9.amzn2.0.1.x86_64 libidn2-2.0.4-1.amzn2.0.2.x86_64 libnghttp2-1.31.1-1.amzn2.0.2.x86_64 libselinux-2.5-12.amzn2.0.2.x86_64 libssh2-1.4.3-10.amzn2.1.2.x86_64 libunistring-0.9.3-9.amzn2.0.2.x86_64 nspr-4.19.0-1.amzn2.x86_64 nss-3.36.0-7.amzn2.x86_64 nss-util-3.36.0-1.amzn2.x86_64 openldap-2.4.44-15.amzn2.x86_64 openssl-libs-1.0.2k-16.amzn2.1.1.x86_64 pcre-8.32-17.amzn2.0.2.x86_64 postgresql-libs-9.2.24-1.amzn2.0.1.x86_64 zlib-1.2.7-17.amzn2.0.2.x86_64
(gdb) where
#0  0x00007f10262dcbbd in mi_reload (root=<optimized out>,
    param=<optimized out>) at sql_cacher.c:1064
#1  0x00007f102a809c05 in run_mi_cmd (cmd=0x7f103740de90, cmd=0x7f103740de90,
    param=0x7ffce9143a80, f=<optimized out>, t=0x7f1037429fc8)
    at ../../mi/mi.h:109
#2  mi_datagram_server (rx_sock=<optimized out>, tx_sock=<optimized out>)
    at datagram_fnc.c:616
#3  0x00007f102a80bee3 in datagram_process (rank=<optimized out>)
    at mi_datagram.c:356
#4  0x00000000004eb68b in start_module_procs () at sr_module.c:797
#5  0x000000000041be8e in main_loop () at main.c:761
#6  main (argc=<optimized out>, argv=<optimized out>) at main.c:1439
(gdb)
#0  0x00007f10262dcbbd in mi_reload (root=<optimized out>,
    param=<optimized out>) at sql_cacher.c:1064
#1  0x00007f102a809c05 in run_mi_cmd (cmd=0x7f103740de90, cmd=0x7f103740de90,
    param=0x7ffce9143a80, f=<optimized out>, t=0x7f1037429fc8)
    at ../../mi/mi.h:109
#2  mi_datagram_server (rx_sock=<optimized out>, tx_sock=<optimized out>)
    at datagram_fnc.c:616
#3  0x00007f102a80bee3 in datagram_process (rank=<optimized out>)
    at mi_datagram.c:356
#4  0x00000000004eb68b in start_module_procs () at sr_module.c:797
#5  0x000000000041be8e in main_loop () at main.c:761
#6  main (argc=<optimized out>, argv=<optimized out>) at main.c:1439
(gdb)

Describe the traffic that generated the bug
No traffic. Initial startup.

To Reproduce
This bug appeared in our Docker test environment. Opensips was running in a Docker container and exited when the PostgreSQL DB didn't have a VIEW created for the sql_cacher.

  1. Configure PostgreSQL DB to cache DB tables and/or views
  2. Have 1 missing table or view that is specified in the cache (In the config below, table5 does not exist in the DB)
  3. Start OpenSIPS
  4. Issue a opensips fifo sql_cacher_reload table5_cache command.
  5. OpenSIPS core dump/crash.

Relevant System Logs

[2019-06-18T14:02:25.470Z] Container: Jun 18 14:02:24 [312] DBG:core:handle_sigs: SIGINT received, program terminates
[2019-06-18T14:02:25.470Z] Container: Jun 18 14:02:24 [312] DBG:core:shutdown_opensips: Asking process 7 [SIP receiver udp:10.10.10.10:5060 ] to terminate
[2019-06-18T14:02:25.470Z] Container: Jun 18 14:02:24 [313] INFO:core:sig_usr: signal 15 received
[2019-06-18T14:02:25.470Z] Container: Jun 18 14:02:24 [314] INFO:core:sig_usr: signal 15 received
[2019-06-18T14:02:25.470Z] Container: Jun 18 14:02:24 [315] INFO:core:sig_usr: signal 15 received
[2019-06-18T14:02:25.470Z] Container: Jun 18 14:02:24 [316] INFO:core:sig_usr: signal 15 received
[2019-06-18T14:02:25.470Z] Container: Jun 18 14:02:24 [317] INFO:core:sig_usr: signal 15 received
[2019-06-18T14:02:25.470Z] Container: Jun 18 14:02:24 [318] INFO:core:sig_usr: signal 15 received
[2019-06-18T14:02:25.470Z] Container: Jun 18 14:02:24 [312] DBG:core:shutdown_opensips: Asking process 8 [SIP receiver udp:203.0.113.2:5060 ] to terminate
[2019-06-18T14:02:25.470Z] Container: Jun 18 14:02:24 [319] DBG:core:ipc_handle_job: received job type 0[RPC] from process 0
[2019-06-18T14:02:25.470Z] Container: Jun 18 14:02:24 [319] DBG:core:rpc_process_terminate: Process 7 exiting with code 0...
[2019-06-18T14:02:25.470Z] Container: Jun 18 14:02:24 [312] DBG:core:shutdown_opensips: Asking process 9 [SIP receiver udp:203.0.113.2:5060 ] to terminate
[2019-06-18T14:02:25.470Z] Container: Jun 18 14:02:24 [320] DBG:core:ipc_handle_job: received job type 0[RPC] from process 0
[2019-06-18T14:02:25.470Z] Container: Jun 18 14:02:24 [320] DBG:core:rpc_process_terminate: Process 8 exiting with code 0...
[2019-06-18T14:02:25.470Z] Container: Jun 18 14:02:24 [312] DBG:core:shutdown_opensips: Asking process 10 [SIP receiver udp:203.0.113.2:5060 ] to terminate
[2019-06-18T14:02:25.470Z] Container: Jun 18 14:02:24 [321] DBG:core:ipc_handle_job: received job type 0[RPC] from process 0
[2019-06-18T14:02:25.470Z] Container: Jun 18 14:02:24 [321] DBG:core:rpc_process_terminate: Process 9 exiting with code 0...
[2019-06-18T14:02:25.470Z] Container: Jun 18 14:02:24 [312] DBG:core:shutdown_opensips: Asking process 11 [SIP receiver udp:203.0.113.2:5060 ] to terminate
[2019-06-18T14:02:25.470Z] Container: Jun 18 14:02:24 [322] DBG:core:ipc_handle_job: received job type 0[RPC] from process 0
[2019-06-18T14:02:25.470Z] Container: Jun 18 14:02:24 [322] DBG:core:rpc_process_terminate: Process 10 exiting with code 0...
[2019-06-18T14:02:25.470Z] Container: Jun 18 14:02:24 [312] DBG:core:shutdown_opensips: Asking process 12 [SIP receiver udp:203.0.113.2:5060 ] to terminate
[2019-06-18T14:02:25.470Z] Container: Jun 18 14:02:24 [323] DBG:core:ipc_handle_job: received job type 0[RPC] from process 0
[2019-06-18T14:02:25.470Z] Container: Jun 18 14:02:24 [323] DBG:core:rpc_process_terminate: Process 11 exiting with code 0...
[2019-06-18T14:02:25.470Z] Container: Jun 18 14:02:24 [312] DBG:core:shutdown_opensips: Asking process 13 [SIP receiver udp:203.0.113.2:5060 ] to terminate
[2019-06-18T14:02:25.470Z] Container: Jun 18 14:02:24 [324] DBG:core:ipc_handle_job: received job type 0[RPC] from process 0
[2019-06-18T14:02:25.470Z] Container: Jun 18 14:02:24 [324] DBG:core:rpc_process_terminate: Process 12 exiting with code 0...
[2019-06-18T14:02:25.470Z] Container: Jun 18 14:02:24 [336] DBG:core:io_wait_loop_epoll: [TCP_main] EPOLLHUP on IN ->connection closed by the remote peer!
[2019-06-18T14:02:25.470Z] Container: Jun 18 14:02:24 [336] DBG:core:receive_fd: EOF on 71
[2019-06-18T14:02:25.470Z] Container: Jun 18 14:02:24 [336] DBG:core:io_watch_del: [TCP_main] io_watch_del op on index 15 71 (0x95aa60, 71, 15, 0x0,0x1) fd_no=35 called

OS/environment information
Docker version 18.09.2, build 6247962

Relevant parts of our config:

loadmodule "db_postgres.so"
modparam("db_postgres", "exec_query_threshold", 500000)
#modparam("db_postgres", "timeout", 5)

loadmodule "db_virtual.so"
modparam("db_virtual", "db_urls", "define set1 CFG_DB_FAILOVER")
modparam("db_virtual", "db_urls", "DB URLS here")
modparam("db_virtual", "db_probe_time", 10)
modparam("db_virtual", "db_max_consec_retrys", 5)

modparam("cachedb_local", "cachedb_url", "local:table1:///table1")
modparam("cachedb_local", "cachedb_url", "local:table2:///table2")
modparam("cachedb_local", "cachedb_url", "local:table3:///table3")
modparam("cachedb_local", "cachedb_url", "local:table4:///table4")
modparam("cachedb_local", "cachedb_url", "local:table5:///table5")
modparam("cachedb_local", "cache_collections", "DIALOG_CACHE=20;table1=20;table2=8;table3=20;table4=20;table5=20")

loadmodule "sql_cacher.so"
modparam("sql_cacher", "sql_fetch_nr_rows", 1000)
#// table1 cache
modparam("sql_cacher", "cache_table",
"id=table1_cache
db_url=virtual://set1
cachedb_url=local:table1:///table1
table=table1
key=phone_number
columns=col1 col2 col3 col4 col5 col6 col7
on_demand=0")
#// table2
modparam("sql_cacher", "cache_table",
"id=table2_cache
db_url=virtual://set1
cachedb_url=local:table2:///table2
table=table2
key=key
columns=col1
on_demand=0")
#// table3
modparam("sql_cacher", "cache_table",
"id=table3_cache
db_url=virtual://set1
cachedb_url=local:table3:///table3
table=table3
key=key
columns=col1
on_demand=0")
#// table4
modparam("sql_cacher", "cache_table",
"id=table4_cache
db_url=virtual://set1
cachedb_url=local:table4:///table4
table=table4
key=key
columns=col1 col2 col3 col4
on_demand=0")
#// table5
modparam("sql_cacher", "cache_table",
"id=table5_cache
db_url=virtual://set1
cachedb_url=local:table5:///table5
table=table5
key=key
columns=col1 col2 col3 col4 col5
on_demand=0")

Additional context
@clifjones clifjones changed the title db_virtual crashes if table/view is not found sql_cacher crashes if table/view is not found Jun 18, 2019
@rvlad-patrascu rvlad-patrascu self-assigned this Jun 21, 2019
rvlad-patrascu added a commit that referenced this issue Jun 25, 2019
@rvlad-patrascu
sql_cacher: fix crashes when failing to validate db connections
dc53d72 
Fixes: #1734
(cherry picked from commit 82c446d)
rvlad-patrascu added a commit that referenced this issue Jun 25, 2019
@rvlad-patrascu
sql_cacher: fix crashes when failing to validate db connections
ab7e931 
Fixes: #1734
(cherry picked from commit 82c446d)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rvlad-patrascu rvlad-patrascu

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rvlad-patrascu @clifjones