Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix crash and memory errors reported by valgrind for v2.1.1 #3334

Closed
4 tasks done
JavierJF opened this issue Mar 4, 2021 · 0 comments · Fixed by #3335
Closed
4 tasks done

Fix crash and memory errors reported by valgrind for v2.1.1 #3334

JavierJF opened this issue Mar 4, 2021 · 0 comments · Fixed by #3335

Comments

@JavierJF
Copy link
Collaborator

JavierJF commented Mar 4, 2021

  • A clear description of the issue

After a crash detected in the CI with backtrace:

(gdb) bt
#0  __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:500
#1  0x0000563709c468c1 in memcpy (__len=16777215, __src=<optimized out>, __dest=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/string_fortified.h:34
#2  MySQL_ResultSet::add_row2 (this=this@entry=0x7fc94fa2b000, row=row@entry=0x7fc8a4f77bd8, offset=<optimized out>,
    offset@entry=0x7fc753200540 "\377\377\377\r") at MySQL_Protocol.cpp:2852
#3  0x0000563709c49c73 in MySQL_ResultSet::init_with_stmt (this=this@entry=0x7fc94fa2b000) at MySQL_Protocol.cpp:2727
#4  0x0000563709c25a0d in MySQL_Session::MySQL_Stmt_Result_to_MySQL_wire (this=0x7fc94fa3a200, stmt=0x7fc94fa0e800, myconn=0x7fc94fa39f00)
    at MySQL_Session.cpp:6344
#5  0x0000563709c26b85 in MySQL_Session::handler_rc0_PROCESSING_STMT_EXECUTE (this=this@entry=0x7fc94fa3a200, myds=myds@entry=0x7fc94fa48c00)
    at MySQL_Session.cpp:3878
#6  0x0000563709c3c20d in MySQL_Session::handler (this=this@entry=0x7fc94fa3a200) at MySQL_Session.cpp:4459
#7  0x0000563709c15a4c in MySQL_Thread::process_all_sessions (this=this@entry=0x7fc94fa00000) at MySQL_Thread.cpp:4883
#8  0x0000563709c1d8a8 in MySQL_Thread::run (this=0x7fc94fa00000) at MySQL_Thread.cpp:4417
#9  0x0000563709bbf7c4 in mysql_worker_thread_func (arg=0x7fc9532dc310) at main.cpp:769
#10 0x00007fc953f51609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#11 0x00007fc953b25103 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Full coredump from this crash omitted due to size

A research was conducted to get the cause of the failure, a fast inspection of the dump pointed out to the recent changes required to 'ps_buffer' (see #3324 for more context into previous related issue):

(gdb) p *MyRS->stmt
$3 = {mem_root = {free = 0x7fc94fa02800, used = 0x0, pre_alloc = 0x7fc94fa02800, min_malloc = 32, block_size = 2024, block_num = 4, first_block_usage = 0,
    error_handler = 0x0}, mysql = 0x7fc94fa6d900, stmt_id = 3, flags = 0, state = MYSQL_STMT_USE_OR_STORE_CALLED, fields = 0x7fc94fa03250, field_count = 3,
  param_count = 0, send_types_to_server = 0 '\000', params = 0x0, bind = 0x7fc94fa034a0, result = {data = 0x7fc8a4f77bd8, embedded_info = 0x0, alloc = {
      free = 0x7fc94fb2ac00, used = 0x7fc81c100c00, pre_alloc = 0x7fc94fb2ac00, min_malloc = 32, block_size = 4072, block_num = 9, first_block_usage = 0,
      error_handler = 0x0}, rows = 3, fields = 0, extension = 0x0}, result_cursor = 0x7fc8a4f77bd8, bind_result_done = 0 '\000', bind_param_done = 1 '\001',
  upsert_status = {warning_count = 0, server_status = 2082, affected_rows = 3, last_insert_id = 0}, last_errno = 0, last_error = '\000' <repeats 512 times>,
  sqlstate = "00000", update_max_length = 1 '\001', prefetch_rows = 1, list = {prev = 0x0, next = 0x7fc94fa35320, data = 0x7fc94fa0e800},
  cursor_exists = 0 '\000', extension = 0x7fc94fa0b040, fetch_row_func = 0x56370a067afd <stmt_buffered_fetch>, execute_count = 1,
  default_rset_handler = 0x56370a068c63 <_mysql_stmt_use_result>, m = 0x0, array_size = 0, row_size = 0, prebind_params = 0, user_data = 0x0,
  result_callback = 0x0, param_callback = 0x0}
(gdb) p *MyRS->stmt->result.data
$4 = {next = 0x7fc895cd53d8, data = 0x7fc8a4f77bf0, length = 3474867094363060023}

Data on the 'stmt->result' appears to be invalid. Trying to reproduce the issue isolated in the test representing the POC (test_ps_async-t.cpp) revealed after replicating different operation conditions through the following query:

(SELECT id, k, REPEAT(c,2000) cc FROM test.sbtest1 LIMIT 10) 
UNION  (SELECT id, k, REPEAT(c,10) cc FROM test.sbtest1 LIMIT 10)

This query was trying to enforce a particular memory usage pattern, this revealed that there were several memory operations being performed over uninitialized values:

==1045151== 1 errors in context 1 of 9:
==1045151== Conditional jump or move depends on uninitialised value(s)
==1045151==    at 0x26450A: ma_alloc_root (ma_alloc.c:81)
==1045151==    by 0x265016: mthd_stmt_read_all_rows (mariadb_stmt.c:237)
==1045151==    by 0x26A637: mysql_stmt_store_result (mariadb_stmt.c:1824)
==1045151==    by 0x27DDC1: mysql_stmt_store_result_start_internal (mariadb_async.c:1469)
==1045151==    by 0x280022: my_context_spawn (ma_context.c:201)
==1045151==  Uninitialised value was created by a heap allocation
==1045151==    at 0x483E77F: malloc (vg_replace_malloc.c:307)
==1045151==    by 0x1B840F: ma_alloc_root(st_ma_mem_root*, unsigned long) (test_ps_async-t.cpp:77)
==1045151==    by 0x1B9A76: main (test_ps_async-t.cpp:382)

The root of this invalid operations should be fixed.

  • ProxySQL version

v2.1.1

  • OS version
NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
  • The steps to reproduce the issue

In the description above.

renecannao added a commit that referenced this issue Mar 11, 2021
Closes #3334: Fix crash and memory errors reported by valgrind for v2.1.1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant