PS-8865: Migrating keyring_vault plugin to component #12
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…UT_LIST_GET_LEN(buf_pool->flush_list) == 0 https://jira.percona.com/browse/PS-8174 Problem: There is a possibility that at shutdown by the time we do the last sweep on flushing the buffer pool there are still pages in the flush list. Those pages are still marked as io_fix->BUF_IO_READ thus they are not eligible for flushing from flush_list. Where is the workflow: 1. ibuf_merge_in_background requested those pages to be read in order to merge the ibuf changes. This will mark the page as BUF_IO_READ and increment buf_pool->n_pend_reads by 1. 2. When IO threads pick them up, it will start to merge the insert bugger changes. 3. On the first change, it will add the page to flush_list. 4. If there are more changes to apply, it will and continue on applying the changes until it is done. 5. Once the io thread finishes applying ibuf records to this page, it will mark the page as BUF_IO_NONE 6. the io thread decreases buf_pool->n_pend_reads by 1. The last sweep on flushing buffer pool considers the round of flushes completed when n_flushed == 0 which is not correct, if it runs when we are at step 4. Also, there is a still another race condition that by the time we tried to flush a page it was still marked as BUF_IO_READ (step 5), but when the page cleaner code checks buf_get_n_pending_read_ios, the IO thread has already decremented leaving one last page on flush_list. Fix: For the last round flushing pages at shutdown, only consider the flush completed if there are no pending read io operations and flush_list size is 0. Added a new function to get flush_list lenght. For this last sweep from PC we do not need mutex as there won't be other threads reading nor writting to it at this stage.
… to allow user to specify a consumer name) https://jira.percona.com/browse/PS-8385 Problem: Currently implementation of redo log consumer UDF does not allow users to specify a custom name for the consumer. This can be misleading when ER_IB_MSG_LOG_WRITER_WAIT_ON_CONSUMER error is issued as it will always report MEB as consumer name. Fix: Adjust the UDF to allow users to optionally specify a consumer name. In case of no consumer name specified, assume MEB. Also adjusted Log_consumer class to have a consumer_type ENUM. This is used log_writer_wait_on_consumers to validate we are not waiting on CONSUMER_TYPE_SERVER (log archive or checkpointer).
…fe.sh MySQL 8.0 has added handling of MYSQLD_RESTART_EXIT (16) which conflicts with handling of mysqld's return values ported from PS-5.7. This patch fixes `main.restart_server` MTR test.
…r generated and upgraded sys tables) https://jira.percona.com/browse/PS-7811 Test main.upgrade_system_tables complained that explicit_encryption system table option is different for freshly generated system tables and ones upgraded from 5.7. The reason for this is how new table creation in mysql_system_tables.sql and table upgrade in mysql_system_tables_fix.sql is done. In mysql_system_tables.sql there is something similar to SET @cmd = "CREATE TABLE IF NOT EXISTS columns_priv ( ... )"; SET @str = CONCAT(@cmd, " ENCRYPTION='", @is_mysql_encrypted, "'"); Here ENCRYPTION for a table is set explicitly not depending if it is ON or OFF. As a result we always have explicit_encryption=1 for generated table and this explicit settings will be used to decide if table should be encrypted or not. At the same time in mysql_system_tables_fix.sql we have SET @str="ALTER TABLE mysql.columns_priv ENCRYPTION='Y'"; SET @cmd = IF(STRCMP(@is_mysql_encrypted,'Y'), 'SET @dummy = 0', @str); So this ALTER isn't performed in case encryption is disabled and we get explicit_encryption=0 option for upgraded table. This will lead to actual table encryption setting being taken from global defaults. As a result we get different behaviour in relation to table encryption setting for newly generated and upgraded tables. To fix this inconsistency the mysql_system_tables_fix.sql was updated to always run mentioned ALTER with explicit ENCRYPTION settings and as a result setting explicit_encryption=1 for upgraded tables.
…h ps-admin) https://jira.percona.com/browse/PS-7917 Problem New tokudb deprecation error was logging a message before logging service for plugins was initialized. Fix Moved the deprecation error to after logging service initialization. Adjusted error message to display loose-X configuration. Otherwise, server will fail to start if tokudb_enabled or tokudb_backup_enabled are specified but plugin is not loaded. Adjusted error message to not mention that those variables require a server restart. They will be read at plugin initialization, thus not requiring a complete server restart. Added check at ps-admin to also display the deprecation error message.
- Removed all functionality related to TokuDB, TokuHotBackup, and Jemalloc. Left option handlers in place in order to print TokuDB removal and exit.
* Introducing the MTR_TERM environment variable, which defaults to xterm. This variable can be configured to basically any visual terminal, and setting it in .profile will result in a configured default setting for the user. * Intrudocing the MTR_LLDB environment variable, to deal with versioned lldb executable names in ubuntu.
…n.pl "--only-combinations" allows to restrict combinations to given names separated by "," e.g.: `mysql-test-run.pl --sute=main --only-combinations=innodb_intrinsic_table` `mysql-test-run.pl --sute=rocksdb --only-combinations=write_committed,write_prepared`
---------------------------------------------------------------------- PS-6849: Backport MTR shutdown report implementation from 5.7 mysql/mysql-server@8aeda36 Bug#30909369: MTR DOES NOT NOTICE A SERVER ABORT AFTER "BYE" Issue: ====== After a test is run, when the server is shutdown, MTR does not detect the exit status of the shutdown, and reports that the test passed, even if the server crashes. Fix: ==== Added a shutdown_report for every run of tests, which will report if the shutdown of the servers was successful or not. The exit status of shutdown is monitored and if a failure is found, the error log is parsed and the shutdown_report will contain the errors. This also fixes Bug#29818311, which reported the same problem. Change-Id: I4d8043915f9b085bfbd25e811c957cd0b4809454 (cherry picked from commit 8aeda36) ---------------------------------------------------------------------- PS-6849 Add JUnit reporting support to the MySQL test suite runner. Summary: WebScaleSQL Feature: JUnit Support for MTR Produces JUnit test reports that can be consumed by tools such as Jenkins. Two new options are added to MTR: --junit-output=FILE Output JUnit test summary XML to FILE. --junit-package=NAME Set the JUnit package name to NAME for this test run. A test run using junit reporting would look like: ./mtr --comment=rpl_row --junit-package=rpl_row --junit-output=rpl_row.xml \ --suite=rpl --mysqld=--binlog-format=row Typically, the package name should be the name for the test run, in this case the suite name and its variation. Tests results are written to the rpl_row.xml file (including test timing and output). If using with Jenkins, the XML file can be used for publishing test results and Jenkins would be able to show how many test failed, test duration and whether to mark the build as failed or unstable (if there were failing tests that succeeded on retry). Test Plan: mtr Reviewers: steaphan Originally Reviewed By: steaphan ---------------------------------------------------------------------- PS-6849 Add "junit" test suite with deliberately failing tests Adds number of tests that are failing in various ways to test how their failures are propagated to the JUnit report: 01_pass: always passes 02_disabled: passes, but is disabled by disabled.def 03_skipped: always skips itself 04_result_mismatch: result file contents differs 05_failed_assertion: asserts always-false condition 06_bootstrap_crash: crashes in the bootstrap phase 07_immediate_crash: crashes on command 08_shutdown_crash: crashes in the shutdown phase 09_boostrap_hang: waits indefinitely in the bootstrap phase 10_immediate_hang: waits indefinitely on command 11_shutdown_hang: waits indefinitely in the shutdown phase 12_shutdown_hang_in_master: master process of a master/slave pair waits indefinitely in the shutdown phase 13_shutdown_hang_in_slave: slave process of a master/slave pair waits indefinitely in the shutdown phase 14_shutdown_hang_in_both: both master and slave wait indefinitely in the shutdown phase 15_fail_every_second_time: fails every second time 16_bootstrap_buffer_overrun: writes behind a buffer during bootstrap 17_immediate_buffer_overrun: writes behind a buffer on command 18_shutdown_buffer_overrun: writes behind a buffer during shutdown 19_leak_memory: allocates but never frees a block of memory 20_leak_memory_in_master: master process leaks memory 21_leak_memory_in_slave: slave process leaks memory 22_leak_memory_in_both: both master and slave process leak memory ---------------------------------------------------------------------- PS-6849 Add a deliberately failing unittests suite Failing unit tests are included into JUnit-style report. To evaluate reporting capability, we need assorted predictable failures. This patch adds a test suite containing test cases that: - succeed; - fail on EXPECT_EQ(); - crash by calling abort(); - wait indefinitely; - write behind a memory buffer; - leak memory. Test suite is disabled by default. Enable with -DWITH_FAILING_GUNIT_TESTS. ---------------------------------------------------------------------- PS-6849 Attach mysqld shutdown and valgrind reports to JUnit report Collect all the output related to shutdown and valgrind reports and add it as a comment to shutdown_report and valgrind_report tests in case failures are found. This will also add corresponding output to JUnit reports. ---------------------------------------------------------------------- PS-6849 Run unit tests under Valgrind too if it was enabled If Valgrind was enabled for mysqld, CTest is now also asked to run its tests using Valgrind's Memcheck tool. ---------------------------------------------------------------------- PS-6849 Mark self-skipping tests as skipped in JUnit reports ---------------------------------------------------------------------- PS-6849 Convey tests output to JUnit reports regardless of test failure Passing unit tests may still have Valgrind warnings in them. That messages are lost unless CTest output is always included into JUnit reports. ---------------------------------------------------------------------- PS-6849 Exclude tests skipped by framework from results xml Same test may be run by framework a few times with different environment settings (like replication mode for example) and in case this environment doesn't match test requirements the test will be reported as skipped. Don't show such tests in results xml file in order not to overload results with false skipped tests. ---------------------------------------------------------------------- PS-7574: Improve communication between MTR test worker and main process https://jira.percona.com/browse/PS-7574 This is a follow up improvement for percona#3899. When MTR worker finishes its job it sends shutdown and valgrind reports to main process. It exits right after that. Two issues are possible at this point: - main process may lose last sent data from a worker; - from time to time main MTR ptocess may silently exit right after closing a worker. To solve both these issues communication protocol between main MTR process and workers is modified in this change. The following changes are done: - Worker shutdown process is split into two steps - collecting and sending shutdown/valgrind reports, actual worker process shutdown. - The GETREPORTS message gets sent to a worker to make it collect and send shutdown reports. Worker process stays active at this point. - Once main process receives all reports it sends BYE to a worker. Worker process exits after receiving this message. Above changes make sure main MTR process has time to collect all data from a worker before it exits.
…it report https://jira.percona.com/browse/PS-7993 Using underscore as a separator between test name and test combination name makes it impossible for any script used to parse results file to find out if there is a test combination in test name. Use dot as a separator in this case to improve this. Added junit_combinations test suite to be able to verify this behavior.
PS-8422: Merge MySQL 8.0.31 (unit_tests run with valgrind) https://jira.percona.com/browse/PS-8422 The DartConfiguration.tcl config is missing in buid directory. The BUILD_TESTING is required to be set for include(CTest) to generate it properly. ------------------------------------------------------------------- PS-8643 fix: Fix compilation WITH_NDB (xline for cmake) https://jira.percona.com/browse/PS-8643 'mgmclient' and 'mgmsrv' are now compiled / linked with proper system / bundled 'editline' or system readline headers / libraries depending on the values of the 'WITH_EDITLINE' / 'WITH_READLINE' CMake options. In order to avoid confusion, 'EDITLINE' / 'READLINE' CMake variables detected in the 'readline.cmake' renamed to 'MY_XLINE_INCLUDE_DIR' and 'MY_XLINE_LIBRARY'. All external references to the 'READLINE_INCLUDE_DIR' / 'EDITLINE_INCLUDE_DIR' changed to 'MY_XLINE_INCLUDE_DIR'. All external references to the 'READLINE_LIBRARY' / 'EDITLINE_LIBRARY' changed to 'MY_XLINE_LIBRARY'. Removed 'INCLUDE_DIRECTORIES()' directive from the bundled 'editline' branch of the 'readline.cmake' which was just polluting global compiler include directories list.
PS-5724: Travis-CI: 8.0.16 MySQL Router does not build cleanly with clang 4 & 5
Turn off warnings for MySQL Router for clang-4 and clang-5 using the `-Wno-missing-braces` compiler flag.
PS-6058: gcc 5.5.0 compilation warnings for router
Fixed:
1.
```
/data/mysql-server/mysql-8.0/router/src/http/src/tls_context.cc: In function ‘constexpr int o11x_version(TlsVersion)’:
/data/mysql-server/mysql-8.0/router/src/http/src/tls_context.cc:99:57: error: expression ‘<throw-expression>’ is not a constant-expression
throw std::invalid_argument("version out of range");
```
2.
```
/data/mysql-server/mysql-8.0/router/src/http/src/tls_server_context.cc: In constructor ‘TlsServerContext::TlsServerContext(TlsVersion, TlsVersion)’:
/data/mysql-server/mysql-8.0/router/src/http/src/tls_server_context.cc:67:43: error: statement has no effect [-Werror=unused-value]
SSL_CTX_set_ecdh_auto(ssl_ctx_.get(), 1);
```
The Problem: When proxy protocol is enabled, MySQL won't cleanup vio_pp_networks variable at shutdown. Solution: Cleanup allocated networks at shutdown. Enhanced proxy_protocol test to cover this scenario.
----------------------------------------------------------- PS-7949 Fix memry leak in gcs_xcom_control_interface-t test https://jira.percona.com/browse/PS-7949 ----------------------------------------------------------- PS-7949 Fix memory leaks in temptable::Allocator tests https://jira.percona.com/browse/PS-7949
PS-5741: Incorrect use of memset_s in keyring_vault.
Fixed the usage of memset_s. The arguments should be:
void memset_s(void *dest, size_t dest_max, int c, size_t n)
where the 2nd argument is size of buffer and the 3rd is
argument is character to fill.
---------------------------------------------------------------------------
PS-7769 - Fix use-after-return error in audit_log_exclude_accounts_validate
---
*Problem:*
`st_mysql_value::val_str` might return a pointer to `buf` which after
the function called is deleted. Therefore the value in `save`, after
reuturnin from the function, is invalid.
In this particular case, the error is not manifesting as val_str`
returns memory allocated with `thd_strmake` and it does not use `buf`.
*Solution:*
Allocate memory with `thd_strmake` so the memory in `save` is not local.
---------------------------------------------------------------------------
Fix test main.bug12969156 when WITH_ASAN=ON
*Problem:*
ASAN complains about stack-buffer-overflow on function `mysql_heartbeat`:
```
==90890==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fe746d06d14 at pc 0x7fe760f5b017 bp 0x7fe746d06cd0 sp 0x7fe746d06478
WRITE of size 24 at 0x7fe746d06d14 thread T16777215
Address 0x7fe746d06d14 is located in stack of thread T26 at offset 340 in frame
#0 0x7fe746d0a55c in mysql_heartbeat(void*) /home/yura/ws/percona-server/plugin/daemon_example/daemon_example.cc:62
This frame has 4 object(s):
[48, 56) 'result' (line 66)
[80, 112) '_db_stack_frame_' (line 63)
[144, 200) 'tm_tmp' (line 67)
[240, 340) 'buffer' (line 65) <== Memory access at offset 340 overflows this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
(longjmp and C++ exceptions *are* supported)
Thread T26 created by T25 here:
#0 0x7fe760f5f6d5 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:216
#1 0x557ccbbcb857 in my_thread_create /home/yura/ws/percona-server/mysys/my_thread.c:104
#2 0x7fe746d0b21a in daemon_example_plugin_init /home/yura/ws/percona-server/plugin/daemon_example/daemon_example.cc:148
#3 0x557ccb4c69c7 in plugin_initialize /home/yura/ws/percona-server/sql/sql_plugin.cc:1279
percona#4 0x557ccb4d19cd in mysql_install_plugin /home/yura/ws/percona-server/sql/sql_plugin.cc:2279
percona#5 0x557ccb4d218f in Sql_cmd_install_plugin::execute(THD*) /home/yura/ws/percona-server/sql/sql_plugin.cc:4664
percona#6 0x557ccb47695e in mysql_execute_command(THD*, bool) /home/yura/ws/percona-server/sql/sql_parse.cc:5160
percona#7 0x557ccb47977c in mysql_parse(THD*, Parser_state*, bool) /home/yura/ws/percona-server/sql/sql_parse.cc:5952
percona#8 0x557ccb47b6c2 in dispatch_command(THD*, COM_DATA const*, enum_server_command) /home/yura/ws/percona-server/sql/sql_parse.cc:1544
percona#9 0x557ccb47de1d in do_command(THD*) /home/yura/ws/percona-server/sql/sql_parse.cc:1065
percona#10 0x557ccb6ac294 in handle_connection /home/yura/ws/percona-server/sql/conn_handler/connection_handler_per_thread.cc:325
percona#11 0x557ccbbfabb0 in pfs_spawn_thread /home/yura/ws/percona-server/storage/perfschema/pfs.cc:2198
percona#12 0x7fe760ab544f in start_thread nptl/pthread_create.c:473
```
The reason is that `my_thread_cancel` is used to finish the daemon thread. This is not and orderly way of finishing the thread. ASAN does not register the stack variables are not used anymore which generates the error above.
This is a benign error as all the variables are on the stack.
*Solution*:
Finish the thread in orderly way by using a signalling variable.
---------------------------------------------------------------------------
PS-8204: Fix XML escape rules for audit plugin
https://jira.percona.com/browse/PS-8204
There was a wrong length specified for some XML
escape rules. As a result of this terminating null symbol from
replacement rule was copied into resulting string. This lead to
quer text truncation in audit log file.
In addition added empty replacement rules for '\b' and 'f' symbols
which just remove them from resulting string. These symboles are
not supported in XML 1.0.
…nd_class=error" for server-side prepared statements https://jira.percona.com/browse/PS-1116 Analysis -------- server-side prepared statements sets thd->lex->sql_command to SQLCOM_END , which results in COMMAND_CLASS to set as 'error' Fix --- Inside function mysql_audit_notify we have set COMMAND_CLASS to correct type based on msg type This chnage is specifically made for audit logs only
Reverted Oracle's fix for the Bug #25873310 "FULLTEXT SEARCH CAN NOT FIND WORD WHICH CONTAINS ',' OR '.'" (commit mysql/mysql-server@c1a5784) as it does not allow special characters to be indexed by NGRAM FTS parser any more which makes Percona-specific variable 'ft_query_extra_word_chars' introduced in the fix for PS-2501 "LP #1689268: Fulltext search can not find word which contains punctuation marks" (https://jira.percona.com/browse/PS-2501) (commit percona@b7cb587) completely useless. Changes in `innodb_fts/include/ngram.inc` were also reverted. `innodb_fts/t/ngram_1.test` was not reverted. `innodb_fts/r/ngram_1.result` was re-recorded as `ngram.inc` was changed.
…h special character https://jira.percona.com/browse/PS-7958 Server crashes during fulltext search in case there is a null character symbol in the middle of a search string. The issue reproducible for NGRAM and MECAB parsers. Updated ngram_parse() and mecab_parse() method to skip control characters while parsing a string. PS-8422: Merge MySQL 8.0.31 (fix innodb_fts.percona_mecab_null_character) After changes by upstream in mysql/mysql-server@f4445048160 we have to change `mysql_charset` to `utf8mb4` for `innodb_fts.percona_mecab_null_character`.
…ould be created for PAM plugin) Although PAM authentication is impossible to test in MTR without system cooperation, it is possible to do minimal plugin testing without testing authentication itself: that INSTALL/UNINSTALL PLUGIN works and that it is possible to CREATE USER with this authentication method. Add such tests to the plugin-specific test directory, add this suite to the default suite list, and install the test files.
… MyRocks in Group Replication This patch adds a compile time option `GROUP_REPLICATION_WITH_ROCKSDB` for the Group Replication plugin that enables MyRocks storage engine. Usage: `cmake -DGROUP_REPLICATION_WITH_ROCKSDB=ON`
…cy to start throttling the commits only when a majority of the nodes are above threshold. https://jira.percona.com/browse/PS-8276 This commit adds a new flow control mode MAJORITY, that makes the cluster to throttle commits only when the majority of the nodes are above the flow control threshold. The defaults have not been changed, so it doesn't change the existing behavior.
…ts using 'grep' https://jira.percona.com/browse/PS-8844 In mysql/mysql-server@75a1d41b2802 there was added error 42 as expected to some tests using 'grep'. This is already handled via ASAN suppressions. Corresponding rule was added in percona@fe198af. Removed error 42 from expected in this change.
…me after freed
`~Buffered_error_logger()` was called after `buffered_error_log_filename` was freed within `sys_var_end()`.
Valgrind found the issue:
```
[ 62%] main.all_persisted_variables w19 [ fail ] Found warnings/errors in error log file!
Test ended at 2023-08-09 18:56:13
==734180== Invalid read of size 1
==734180== at 0x7AAF9CC: Buffered_error_logger::write_to_disk_() (buffered_error_log.cc:54)
==734180== by 0x7AAF8A8: Buffered_error_logger::write_to_disk() (buffered_error_log.cc:44)
==734180== by 0x7AAF6D9: Buffered_error_logger::~Buffered_error_logger() (buffered_error_log.cc:29)
==734180== by 0xAF38494: __run_exit_handlers (exit.c:113)
==734180== by 0xAF3860F: exit (exit.c:143)
==734180== by 0x634DD9C: mysqld_exit(int) (mysqld.cc:2627)
==734180== by 0x6360A45: mysqld_main(int, char**) (mysqld.cc:8771)
==734180== by 0x5E8827C: main (main.cc:25)
```
Add a test to check FNV1A_64, FNV_64, and MURMUR_HASH user-defined functions.
https://jira.percona.com/browse/PS-8844 This patch fixes the test failure of main.mysqldump_gtid_purged that failed due to the uninitialized variable $redirect_stderr in the start_proc_in_background.inc.
…ded port in the result file https://jira.percona.com/browse/PS-8886 The test case federated.federated_double_type failed when run with a different MTR build thread id because of the hardcoded port number in the result file. This has been fixed by replacing the port number with a text with the use of MTR's replace_result command.
Cirrus CI community arm workers are always busy so we need to use our own AWS jobs.
https://jira.percona.com/browse/PS-8844 The test clone.plugin_mismatch fails when run from installed directories due to the different path of the plugin directory used by the test. Failure: CURRENT_TEST: clone.plugin_mismatch mysqltest: At line 28: Command "remove_file" failed with error 1. my_errno=2. The test involves simulating the secnario of missing plugins on recipient server. To do the same, the test earlier used to copy all plugins to recipient server's plugin_dir and delete some plugins to test the clone plugin's behavior when recipeint had some plugins missing. During the test investigation, it was found that it is just sufficient to have only the clone plugin copied to the recipient server so that it simulates the missing plugins. This patch makes the the test independent of plugin paths so it works on both build and installed directories.
…am_hypergraph, index_merge_innodb, index_merge_rocksdb2
The issue was:
```
[ 95%] Linking CXX shared module ../../plugin_output_directory/ha_rocksdb.so
Undefined symbols for architecture x86_64:
"_my_charset_utf16_bin", referenced from:
myrocks::get_segment_size_from_collation(CHARSET_INFO const*) in rdb_datadic.cc.o
"_my_charset_utf16le_bin", referenced from:
myrocks::get_segment_size_from_collation(CHARSET_INFO const*) in rdb_datadic.cc.o
"_my_charset_utf32_bin", referenced from:
myrocks::get_segment_size_from_collation(CHARSET_INFO const*) in rdb_datadic.cc.o
"_my_collation_8bit_bin_handler", referenced from:
myrocks::rdb_is_binary_collation(CHARSET_INFO const*) in rdb_datadic.cc.o
"_my_collation_8bit_simple_ci_handler", referenced from:
myrocks::rdb_is_simple_collation(CHARSET_INFO const*) in rdb_datadic.cc.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [plugin_output_directory/ha_rocksdb.so] Error 1
make[1]: *** [storage/rocksdb/CMakeFiles/rocksdb.dir/all] Error 2
```
oleksandr-kachan
force-pushed
the
PS-8865-keyring_vault_to_comp
branch
from
955e064
to
0528dd4
Compare
inikep
added a commit
that referenced
this pull request
Oct 16, 2023
PS-5741: Incorrect use of memset_s in keyring_vault.
Fixed the usage of memset_s. The arguments should be:
void memset_s(void *dest, size_t dest_max, int c, size_t n)
where the 2nd argument is size of buffer and the 3rd is
argument is character to fill.
---------------------------------------------------------------------------
PS-7769 - Fix use-after-return error in audit_log_exclude_accounts_validate
---
*Problem:*
`st_mysql_value::val_str` might return a pointer to `buf` which after
the function called is deleted. Therefore the value in `save`, after
reuturnin from the function, is invalid.
In this particular case, the error is not manifesting as val_str`
returns memory allocated with `thd_strmake` and it does not use `buf`.
*Solution:*
Allocate memory with `thd_strmake` so the memory in `save` is not local.
---------------------------------------------------------------------------
Fix test main.bug12969156 when WITH_ASAN=ON
*Problem:*
ASAN complains about stack-buffer-overflow on function `mysql_heartbeat`:
```
==90890==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fe746d06d14 at pc 0x7fe760f5b017 bp 0x7fe746d06cd0 sp 0x7fe746d06478
WRITE of size 24 at 0x7fe746d06d14 thread T16777215
Address 0x7fe746d06d14 is located in stack of thread T26 at offset 340 in frame
#0 0x7fe746d0a55c in mysql_heartbeat(void*) /home/yura/ws/percona-server/plugin/daemon_example/daemon_example.cc:62
This frame has 4 object(s):
[48, 56) 'result' (line 66)
[80, 112) '_db_stack_frame_' (line 63)
[144, 200) 'tm_tmp' (line 67)
[240, 340) 'buffer' (line 65) <== Memory access at offset 340 overflows this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
(longjmp and C++ exceptions *are* supported)
Thread T26 created by T25 here:
#0 0x7fe760f5f6d5 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:216
#1 0x557ccbbcb857 in my_thread_create /home/yura/ws/percona-server/mysys/my_thread.c:104
#2 0x7fe746d0b21a in daemon_example_plugin_init /home/yura/ws/percona-server/plugin/daemon_example/daemon_example.cc:148
#3 0x557ccb4c69c7 in plugin_initialize /home/yura/ws/percona-server/sql/sql_plugin.cc:1279
#4 0x557ccb4d19cd in mysql_install_plugin /home/yura/ws/percona-server/sql/sql_plugin.cc:2279
#5 0x557ccb4d218f in Sql_cmd_install_plugin::execute(THD*) /home/yura/ws/percona-server/sql/sql_plugin.cc:4664
#6 0x557ccb47695e in mysql_execute_command(THD*, bool) /home/yura/ws/percona-server/sql/sql_parse.cc:5160
#7 0x557ccb47977c in mysql_parse(THD*, Parser_state*, bool) /home/yura/ws/percona-server/sql/sql_parse.cc:5952
#8 0x557ccb47b6c2 in dispatch_command(THD*, COM_DATA const*, enum_server_command) /home/yura/ws/percona-server/sql/sql_parse.cc:1544
#9 0x557ccb47de1d in do_command(THD*) /home/yura/ws/percona-server/sql/sql_parse.cc:1065
#10 0x557ccb6ac294 in handle_connection /home/yura/ws/percona-server/sql/conn_handler/connection_handler_per_thread.cc:325
#11 0x557ccbbfabb0 in pfs_spawn_thread /home/yura/ws/percona-server/storage/perfschema/pfs.cc:2198
#12 0x7fe760ab544f in start_thread nptl/pthread_create.c:473
```
The reason is that `my_thread_cancel` is used to finish the daemon thread. This is not and orderly way of finishing the thread. ASAN does not register the stack variables are not used anymore which generates the error above.
This is a benign error as all the variables are on the stack.
*Solution*:
Finish the thread in orderly way by using a signalling variable.
---------------------------------------------------------------------------
PS-8204: Fix XML escape rules for audit plugin
https://jira.percona.com/browse/PS-8204
There was a wrong length specified for some XML
escape rules. As a result of this terminating null symbol from
replacement rule was copied into resulting string. This lead to
quer text truncation in audit log file.
In addition added empty replacement rules for '\b' and 'f' symbols
which just remove them from resulting string. These symboles are
not supported in XML 1.0.
---------------------------------------------------------------------------
PS-8854: Add main.percona_udf MTR test
Add a test to check FNV1A_64, FNV_64, and MURMUR_HASH user-defined functions.
inikep
added a commit
that referenced
this pull request
Oct 16, 2023
PS-5741: Incorrect use of memset_s in keyring_vault.
Fixed the usage of memset_s. The arguments should be:
void memset_s(void *dest, size_t dest_max, int c, size_t n)
where the 2nd argument is size of buffer and the 3rd is
argument is character to fill.
---------------------------------------------------------------------------
PS-7769 - Fix use-after-return error in audit_log_exclude_accounts_validate
---
*Problem:*
`st_mysql_value::val_str` might return a pointer to `buf` which after
the function called is deleted. Therefore the value in `save`, after
reuturnin from the function, is invalid.
In this particular case, the error is not manifesting as val_str`
returns memory allocated with `thd_strmake` and it does not use `buf`.
*Solution:*
Allocate memory with `thd_strmake` so the memory in `save` is not local.
---------------------------------------------------------------------------
Fix test main.bug12969156 when WITH_ASAN=ON
*Problem:*
ASAN complains about stack-buffer-overflow on function `mysql_heartbeat`:
```
==90890==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fe746d06d14 at pc 0x7fe760f5b017 bp 0x7fe746d06cd0 sp 0x7fe746d06478
WRITE of size 24 at 0x7fe746d06d14 thread T16777215
Address 0x7fe746d06d14 is located in stack of thread T26 at offset 340 in frame
#0 0x7fe746d0a55c in mysql_heartbeat(void*) /home/yura/ws/percona-server/plugin/daemon_example/daemon_example.cc:62
This frame has 4 object(s):
[48, 56) 'result' (line 66)
[80, 112) '_db_stack_frame_' (line 63)
[144, 200) 'tm_tmp' (line 67)
[240, 340) 'buffer' (line 65) <== Memory access at offset 340 overflows this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
(longjmp and C++ exceptions *are* supported)
Thread T26 created by T25 here:
#0 0x7fe760f5f6d5 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:216
#1 0x557ccbbcb857 in my_thread_create /home/yura/ws/percona-server/mysys/my_thread.c:104
#2 0x7fe746d0b21a in daemon_example_plugin_init /home/yura/ws/percona-server/plugin/daemon_example/daemon_example.cc:148
#3 0x557ccb4c69c7 in plugin_initialize /home/yura/ws/percona-server/sql/sql_plugin.cc:1279
#4 0x557ccb4d19cd in mysql_install_plugin /home/yura/ws/percona-server/sql/sql_plugin.cc:2279
#5 0x557ccb4d218f in Sql_cmd_install_plugin::execute(THD*) /home/yura/ws/percona-server/sql/sql_plugin.cc:4664
#6 0x557ccb47695e in mysql_execute_command(THD*, bool) /home/yura/ws/percona-server/sql/sql_parse.cc:5160
#7 0x557ccb47977c in mysql_parse(THD*, Parser_state*, bool) /home/yura/ws/percona-server/sql/sql_parse.cc:5952
#8 0x557ccb47b6c2 in dispatch_command(THD*, COM_DATA const*, enum_server_command) /home/yura/ws/percona-server/sql/sql_parse.cc:1544
#9 0x557ccb47de1d in do_command(THD*) /home/yura/ws/percona-server/sql/sql_parse.cc:1065
#10 0x557ccb6ac294 in handle_connection /home/yura/ws/percona-server/sql/conn_handler/connection_handler_per_thread.cc:325
#11 0x557ccbbfabb0 in pfs_spawn_thread /home/yura/ws/percona-server/storage/perfschema/pfs.cc:2198
#12 0x7fe760ab544f in start_thread nptl/pthread_create.c:473
```
The reason is that `my_thread_cancel` is used to finish the daemon thread. This is not and orderly way of finishing the thread. ASAN does not register the stack variables are not used anymore which generates the error above.
This is a benign error as all the variables are on the stack.
*Solution*:
Finish the thread in orderly way by using a signalling variable.
---------------------------------------------------------------------------
PS-8204: Fix XML escape rules for audit plugin
https://jira.percona.com/browse/PS-8204
There was a wrong length specified for some XML
escape rules. As a result of this terminating null symbol from
replacement rule was copied into resulting string. This lead to
quer text truncation in audit log file.
In addition added empty replacement rules for '\b' and 'f' symbols
which just remove them from resulting string. These symboles are
not supported in XML 1.0.
---------------------------------------------------------------------------
PS-8854: Add main.percona_udf MTR test
Add a test to check FNV1A_64, FNV_64, and MURMUR_HASH user-defined functions.
inikep
force-pushed
the
internal-8.1.0-linear
branch
from
4d76cf8
to
0755d44
Compare
inikep
added a commit
that referenced
this pull request
Oct 25, 2023
PS-5741: Incorrect use of memset_s in keyring_vault.
Fixed the usage of memset_s. The arguments should be:
void memset_s(void *dest, size_t dest_max, int c, size_t n)
where the 2nd argument is size of buffer and the 3rd is
argument is character to fill.
---------------------------------------------------------------------------
PS-7769 - Fix use-after-return error in audit_log_exclude_accounts_validate
---
*Problem:*
`st_mysql_value::val_str` might return a pointer to `buf` which after
the function called is deleted. Therefore the value in `save`, after
reuturnin from the function, is invalid.
In this particular case, the error is not manifesting as val_str`
returns memory allocated with `thd_strmake` and it does not use `buf`.
*Solution:*
Allocate memory with `thd_strmake` so the memory in `save` is not local.
---------------------------------------------------------------------------
Fix test main.bug12969156 when WITH_ASAN=ON
*Problem:*
ASAN complains about stack-buffer-overflow on function `mysql_heartbeat`:
```
==90890==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fe746d06d14 at pc 0x7fe760f5b017 bp 0x7fe746d06cd0 sp 0x7fe746d06478
WRITE of size 24 at 0x7fe746d06d14 thread T16777215
Address 0x7fe746d06d14 is located in stack of thread T26 at offset 340 in frame
#0 0x7fe746d0a55c in mysql_heartbeat(void*) /home/yura/ws/percona-server/plugin/daemon_example/daemon_example.cc:62
This frame has 4 object(s):
[48, 56) 'result' (line 66)
[80, 112) '_db_stack_frame_' (line 63)
[144, 200) 'tm_tmp' (line 67)
[240, 340) 'buffer' (line 65) <== Memory access at offset 340 overflows this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
(longjmp and C++ exceptions *are* supported)
Thread T26 created by T25 here:
#0 0x7fe760f5f6d5 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:216
#1 0x557ccbbcb857 in my_thread_create /home/yura/ws/percona-server/mysys/my_thread.c:104
#2 0x7fe746d0b21a in daemon_example_plugin_init /home/yura/ws/percona-server/plugin/daemon_example/daemon_example.cc:148
#3 0x557ccb4c69c7 in plugin_initialize /home/yura/ws/percona-server/sql/sql_plugin.cc:1279
#4 0x557ccb4d19cd in mysql_install_plugin /home/yura/ws/percona-server/sql/sql_plugin.cc:2279
#5 0x557ccb4d218f in Sql_cmd_install_plugin::execute(THD*) /home/yura/ws/percona-server/sql/sql_plugin.cc:4664
#6 0x557ccb47695e in mysql_execute_command(THD*, bool) /home/yura/ws/percona-server/sql/sql_parse.cc:5160
#7 0x557ccb47977c in mysql_parse(THD*, Parser_state*, bool) /home/yura/ws/percona-server/sql/sql_parse.cc:5952
#8 0x557ccb47b6c2 in dispatch_command(THD*, COM_DATA const*, enum_server_command) /home/yura/ws/percona-server/sql/sql_parse.cc:1544
#9 0x557ccb47de1d in do_command(THD*) /home/yura/ws/percona-server/sql/sql_parse.cc:1065
#10 0x557ccb6ac294 in handle_connection /home/yura/ws/percona-server/sql/conn_handler/connection_handler_per_thread.cc:325
#11 0x557ccbbfabb0 in pfs_spawn_thread /home/yura/ws/percona-server/storage/perfschema/pfs.cc:2198
#12 0x7fe760ab544f in start_thread nptl/pthread_create.c:473
```
The reason is that `my_thread_cancel` is used to finish the daemon thread. This is not and orderly way of finishing the thread. ASAN does not register the stack variables are not used anymore which generates the error above.
This is a benign error as all the variables are on the stack.
*Solution*:
Finish the thread in orderly way by using a signalling variable.
---------------------------------------------------------------------------
PS-8204: Fix XML escape rules for audit plugin
https://jira.percona.com/browse/PS-8204
There was a wrong length specified for some XML
escape rules. As a result of this terminating null symbol from
replacement rule was copied into resulting string. This lead to
quer text truncation in audit log file.
In addition added empty replacement rules for '\b' and 'f' symbols
which just remove them from resulting string. These symboles are
not supported in XML 1.0.
---------------------------------------------------------------------------
PS-8854: Add main.percona_udf MTR test
Add a test to check FNV1A_64, FNV_64, and MURMUR_HASH user-defined functions.
inikep
added a commit
that referenced
this pull request
Oct 26, 2023
PS-5741: Incorrect use of memset_s in keyring_vault.
Fixed the usage of memset_s. The arguments should be:
void memset_s(void *dest, size_t dest_max, int c, size_t n)
where the 2nd argument is size of buffer and the 3rd is
argument is character to fill.
---------------------------------------------------------------------------
PS-7769 - Fix use-after-return error in audit_log_exclude_accounts_validate
---
*Problem:*
`st_mysql_value::val_str` might return a pointer to `buf` which after
the function called is deleted. Therefore the value in `save`, after
reuturnin from the function, is invalid.
In this particular case, the error is not manifesting as val_str`
returns memory allocated with `thd_strmake` and it does not use `buf`.
*Solution:*
Allocate memory with `thd_strmake` so the memory in `save` is not local.
---------------------------------------------------------------------------
Fix test main.bug12969156 when WITH_ASAN=ON
*Problem:*
ASAN complains about stack-buffer-overflow on function `mysql_heartbeat`:
```
==90890==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fe746d06d14 at pc 0x7fe760f5b017 bp 0x7fe746d06cd0 sp 0x7fe746d06478
WRITE of size 24 at 0x7fe746d06d14 thread T16777215
Address 0x7fe746d06d14 is located in stack of thread T26 at offset 340 in frame
#0 0x7fe746d0a55c in mysql_heartbeat(void*) /home/yura/ws/percona-server/plugin/daemon_example/daemon_example.cc:62
This frame has 4 object(s):
[48, 56) 'result' (line 66)
[80, 112) '_db_stack_frame_' (line 63)
[144, 200) 'tm_tmp' (line 67)
[240, 340) 'buffer' (line 65) <== Memory access at offset 340 overflows this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
(longjmp and C++ exceptions *are* supported)
Thread T26 created by T25 here:
#0 0x7fe760f5f6d5 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:216
#1 0x557ccbbcb857 in my_thread_create /home/yura/ws/percona-server/mysys/my_thread.c:104
#2 0x7fe746d0b21a in daemon_example_plugin_init /home/yura/ws/percona-server/plugin/daemon_example/daemon_example.cc:148
#3 0x557ccb4c69c7 in plugin_initialize /home/yura/ws/percona-server/sql/sql_plugin.cc:1279
#4 0x557ccb4d19cd in mysql_install_plugin /home/yura/ws/percona-server/sql/sql_plugin.cc:2279
#5 0x557ccb4d218f in Sql_cmd_install_plugin::execute(THD*) /home/yura/ws/percona-server/sql/sql_plugin.cc:4664
#6 0x557ccb47695e in mysql_execute_command(THD*, bool) /home/yura/ws/percona-server/sql/sql_parse.cc:5160
#7 0x557ccb47977c in mysql_parse(THD*, Parser_state*, bool) /home/yura/ws/percona-server/sql/sql_parse.cc:5952
#8 0x557ccb47b6c2 in dispatch_command(THD*, COM_DATA const*, enum_server_command) /home/yura/ws/percona-server/sql/sql_parse.cc:1544
#9 0x557ccb47de1d in do_command(THD*) /home/yura/ws/percona-server/sql/sql_parse.cc:1065
#10 0x557ccb6ac294 in handle_connection /home/yura/ws/percona-server/sql/conn_handler/connection_handler_per_thread.cc:325
#11 0x557ccbbfabb0 in pfs_spawn_thread /home/yura/ws/percona-server/storage/perfschema/pfs.cc:2198
#12 0x7fe760ab544f in start_thread nptl/pthread_create.c:473
```
The reason is that `my_thread_cancel` is used to finish the daemon thread. This is not and orderly way of finishing the thread. ASAN does not register the stack variables are not used anymore which generates the error above.
This is a benign error as all the variables are on the stack.
*Solution*:
Finish the thread in orderly way by using a signalling variable.
---------------------------------------------------------------------------
PS-8204: Fix XML escape rules for audit plugin
https://jira.percona.com/browse/PS-8204
There was a wrong length specified for some XML
escape rules. As a result of this terminating null symbol from
replacement rule was copied into resulting string. This lead to
quer text truncation in audit log file.
In addition added empty replacement rules for '\b' and 'f' symbols
which just remove them from resulting string. These symboles are
not supported in XML 1.0.
---------------------------------------------------------------------------
PS-8854: Add main.percona_udf MTR test
Add a test to check FNV1A_64, FNV_64, and MURMUR_HASH user-defined functions.
inikep
added a commit
that referenced
this pull request
Oct 26, 2023
PS-5741: Incorrect use of memset_s in keyring_vault.
Fixed the usage of memset_s. The arguments should be:
void memset_s(void *dest, size_t dest_max, int c, size_t n)
where the 2nd argument is size of buffer and the 3rd is
argument is character to fill.
---------------------------------------------------------------------------
PS-7769 - Fix use-after-return error in audit_log_exclude_accounts_validate
---
*Problem:*
`st_mysql_value::val_str` might return a pointer to `buf` which after
the function called is deleted. Therefore the value in `save`, after
reuturnin from the function, is invalid.
In this particular case, the error is not manifesting as val_str`
returns memory allocated with `thd_strmake` and it does not use `buf`.
*Solution:*
Allocate memory with `thd_strmake` so the memory in `save` is not local.
---------------------------------------------------------------------------
Fix test main.bug12969156 when WITH_ASAN=ON
*Problem:*
ASAN complains about stack-buffer-overflow on function `mysql_heartbeat`:
```
==90890==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fe746d06d14 at pc 0x7fe760f5b017 bp 0x7fe746d06cd0 sp 0x7fe746d06478
WRITE of size 24 at 0x7fe746d06d14 thread T16777215
Address 0x7fe746d06d14 is located in stack of thread T26 at offset 340 in frame
#0 0x7fe746d0a55c in mysql_heartbeat(void*) /home/yura/ws/percona-server/plugin/daemon_example/daemon_example.cc:62
This frame has 4 object(s):
[48, 56) 'result' (line 66)
[80, 112) '_db_stack_frame_' (line 63)
[144, 200) 'tm_tmp' (line 67)
[240, 340) 'buffer' (line 65) <== Memory access at offset 340 overflows this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
(longjmp and C++ exceptions *are* supported)
Thread T26 created by T25 here:
#0 0x7fe760f5f6d5 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:216
#1 0x557ccbbcb857 in my_thread_create /home/yura/ws/percona-server/mysys/my_thread.c:104
#2 0x7fe746d0b21a in daemon_example_plugin_init /home/yura/ws/percona-server/plugin/daemon_example/daemon_example.cc:148
#3 0x557ccb4c69c7 in plugin_initialize /home/yura/ws/percona-server/sql/sql_plugin.cc:1279
#4 0x557ccb4d19cd in mysql_install_plugin /home/yura/ws/percona-server/sql/sql_plugin.cc:2279
#5 0x557ccb4d218f in Sql_cmd_install_plugin::execute(THD*) /home/yura/ws/percona-server/sql/sql_plugin.cc:4664
#6 0x557ccb47695e in mysql_execute_command(THD*, bool) /home/yura/ws/percona-server/sql/sql_parse.cc:5160
#7 0x557ccb47977c in mysql_parse(THD*, Parser_state*, bool) /home/yura/ws/percona-server/sql/sql_parse.cc:5952
#8 0x557ccb47b6c2 in dispatch_command(THD*, COM_DATA const*, enum_server_command) /home/yura/ws/percona-server/sql/sql_parse.cc:1544
#9 0x557ccb47de1d in do_command(THD*) /home/yura/ws/percona-server/sql/sql_parse.cc:1065
#10 0x557ccb6ac294 in handle_connection /home/yura/ws/percona-server/sql/conn_handler/connection_handler_per_thread.cc:325
#11 0x557ccbbfabb0 in pfs_spawn_thread /home/yura/ws/percona-server/storage/perfschema/pfs.cc:2198
#12 0x7fe760ab544f in start_thread nptl/pthread_create.c:473
```
The reason is that `my_thread_cancel` is used to finish the daemon thread. This is not and orderly way of finishing the thread. ASAN does not register the stack variables are not used anymore which generates the error above.
This is a benign error as all the variables are on the stack.
*Solution*:
Finish the thread in orderly way by using a signalling variable.
---------------------------------------------------------------------------
PS-8204: Fix XML escape rules for audit plugin
https://jira.percona.com/browse/PS-8204
There was a wrong length specified for some XML
escape rules. As a result of this terminating null symbol from
replacement rule was copied into resulting string. This lead to
quer text truncation in audit log file.
In addition added empty replacement rules for '\b' and 'f' symbols
which just remove them from resulting string. These symboles are
not supported in XML 1.0.
---------------------------------------------------------------------------
PS-8854: Add main.percona_udf MTR test
Add a test to check FNV1A_64, FNV_64, and MURMUR_HASH user-defined functions.
inikep
added a commit
that referenced
this pull request
Jan 17, 2024
PS-5741: Incorrect use of memset_s in keyring_vault.
Fixed the usage of memset_s. The arguments should be:
void memset_s(void *dest, size_t dest_max, int c, size_t n)
where the 2nd argument is size of buffer and the 3rd is
argument is character to fill.
---------------------------------------------------------------------------
PS-7769 - Fix use-after-return error in audit_log_exclude_accounts_validate
---
*Problem:*
`st_mysql_value::val_str` might return a pointer to `buf` which after
the function called is deleted. Therefore the value in `save`, after
reuturnin from the function, is invalid.
In this particular case, the error is not manifesting as val_str`
returns memory allocated with `thd_strmake` and it does not use `buf`.
*Solution:*
Allocate memory with `thd_strmake` so the memory in `save` is not local.
---------------------------------------------------------------------------
Fix test main.bug12969156 when WITH_ASAN=ON
*Problem:*
ASAN complains about stack-buffer-overflow on function `mysql_heartbeat`:
```
==90890==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fe746d06d14 at pc 0x7fe760f5b017 bp 0x7fe746d06cd0 sp 0x7fe746d06478
WRITE of size 24 at 0x7fe746d06d14 thread T16777215
Address 0x7fe746d06d14 is located in stack of thread T26 at offset 340 in frame
#0 0x7fe746d0a55c in mysql_heartbeat(void*) /home/yura/ws/percona-server/plugin/daemon_example/daemon_example.cc:62
This frame has 4 object(s):
[48, 56) 'result' (line 66)
[80, 112) '_db_stack_frame_' (line 63)
[144, 200) 'tm_tmp' (line 67)
[240, 340) 'buffer' (line 65) <== Memory access at offset 340 overflows this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
(longjmp and C++ exceptions *are* supported)
Thread T26 created by T25 here:
#0 0x7fe760f5f6d5 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:216
#1 0x557ccbbcb857 in my_thread_create /home/yura/ws/percona-server/mysys/my_thread.c:104
#2 0x7fe746d0b21a in daemon_example_plugin_init /home/yura/ws/percona-server/plugin/daemon_example/daemon_example.cc:148
#3 0x557ccb4c69c7 in plugin_initialize /home/yura/ws/percona-server/sql/sql_plugin.cc:1279
#4 0x557ccb4d19cd in mysql_install_plugin /home/yura/ws/percona-server/sql/sql_plugin.cc:2279
#5 0x557ccb4d218f in Sql_cmd_install_plugin::execute(THD*) /home/yura/ws/percona-server/sql/sql_plugin.cc:4664
#6 0x557ccb47695e in mysql_execute_command(THD*, bool) /home/yura/ws/percona-server/sql/sql_parse.cc:5160
#7 0x557ccb47977c in mysql_parse(THD*, Parser_state*, bool) /home/yura/ws/percona-server/sql/sql_parse.cc:5952
#8 0x557ccb47b6c2 in dispatch_command(THD*, COM_DATA const*, enum_server_command) /home/yura/ws/percona-server/sql/sql_parse.cc:1544
#9 0x557ccb47de1d in do_command(THD*) /home/yura/ws/percona-server/sql/sql_parse.cc:1065
#10 0x557ccb6ac294 in handle_connection /home/yura/ws/percona-server/sql/conn_handler/connection_handler_per_thread.cc:325
#11 0x557ccbbfabb0 in pfs_spawn_thread /home/yura/ws/percona-server/storage/perfschema/pfs.cc:2198
#12 0x7fe760ab544f in start_thread nptl/pthread_create.c:473
```
The reason is that `my_thread_cancel` is used to finish the daemon thread. This is not and orderly way of finishing the thread. ASAN does not register the stack variables are not used anymore which generates the error above.
This is a benign error as all the variables are on the stack.
*Solution*:
Finish the thread in orderly way by using a signalling variable.
---------------------------------------------------------------------------
PS-8204: Fix XML escape rules for audit plugin
https://jira.percona.com/browse/PS-8204
There was a wrong length specified for some XML
escape rules. As a result of this terminating null symbol from
replacement rule was copied into resulting string. This lead to
quer text truncation in audit log file.
In addition added empty replacement rules for '\b' and 'f' symbols
which just remove them from resulting string. These symboles are
not supported in XML 1.0.
---------------------------------------------------------------------------
PS-8854: Add main.percona_udf MTR test
Add a test to check FNV1A_64, FNV_64, and MURMUR_HASH user-defined functions.
inikep
added a commit
that referenced
this pull request
Jan 17, 2024
PS-5741: Incorrect use of memset_s in keyring_vault.
Fixed the usage of memset_s. The arguments should be:
void memset_s(void *dest, size_t dest_max, int c, size_t n)
where the 2nd argument is size of buffer and the 3rd is
argument is character to fill.
---------------------------------------------------------------------------
PS-7769 - Fix use-after-return error in audit_log_exclude_accounts_validate
---
*Problem:*
`st_mysql_value::val_str` might return a pointer to `buf` which after
the function called is deleted. Therefore the value in `save`, after
reuturnin from the function, is invalid.
In this particular case, the error is not manifesting as val_str`
returns memory allocated with `thd_strmake` and it does not use `buf`.
*Solution:*
Allocate memory with `thd_strmake` so the memory in `save` is not local.
---------------------------------------------------------------------------
Fix test main.bug12969156 when WITH_ASAN=ON
*Problem:*
ASAN complains about stack-buffer-overflow on function `mysql_heartbeat`:
```
==90890==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fe746d06d14 at pc 0x7fe760f5b017 bp 0x7fe746d06cd0 sp 0x7fe746d06478
WRITE of size 24 at 0x7fe746d06d14 thread T16777215
Address 0x7fe746d06d14 is located in stack of thread T26 at offset 340 in frame
#0 0x7fe746d0a55c in mysql_heartbeat(void*) /home/yura/ws/percona-server/plugin/daemon_example/daemon_example.cc:62
This frame has 4 object(s):
[48, 56) 'result' (line 66)
[80, 112) '_db_stack_frame_' (line 63)
[144, 200) 'tm_tmp' (line 67)
[240, 340) 'buffer' (line 65) <== Memory access at offset 340 overflows this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
(longjmp and C++ exceptions *are* supported)
Thread T26 created by T25 here:
#0 0x7fe760f5f6d5 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:216
#1 0x557ccbbcb857 in my_thread_create /home/yura/ws/percona-server/mysys/my_thread.c:104
#2 0x7fe746d0b21a in daemon_example_plugin_init /home/yura/ws/percona-server/plugin/daemon_example/daemon_example.cc:148
#3 0x557ccb4c69c7 in plugin_initialize /home/yura/ws/percona-server/sql/sql_plugin.cc:1279
#4 0x557ccb4d19cd in mysql_install_plugin /home/yura/ws/percona-server/sql/sql_plugin.cc:2279
#5 0x557ccb4d218f in Sql_cmd_install_plugin::execute(THD*) /home/yura/ws/percona-server/sql/sql_plugin.cc:4664
#6 0x557ccb47695e in mysql_execute_command(THD*, bool) /home/yura/ws/percona-server/sql/sql_parse.cc:5160
#7 0x557ccb47977c in mysql_parse(THD*, Parser_state*, bool) /home/yura/ws/percona-server/sql/sql_parse.cc:5952
#8 0x557ccb47b6c2 in dispatch_command(THD*, COM_DATA const*, enum_server_command) /home/yura/ws/percona-server/sql/sql_parse.cc:1544
#9 0x557ccb47de1d in do_command(THD*) /home/yura/ws/percona-server/sql/sql_parse.cc:1065
#10 0x557ccb6ac294 in handle_connection /home/yura/ws/percona-server/sql/conn_handler/connection_handler_per_thread.cc:325
#11 0x557ccbbfabb0 in pfs_spawn_thread /home/yura/ws/percona-server/storage/perfschema/pfs.cc:2198
#12 0x7fe760ab544f in start_thread nptl/pthread_create.c:473
```
The reason is that `my_thread_cancel` is used to finish the daemon thread. This is not and orderly way of finishing the thread. ASAN does not register the stack variables are not used anymore which generates the error above.
This is a benign error as all the variables are on the stack.
*Solution*:
Finish the thread in orderly way by using a signalling variable.
---------------------------------------------------------------------------
PS-8204: Fix XML escape rules for audit plugin
https://jira.percona.com/browse/PS-8204
There was a wrong length specified for some XML
escape rules. As a result of this terminating null symbol from
replacement rule was copied into resulting string. This lead to
quer text truncation in audit log file.
In addition added empty replacement rules for '\b' and 'f' symbols
which just remove them from resulting string. These symboles are
not supported in XML 1.0.
---------------------------------------------------------------------------
PS-8854: Add main.percona_udf MTR test
Add a test to check FNV1A_64, FNV_64, and MURMUR_HASH user-defined functions.
inikep
pushed a commit
that referenced
this pull request
Feb 8, 2024
For the Ndb_cluster_connection::configure_tls() method added in wl#15135, this patch adds the MGM TLS level as a second argument. When some data node requires TLS, but an API node does not have a certificate, the API node will fail quickly. In mysqld, this adds the --ndb-mgm-tls option. In Cluster/J, a new property com.mysql.clusterj.tls.strict that takes numeric values 0 or 1 and defaults to 0. Add an NDBAPI test, testMgmd -n ApiWithoutCertificate Change-Id: If62349c288b0bef2b4594662ade3befa4dc5ed8a
inikep
added a commit
that referenced
this pull request
Feb 8, 2024
PS-5741: Incorrect use of memset_s in keyring_vault.
Fixed the usage of memset_s. The arguments should be:
void memset_s(void *dest, size_t dest_max, int c, size_t n)
where the 2nd argument is size of buffer and the 3rd is
argument is character to fill.
---------------------------------------------------------------------------
PS-7769 - Fix use-after-return error in audit_log_exclude_accounts_validate
---
*Problem:*
`st_mysql_value::val_str` might return a pointer to `buf` which after
the function called is deleted. Therefore the value in `save`, after
reuturnin from the function, is invalid.
In this particular case, the error is not manifesting as val_str`
returns memory allocated with `thd_strmake` and it does not use `buf`.
*Solution:*
Allocate memory with `thd_strmake` so the memory in `save` is not local.
---------------------------------------------------------------------------
Fix test main.bug12969156 when WITH_ASAN=ON
*Problem:*
ASAN complains about stack-buffer-overflow on function `mysql_heartbeat`:
```
==90890==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fe746d06d14 at pc 0x7fe760f5b017 bp 0x7fe746d06cd0 sp 0x7fe746d06478
WRITE of size 24 at 0x7fe746d06d14 thread T16777215
Address 0x7fe746d06d14 is located in stack of thread T26 at offset 340 in frame
#0 0x7fe746d0a55c in mysql_heartbeat(void*) /home/yura/ws/percona-server/plugin/daemon_example/daemon_example.cc:62
This frame has 4 object(s):
[48, 56) 'result' (line 66)
[80, 112) '_db_stack_frame_' (line 63)
[144, 200) 'tm_tmp' (line 67)
[240, 340) 'buffer' (line 65) <== Memory access at offset 340 overflows this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
(longjmp and C++ exceptions *are* supported)
Thread T26 created by T25 here:
#0 0x7fe760f5f6d5 in __interceptor_pthread_create ../../../../src/libsanitizer/asan/asan_interceptors.cpp:216
#1 0x557ccbbcb857 in my_thread_create /home/yura/ws/percona-server/mysys/my_thread.c:104
#2 0x7fe746d0b21a in daemon_example_plugin_init /home/yura/ws/percona-server/plugin/daemon_example/daemon_example.cc:148
#3 0x557ccb4c69c7 in plugin_initialize /home/yura/ws/percona-server/sql/sql_plugin.cc:1279
#4 0x557ccb4d19cd in mysql_install_plugin /home/yura/ws/percona-server/sql/sql_plugin.cc:2279
#5 0x557ccb4d218f in Sql_cmd_install_plugin::execute(THD*) /home/yura/ws/percona-server/sql/sql_plugin.cc:4664
#6 0x557ccb47695e in mysql_execute_command(THD*, bool) /home/yura/ws/percona-server/sql/sql_parse.cc:5160
#7 0x557ccb47977c in mysql_parse(THD*, Parser_state*, bool) /home/yura/ws/percona-server/sql/sql_parse.cc:5952
#8 0x557ccb47b6c2 in dispatch_command(THD*, COM_DATA const*, enum_server_command) /home/yura/ws/percona-server/sql/sql_parse.cc:1544
#9 0x557ccb47de1d in do_command(THD*) /home/yura/ws/percona-server/sql/sql_parse.cc:1065
#10 0x557ccb6ac294 in handle_connection /home/yura/ws/percona-server/sql/conn_handler/connection_handler_per_thread.cc:325
#11 0x557ccbbfabb0 in pfs_spawn_thread /home/yura/ws/percona-server/storage/perfschema/pfs.cc:2198
#12 0x7fe760ab544f in start_thread nptl/pthread_create.c:473
```
The reason is that `my_thread_cancel` is used to finish the daemon thread. This is not and orderly way of finishing the thread. ASAN does not register the stack variables are not used anymore which generates the error above.
This is a benign error as all the variables are on the stack.
*Solution*:
Finish the thread in orderly way by using a signalling variable.
---------------------------------------------------------------------------
PS-8204: Fix XML escape rules for audit plugin
https://jira.percona.com/browse/PS-8204
There was a wrong length specified for some XML
escape rules. As a result of this terminating null symbol from
replacement rule was copied into resulting string. This lead to
quer text truncation in audit log file.
In addition added empty replacement rules for '\b' and 'f' symbols
which just remove them from resulting string. These symboles are
not supported in XML 1.0.
---------------------------------------------------------------------------
PS-8854: Add main.percona_udf MTR test
Add a test to check FNV1A_64, FNV_64, and MURMUR_HASH user-defined functions.
inikep
pushed a commit
that referenced
this pull request
Mar 15, 2024
…ocal DDL
executed
https://perconadev.atlassian.net/browse/PS-9018
Problem
-------
In high concurrency scenarios, MySQL replica can enter into a deadlock due to a
race condition between the replica applier thread and the client thread
performing a binlog group commit.
Analysis
--------
It needs at least 3 threads for this deadlock to happen
1. One client thread
2. Two replica applier threads
How this deadlock happens?
--------------------------
0. Binlog is enabled on replica, but log_replica_updates is disabled.
1. Initially, both "Commit Order" and "Binlog Flush" queues are empty.
2. Replica applier thread 1 enters the group commit pipeline to register in the
"Commit Order" queue since `log-replica-updates` is disabled on the replica
node.
3. Since both "Commit Order" and "Binlog Flush" queues are empty, the applier
thread 1
3.1. Becomes leader (In Commit_stage_manager::enroll_for()).
3.2. Registers in the commit order queue.
3.3. Acquires the lock MYSQL_BIN_LOG::LOCK_log.
3.4. Commit Order queue is emptied, but the lock MYSQL_BIN_LOG::LOCK_log is
not yet released.
NOTE: SE commit for applier thread is already done by the time it reaches
here.
4. Replica applier thread 2 enters the group commit pipeline to register in the
"Commit Order" queue since `log-replica-updates` is disabled on the replica
node.
5. Since the "Commit Order" queue is empty (emptied by applier thread 1 in 3.4), the
applier thread 2
5.1. Becomes leader (In Commit_stage_manager::enroll_for())
5.2. Registers in the commit order queue.
5.3. Tries to acquire the lock MYSQL_BIN_LOG::LOCK_log. Since it is held by applier
thread 1 it will wait until the lock is released.
6. Client thread enters the group commit pipeline to register in the
"Binlog Flush" queue.
7. Since "Commit Order" queue is not empty (there is applier thread 2 in the
queue), it enters the conditional wait `m_stage_cond_leader` with an
intention to become the leader for both the "Binlog Flush" and
"Commit Order" queues.
8. Applier thread 1 releases the lock MYSQL_BIN_LOG::LOCK_log and proceeds to update
the GTID by calling gtid_state->update_commit_group() from
Commit_order_manager::flush_engine_and_signal_threads().
9. Applier thread 2 acquires the lock MYSQL_BIN_LOG::LOCK_log.
9.1. It checks if there is any thread waiting in the "Binlog Flush" queue
to become the leader. Here it finds the client thread waiting to be
the leader.
9.2. It releases the lock MYSQL_BIN_LOG::LOCK_log and signals on the
cond_var `m_stage_cond_leader` and enters a conditional wait until the
thread's `tx_commit_pending` is set to false by the client thread
(will be done in the
Commit_stage_manager::process_final_stage_for_ordered_commit_group()
called by client thread from fetch_and_process_flush_stage_queue()).
10. The client thread wakes up from the cond_var `m_stage_cond_leader`. The
thread has now become a leader and it is its responsibility to update GTID
of applier thread 2.
10.1. It acquires the lock MYSQL_BIN_LOG::LOCK_log.
10.2. Returns from `enroll_for()` and proceeds to process the
"Commit Order" and "Binlog Flush" queues.
10.3. Fetches the "Commit Order" and "Binlog Flush" queues.
10.4. Performs the storage engine flush by calling ha_flush_logs() from
fetch_and_process_flush_stage_queue().
10.5. Proceeds to update the GTID of threads in "Commit Order" queue by
calling gtid_state->update_commit_group() from
Commit_stage_manager::process_final_stage_for_ordered_commit_group().
11. At this point, we will have
- Client thread performing GTID update on behalf if applier thread 2 (from step 10.5), and
- Applier thread 1 performing GTID update for itself (from step 8).
Due to the lack of proper synchronization between the above two threads,
there exists a time window where both threads can call
gtid_state->update_commit_group() concurrently.
In subsequent steps, both threads simultaneously try to modify the contents
of the array `commit_group_sidnos` which is used to track the lock status of
sidnos. This concurrent access to `update_commit_group()` can cause a
lock-leak resulting in one thread acquiring the sidno lock and not
releasing at all.
-----------------------------------------------------------------------------------------------------------
Client thread Applier Thread 1
-----------------------------------------------------------------------------------------------------------
update_commit_group() => global_sid_lock->rdlock(); update_commit_group() => global_sid_lock->rdlock();
calls update_gtids_impl_lock_sidnos() calls update_gtids_impl_lock_sidnos()
set commit_group_sidno[2] = true set commit_group_sidno[2] = true
lock_sidno(2) -> successful
lock_sidno(2) -> waits
update_gtids_impl_own_gtid() -> Add the thd->owned_gtid in `executed_gtids()`
if (commit_group_sidnos[2]) {
unlock_sidno(2);
commit_group_sidnos[2] = false;
}
Applier thread continues..
lock_sidno(2) -> successful
update_gtids_impl_own_gtid() -> Add the thd->owned_gtid in `executed_gtids()`
if (commit_group_sidnos[2]) { <=== this check fails and lock is not released.
unlock_sidno(2);
commit_group_sidnos[2] = false;
}
Client thread continues without releasing the lock
-----------------------------------------------------------------------------------------------------------
12. As the above lock-leak can also happen the other way i.e, the applier
thread fails to unlock, there can be different consequences hereafter.
13. If the client thread continues without releasing the lock, then at a later
stage, it can enter into a deadlock with the applier thread performing a
GTID update with stack trace.
Client_thread
-------------
#1 __GI___lll_lock_wait
#2 ___pthread_mutex_lock
#3 native_mutex_lock <= waits for commit lock while holding sidno lock
#4 Commit_stage_manager::enroll_for
#5 MYSQL_BIN_LOG::change_stage
#6 MYSQL_BIN_LOG::ordered_commit
#7 MYSQL_BIN_LOG::commit
#8 ha_commit_trans
#9 trans_commit_implicit
#10 mysql_create_like_table
#11 Sql_cmd_create_table::execute
#12 mysql_execute_command
percona#13 dispatch_sql_command
Applier thread
--------------
#1 ___pthread_mutex_lock
#2 native_mutex_lock
#3 safe_mutex_lock
#4 Gtid_state::update_gtids_impl_lock_sidnos <= waits for sidno lock
#5 Gtid_state::update_commit_group
#6 Commit_order_manager::flush_engine_and_signal_threads <= acquires commit lock here
#7 Commit_order_manager::finish
#8 Commit_order_manager::wait_and_finish
#9 ha_commit_low
#10 trx_coordinator::commit_in_engines
#11 MYSQL_BIN_LOG::commit
#12 ha_commit_trans
percona#13 trans_commit
percona#14 Xid_log_event::do_commit
percona#15 Xid_apply_log_event::do_apply_event_worker
percona#16 Slave_worker::slave_worker_exec_event
percona#17 slave_worker_exec_job_group
percona#18 handle_slave_worker
14. If the applier thread continues without releasing the lock, then at a later
stage, it can perform recursive locking while setting the GTID for the next
transaction (in set_gtid_next()).
In debug builds the above case hits the assertion
`safe_mutex_assert_not_owner()` meaning the lock is already acquired by the
replica applier thread when it tries to re-acquire the lock.
Solution
--------
In the above problematic example, when seen from each thread
individually, we can conclude that there is no problem in the order of lock
acquisition, thus there is no need to change the lock order.
However, the root cause for this problem is that multiple threads can
concurrently access to the array `Gtid_state::commit_group_sidnos`.
In its initial implementation, it was expected that threads should
hold the `MYSQL_BIN_LOG::LOCK_commit` before modifying its contents. But it
was not considered when upstream implemented WL#7846 (MTS:
slave-preserve-commit-order when log-slave-updates/binlog is disabled).
With this patch, we now ensure that `MYSQL_BIN_LOG::LOCK_commit` is acquired
when the client thread (binlog flush leader) when it tries to perform GTID
update on behalf of threads waiting in "Commit Order" queue, thus providing a
guarantee that `Gtid_state::commit_group_sidnos` array is never accessed
without the protection of `MYSQL_BIN_LOG::LOCK_commit`.
inikep
pushed a commit
that referenced
this pull request
Jun 14, 2024
When built with ASAN, a use-after-free is reported for the TcpPortPool.
AddressSanitizer: heap-use-after-free on address 0x60200019f190 at pc
0x00000076a18d bp 0x7fff51e7d1d0 sp 0x7fff51e7d1c0
#4 0x770b73 in UniqueId::ProcessUniqueIds::erase(unsigned int)
../router/tests/helpers/tcp_port_pool.h:112
#5 0x770c48 in UniqueId::~UniqueId()
../router/tests/helpers/tcp_port_pool.cc:234
...
#12 0x82faa3 in testing::UnitTest::~UnitTest()
../extra/googletest/googletest-release-1.12.0/googletest/src/gtest.cc:5496
percona#13 0x7f5fe085ace8 in __run_exit_handlers (/lib64/libc.so.6+0x39ce8)
0x60200019f190 is located 0 bytes inside of 16-byte region
[0x60200019f190,0x60200019f1a0)
freed by thread T0 here:
#0 0x7f5fe3cbd10f in operator delete(void*, unsigned long)
(/lib64/libasan.so.6+0xb710f)
#1 0x7f5fe085ace8 in __run_exit_handlers (/lib64/libc.so.6+0x39ce8)
Background
==========
__run_exit_handlers destroys "static" and "global" variables in reverse
order of their creation.
googletest's unit-tests are a static, and the TcpPortPool also has
ProcessUniqueId's which contains the process-wide unique-ids.
At construct: unittest -> tcp-port-pool -> proces-unique-ids
At destruct : process-unique-ids -> tcp-port-pool -> 💥
The use-after-free happens as the process-unique-ids static is
destructed before the tcp-port-pool which tries to its Ids from the
process-unique-ids.
Change
======
- extend the lifetime of the process-unique-ids to after the last use of
the tcp-port-pool via a std::shared_ptr<>
Change-Id: I75b8b781e1d240f18ca72f2c86182639a7699f06
inikep
pushed a commit
that referenced
this pull request
Jun 14, 2024
…ocal DDL
executed
https://perconadev.atlassian.net/browse/PS-9018
Merge remote-tracking branch 'venki/PS-9018-8.0-gca' into HEAD
Problem
-------
In high concurrency scenarios, MySQL replica can enter into a deadlock due to a
race condition between the replica applier thread and the client thread
performing a binlog group commit.
Analysis
--------
It needs at least 3 threads for this deadlock to happen
1. One client thread
2. Two replica applier threads
How this deadlock happens?
--------------------------
0. Binlog is enabled on replica, but log_replica_updates is disabled.
1. Initially, both "Commit Order" and "Binlog Flush" queues are empty.
2. Replica applier thread 1 enters the group commit pipeline to register in the
"Commit Order" queue since `log-replica-updates` is disabled on the replica
node.
3. Since both "Commit Order" and "Binlog Flush" queues are empty, the applier
thread 1
3.1. Becomes leader (In Commit_stage_manager::enroll_for()).
3.2. Registers in the commit order queue.
3.3. Acquires the lock MYSQL_BIN_LOG::LOCK_log.
3.4. Commit Order queue is emptied, but the lock MYSQL_BIN_LOG::LOCK_log is
not yet released.
NOTE: SE commit for applier thread is already done by the time it reaches
here.
4. Replica applier thread 2 enters the group commit pipeline to register in the
"Commit Order" queue since `log-replica-updates` is disabled on the replica
node.
5. Since the "Commit Order" queue is empty (emptied by applier thread 1 in 3.4), the
applier thread 2
5.1. Becomes leader (In Commit_stage_manager::enroll_for())
5.2. Registers in the commit order queue.
5.3. Tries to acquire the lock MYSQL_BIN_LOG::LOCK_log. Since it is held by applier
thread 1 it will wait until the lock is released.
6. Client thread enters the group commit pipeline to register in the
"Binlog Flush" queue.
7. Since "Commit Order" queue is not empty (there is applier thread 2 in the
queue), it enters the conditional wait `m_stage_cond_leader` with an
intention to become the leader for both the "Binlog Flush" and
"Commit Order" queues.
8. Applier thread 1 releases the lock MYSQL_BIN_LOG::LOCK_log and proceeds to update
the GTID by calling gtid_state->update_commit_group() from
Commit_order_manager::flush_engine_and_signal_threads().
9. Applier thread 2 acquires the lock MYSQL_BIN_LOG::LOCK_log.
9.1. It checks if there is any thread waiting in the "Binlog Flush" queue
to become the leader. Here it finds the client thread waiting to be
the leader.
9.2. It releases the lock MYSQL_BIN_LOG::LOCK_log and signals on the
cond_var `m_stage_cond_leader` and enters a conditional wait until the
thread's `tx_commit_pending` is set to false by the client thread
(will be done in the
Commit_stage_manager::process_final_stage_for_ordered_commit_group()
called by client thread from fetch_and_process_flush_stage_queue()).
10. The client thread wakes up from the cond_var `m_stage_cond_leader`. The
thread has now become a leader and it is its responsibility to update GTID
of applier thread 2.
10.1. It acquires the lock MYSQL_BIN_LOG::LOCK_log.
10.2. Returns from `enroll_for()` and proceeds to process the
"Commit Order" and "Binlog Flush" queues.
10.3. Fetches the "Commit Order" and "Binlog Flush" queues.
10.4. Performs the storage engine flush by calling ha_flush_logs() from
fetch_and_process_flush_stage_queue().
10.5. Proceeds to update the GTID of threads in "Commit Order" queue by
calling gtid_state->update_commit_group() from
Commit_stage_manager::process_final_stage_for_ordered_commit_group().
11. At this point, we will have
- Client thread performing GTID update on behalf if applier thread 2 (from step 10.5), and
- Applier thread 1 performing GTID update for itself (from step 8).
Due to the lack of proper synchronization between the above two threads,
there exists a time window where both threads can call
gtid_state->update_commit_group() concurrently.
In subsequent steps, both threads simultaneously try to modify the contents
of the array `commit_group_sidnos` which is used to track the lock status of
sidnos. This concurrent access to `update_commit_group()` can cause a
lock-leak resulting in one thread acquiring the sidno lock and not
releasing at all.
-----------------------------------------------------------------------------------------------------------
Client thread Applier Thread 1
-----------------------------------------------------------------------------------------------------------
update_commit_group() => global_sid_lock->rdlock(); update_commit_group() => global_sid_lock->rdlock();
calls update_gtids_impl_lock_sidnos() calls update_gtids_impl_lock_sidnos()
set commit_group_sidno[2] = true set commit_group_sidno[2] = true
lock_sidno(2) -> successful
lock_sidno(2) -> waits
update_gtids_impl_own_gtid() -> Add the thd->owned_gtid in `executed_gtids()`
if (commit_group_sidnos[2]) {
unlock_sidno(2);
commit_group_sidnos[2] = false;
}
Applier thread continues..
lock_sidno(2) -> successful
update_gtids_impl_own_gtid() -> Add the thd->owned_gtid in `executed_gtids()`
if (commit_group_sidnos[2]) { <=== this check fails and lock is not released.
unlock_sidno(2);
commit_group_sidnos[2] = false;
}
Client thread continues without releasing the lock
-----------------------------------------------------------------------------------------------------------
12. As the above lock-leak can also happen the other way i.e, the applier
thread fails to unlock, there can be different consequences hereafter.
13. If the client thread continues without releasing the lock, then at a later
stage, it can enter into a deadlock with the applier thread performing a
GTID update with stack trace.
Client_thread
-------------
#1 __GI___lll_lock_wait
#2 ___pthread_mutex_lock
#3 native_mutex_lock <= waits for commit lock while holding sidno lock
#4 Commit_stage_manager::enroll_for
#5 MYSQL_BIN_LOG::change_stage
#6 MYSQL_BIN_LOG::ordered_commit
#7 MYSQL_BIN_LOG::commit
#8 ha_commit_trans
#9 trans_commit_implicit
#10 mysql_create_like_table
#11 Sql_cmd_create_table::execute
#12 mysql_execute_command
percona#13 dispatch_sql_command
Applier thread
--------------
#1 ___pthread_mutex_lock
#2 native_mutex_lock
#3 safe_mutex_lock
#4 Gtid_state::update_gtids_impl_lock_sidnos <= waits for sidno lock
#5 Gtid_state::update_commit_group
#6 Commit_order_manager::flush_engine_and_signal_threads <= acquires commit lock here
#7 Commit_order_manager::finish
#8 Commit_order_manager::wait_and_finish
#9 ha_commit_low
#10 trx_coordinator::commit_in_engines
#11 MYSQL_BIN_LOG::commit
#12 ha_commit_trans
percona#13 trans_commit
percona#14 Xid_log_event::do_commit
percona#15 Xid_apply_log_event::do_apply_event_worker
percona#16 Slave_worker::slave_worker_exec_event
percona#17 slave_worker_exec_job_group
percona#18 handle_slave_worker
14. If the applier thread continues without releasing the lock, then at a later
stage, it can perform recursive locking while setting the GTID for the next
transaction (in set_gtid_next()).
In debug builds the above case hits the assertion
`safe_mutex_assert_not_owner()` meaning the lock is already acquired by the
replica applier thread when it tries to re-acquire the lock.
Solution
--------
In the above problematic example, when seen from each thread
individually, we can conclude that there is no problem in the order of lock
acquisition, thus there is no need to change the lock order.
However, the root cause for this problem is that multiple threads can
concurrently access to the array `Gtid_state::commit_group_sidnos`.
In its initial implementation, it was expected that threads should
hold the `MYSQL_BIN_LOG::LOCK_commit` before modifying its contents. But it
was not considered when upstream implemented WL#7846 (MTS:
slave-preserve-commit-order when log-slave-updates/binlog is disabled).
With this patch, we now ensure that `MYSQL_BIN_LOG::LOCK_commit` is acquired
when the client thread (binlog flush leader) when it tries to perform GTID
update on behalf of threads waiting in "Commit Order" queue, thus providing a
guarantee that `Gtid_state::commit_group_sidnos` array is never accessed
without the protection of `MYSQL_BIN_LOG::LOCK_commit`.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.