New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Warning from compiling clone_os.cc #1270
Comments
Another one, but an error rather than a warning
|
Should read |
Was already fixed last week in the clone PR, apparently an older version had been merged instead |
I'm in the process of fixing this. A slightly older version of the PR was pushed, and the patch summaries were also missing. The fixed patch should be on its way via a forced update. |
The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors
The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors
Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes #1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: #1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 2245706
Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook#1271 GitHub Author: Laurynas Biveinis <laurynas.biveinis@gmail.com> Test Plan: Imported from GitHub, without a `Test Plan:` line. Reviewers: chni Reviewed By: chni Subscribers: webscalesql-eng@fb.com Differential Revision: https://phabricator.intern.facebook.com/D43259662 Tags: accept2ship
Should be fixed by 5728dd7 |
Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook/mysql-5.6#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook/mysql-5.6#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook/mysql-5.6#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
Summary: Pull Request resolved: facebook#1250 MyRocks Clone (1/n): make InnoDB clone MTR tests SE-agnostic The MyRocks clone MTR tests will reuse the InnoDB tests by including them. Adjust the tests so that they work this way, i.e. make them storage engine-agnostic by checking the default_storage_engine system variable and guarding InnoDB-specific tests this way. This includes the tests that artificially desync the InnoDB redo log position. MyRocks Clone (2/n): fix bug 109919 (Clone plugin does not support arbitrary length O_DIRECT files) The existing clone plugin I/O routines handle O_DIRECT files with the assumption that their sizes are always a multiple of a block size. This is too restrictive because in general O_DIRECT files can be of any size and MyRocks files actually are. Add a flag to Ha_clone_file to support arbitrary file lengths under O_DIRECT: o_direct_uneven_file_size. Pass it to clone_client.h/cc Client::get_aligned_buffer to allocate the buffer that bumps the length to the next CLONE_OS_ALIGN multiple if it's not already one. Introduce a pad_for_o_direct helper that performs this calculation for the passed length as needed. For writes, if O_DIRECT is enabled and the buffer length is not a multiple of CLONE_OS_ALIGN, pad it with zeros to the next multiple. Then check the actual bytes written, and if any padding bytes were actually written, call my_chsize to truncate the file to the needed size. MyRocks Clone (3/n): fix bug 109920 (SEs cannot add to clone data size estimates) In the current implementation, once the first SE sets the data size estimate for a particular stage, there is no way for other SEs to add to it. Address this by adding Ha_clone_cbk::add_to_data_size_estimate method. MyRocks Clone (4/n): fix bug 109921 (Cannot clone with synchronization across SEs) The existing clone implementation synchronizes InnoDB and binlog directly. There is no way to make other storage engines join this synchronization, thus replace it with synchronizing by a performance_schema.log_status query. - Add table_log_status::get_row method to allow doing the log_status query from inside the server. - Handlerton: introduce two new methods: clone_precopy - currently unused, which will handle any non-consistent-snapshot copying before the main consistent copy, and set_log_stop - which provides the SE the clone stop position information. - Plugin: introduce Ha_clone_common_cbk, which derives from Ha_clone_cbk, and from which derive all the concrete classes: Client_Cbk, Local_Cbk & Server_Cbk. - Ha_clone_common_cbk contains two methods: precopy and synchronize_engines. The former calls precopy for all the participating clone SE handlertons, if implemented. The latter, only to be called from the main clone thread, performs the performance_schema.LOG_STATUS query, splits its result per-SE and calls handlerton set_log_stop for all SEs. - The pfs.log_status query still must be called by InnoDB at a certain point in clone. That means that InnoDB is still a special SE in that it must always come first in the clone locator vectors. This will also handle donor waits on clone restarts, as that is implemented in InnoDB-specific code. To ensure this, add a helper clone_hton.cc:make_innodb_first which shuffles the vectors as needed. Call it in all the clone-starting code paths, add asserts that InnoDB comes first. - Since hton_clone_copy will call into InnoDB clone copy, which at certain point will synchronize engines, calling into the plugin again, and the calling into SEs again, introduce a backup locator vector Ha_clone_cbk::all_locators, and its setters/getters set_all_locators, reset_all_locators, get_all_locators. Use them in the re-entrant hton_clone_copy code. - Introduce a new InnoDB clone state named SST COPY, which calls precopy and has zero files and chunks set to copy in the regular InnoDB clone way, i.e. all it does it the precopy. - The redo log archiving keeps running throughout the precopy stage. - On stage transition from the precopy to the redo log copy. call synchronize. - Implement innodb_clone_set_log_stop by parsing out the desired clone stop LSN from the passed JSON from the log_status query. - Make InnoDB redo log archiving stop at that LSN instead of on reaching the end of redo log. - Support stopping redo log archiving in a middle of a log block. As the old implementation may encounter this case only at the end of the redo log, it took that block from the log buffer in memory. This no longer works with the new implementation, where the needed block might reside on disk only. Address this by forcing the whole block containing the target LSN to disk, reading it from there, and patching its memory copy to stop at the required LSN before giving it for the clone to copy. - To implement redo log block reading on a running server, adapt crash recovery function recv_read_log_seg to work both for crash recovery and on a running server. This is similar to what Percona Server does for its changed page tracking feature. - Make sure not to archive not-yet-written log: pfs.log_status query gets the log system LSN, which is the value of highest log reservation. Thus in an archiving iteration, archive up to the lower values of log archiving LSN, and write_lsn. Set the log archiving LSN to LSN_MAX at clone initialization, so that clone archiving may proceed until the stop LSN is set. - The above implementation results in cloned LSN != log end LSN for the cloned instance, thus adjust crash recovery to work with that. The existing crash recovery throws away any crash-recovered metadata and reads its snapshot from the data files, whereas the snapshot was not written in any consistency with the clone LSN. Fix by no longer throwing away the crash-recovered InnoDB dynamic metadata and by not re-reading the the last metadata snapshot. This forces also not to re-serialize the metadata in the crash recovery as that writes redo log which is to be avoided during clone crash recovery. - Additionally, at the start of the redo log copy the old implementation requested all InnoDB updates to be flushed to disk immediately, to avoid clone recovery generating new redo log. This is also disabled as it becomes redundant with the fix. This issue is similar to Percona XtraBackup https://jira.percona.com/browse/PXB-1634 - Disable clone.local_xa test because the new synchronization no longer blocks on XA transactions. - Re-record two clone-related tests in the component_keyring_file MTR suite. - Temporarily workaround the MyRocks bug where log_status query hangs if RocksDB file deletions were not disabled first. Note that this change breaks some of the clone-related group replication MTR tests. As they are not tested in fb-mysql and have been already broken, this patch makes no effort to fix them. MyRocks Clone (5/n): fix bug 109922 (SEs do not coordinate clone rollbacks on instance startup) If InnoDB clone decides to rollback during its startup, other storage engines are not aware of this and proceed as usual, resulting in inconsistent instance between storage engines. Ideally this should be fixed through a 2PC-like design where all the SEs vote to rollback the clone or to continue during the startup, but for MyRocks clone a simpler solution suffices: make InnoDB write a special marker file if it decides to rollback on clone startup. MyRocks Clone (6/n): on remote clone error, always send COM_ACK to InnoDB too In the case of multithreaded clone, the waits for the donor threads for the current clone stage to complete are implemented fully in InnoDB, even though that, and error handling, should be plugin-level features. With two storage engines participating in the clone this allows the situation where the other storage engine client reports an error with COM_ACK to one donor thread, while some other threads are in InnoDB clone state change waits. Those waits can abort if InnoDB clone gets an error from the client, but they know nothing about other SE errors, and will be aborted only by timeouts. As a minimal fix that does not involve pulling the whole error handling and state change waiting logic to the plugin layer, make the clone client always send an error-containing COM_ACK to InnoDB too, whenever it sends one to another storage engine. MyRocks Clone (7/n): MyRocks clone implementation - Cleanup any stale RocksDB checkpoint directories on server shutdown too. - For MyRocks clone network protocol, define four packet types (clone/common.h): - LOCATOR_V1: MyRocks-specific clone locator, consisting of a monotonically increasing clone session ID for the donor with an optional applier state (an array of file IDs and applied offsets) for restarts, or a zero in which case it is a version check empty locator used for handshakes. - FILE_NAME_MAP_V1: establishes a relation between a file ID, its name and target size. - FILE_CHUNK_V1: cloned file chunk data - ADD_ESTIMATE_V1: how many bytes should be added to the estimated clone data size. - All packet types share the common package header, which stores the version, the payload length without the header, and the packet type. - (clone/common.h): classes locator, file_metadata, chunk_metadata, & estimate_metadata correspond to the four packet types. They are subclassed on the donor side to add serialization routines and other donor-specific code and on the client side to add deserialization routines and other client-specific code. Re. serialization/deserialization, an exception is the locator, whose routines are implemented in the common code, because it may be passed both - ways. - (clone/common.h): Introduce a session class that handles the task ID set and errors, which is common for both donor and client. For errors, allow saving one error, with an optional associated path and errno, that can be set by any thread and later reported to the clone plugin. - (clone/common.cc) Setup MyRocks clone capabilities as HA_CLONE_HYBRID (not - applicable but matches InnoDB), HA_CLONE_MULTI_TASK (can clone with multiple - threads), & HA_CLONE_RESTART (can restart after a network I/O error). - (clone/common.cc) Enable O_DIRECT for clone file I/O if, on client if rocksdb_use_direct_io_for_flush_and_compaction is enabled, and on donor if rocksdb_use_direct_reads is enabled. - On the donor side (clone/donor.cc), encapsulate RocksDB checkpoints in a rdb_checkpoint class. Each checkpoint gets its own ID, which is also a donor session ID. The ID is constant for rolling checkpoints. - Subclass the common file_metadata to donor_file_metadata, chunk_metadata to donor_chunk_metadata, estimate_metadata to donor_estimate. - Implement donor going through a sequence of states: NONE, ROLLING, FINAL, FINAL_WITH_LOGS. To support multiple clone donor sessions, implement a manager class that looks up the donor instance by locator. - Implement MyRocks clone donor interface: - rocksdb_clone_start: handle the possible start modes: - HA_CLONE_MODE_START: create and register a new donor instance, register its master task ID. - HA_CLONE_MODE_ADD_TASK: lookup the existing donor instance, register a new task ID. - HA_CLONE_MODE_RESTART: lookup the existing donor instance, wait for the old master session to drop it by master task ID reference count, then process the locator applier state to reset the donor state. - rocksdb_clone_precopy: lookup the donor instance, create a RocksDB snapshot if state == NONE, otherwise send files by chunks and roll the checkpoint as needed. - rocksdb_clone_set_log_stop: add the WAL files to the clone file to copy set. - rocksdb_clone_copy: copy the clone files in any donor state. - rocksdb_clone_ack: acknowledge errors sent by client. - rocksdb_clone_end: for master task ID, stop accepting new worker join requests, wait until existing workers quit, and, in the case of restartable error, wait for any restarts to happen. Then decrement ref count for the master task ID (which will handle both the cases of restarted and not restarted) clone. For all other task IDs, simply unregister them. - On the client side (clone/client.cc), have file_in_progress class to track each individual file progress during application, and completed_file for fully-applied files. - Subclass the common file_metadata to client_file_metadata, implement its reading from the net buffer. Likewise for client_chunk_metadata subclassing chunk_metadata, and client_estimate for estimate_metadata. - TODO client class. - Implement MyRocks clone client interface: - rocksdb_clone_apply_begin: handle the possible start modes: - HA_CLONE_MODE_START: initialize the client instance for a new clone - HA_CLONE_MODE_ADD_TASK: register a new task ID for an ongoing clone - HA_CLONE_MODE_RESTART: restart the interrupted clone session - HA_CLONE_MODE_VERSION: prepare new version locator - rocksdb_clone_apply: if passed an error code, save it. Otherwise verify the passed buffer and dispatch ADD_ESTIMATE_V1, FILE_NAME_MAP_V1, & FILE_CHUNK_V1 packets accordingly. - rocksdb_clone_apply_end: for master task, wait for all other tasks to complete and finish the clone. For both master and non-master tasks unregister their IDs. - Add rocksdb_sys_vars tests for the new clone-related system variables. They are included in the current patch and not the MTR one in order not to break the existing rocksdb_sys_vars.all_vars test. - On instance startup, do clone fixup (clone/common.cc). It consists of: - Check whether InnoDB clone rollback marker is present. If yes, force rollback of MyRocks clone. - If the cloned temporary datadir contains clone-in-progress marker, assume half-completed clone and abort. - Move the cloned datadirs and WAL dirs in-place. For the datadir, do a directory-level rename with a save and later delete of the old datadir. For the WAL dir do not try to do parent directory-level renames because the mysqld process may not have permissions to delete that directory. Instead, rename all the old files individually, move new files in, delete the old files. - For all the rename operations, handle them returning EXDEV error (a rename would cross a filesystem boundary) by falling back to copy and delete. As we expect that it is the cloned RocksDB log files that are most affected by this, the total copy I/O wouldn't be large, and so the default MySQL copy routine my_copy is used, without extra copy throughput optimizations. Add a fault injection point for MTR to test this code paths. - Remove any RocksDB clone checkpoints found. - On instance startup, always move WALs to the target directory if they are found in the rocksdb_datadir yet the rocksdb_wal_dir is different. This handles the case of setting rocksdb_wal_dir while instance is shutdown, both for clone fixup and regular operation. - Add PFS instrumentation for all the instrumentable entities in this patch (mutexes, rwlocks, file I/O). MyRocks Clone (8/n): MyRocks clone MTR tests - Add a new MTR test suite rocksdb_clone - Support new variable MTR clone/include/clone_command.inc: clone_single_threaded, setting which forces single-threaded clone operation. - Reuse a big part of InnoDB clone tests by including their test files, with the exception of tests involving DDL, as concurrent DDL is not supported by MyRocks clone at the moment, and debug tests that adjust cloned instance LSN, because they desynchronize storage engines. The reused tests are: local_basic, local_boundary, local_dml, local_dml_auto_tune, local_dml_no_redo, local_insert, local_stage, monitor_progress, remote_basic, remote_basic_replace, remote_boundary, remote_dml, remote_dml_auto_tune, remote_dml_gtid, remote_dml_no_binlog, remote_dml_no_ssl, remote_dml_recovery, remote_dml_replace, remote_dml_ssl, remote_dml_ssl_certificate, remote_dml_upgrade, remote_insert, remote_monitor_progress, remote_restart, remote_skip_validation, remote_stage, remote_stage_restart. - As a part of the above, add MyRocks support to local_basic and monitor_progress MTR tests. - Besides the reused InnoDB clone tests, add some new MyRocks-specific tests: - local_minimal_1_thread, which is the simplest possible local clone test. - local_minimal_multithreaded, the simplest possible but multithreaded local clone test. - local_minimal_direct_io, which is the simplest possible local clone test in all combinations of rocksdb_use_direct_reads and rocksdb_use_direct_io_for_flush_and_compaction settings, which are the O_DIRECT-enabling options. - local_dml_o_direct_donor, local_dml_o_direct_client, local_dml_o_direct_both that reuse InnoDB local_dml test, but combined with either or both of the O_DIRECT-enabling options. - local_minimal_wal_dir, which clones with various rocksdb_wal_dir settings on donor and client. - local_minimal_wal_dir_copy_delete, which uses a debug injection to test the move-across-filesystem-boundary code path in clone startup. - remote_minimal_direct_io_{1, 2, 3, 4} tests for all the combinations of O_DIRECT-enabling options for donor and client instances. - remote_fail_apply, which uses a debug injection to fail the clone client during remote clone. - remote_inplace_restart, which performs a remote in-place clone with a simulated transient error on the client. - remote_kill_donor, which kill the donor instance at various points of remote cloning. - rolling_checkpoint for testing various combinations of clone rolling checkpoint tuning options. - three_se_basic to test a simple clone across InnoDB, MyRocks, & Blackhole engines. Fix moving RocksDB WALs on startup across file system boundaries Reviewed By: sunshine-Chun Differential Revision: D43160162 Pulled By: hermanlee fbshipit-source-id: 5358a57 -------------------------------------------------------------------------------- Rdb clone fixes (facebook#1271) Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 2245706 -------------------------------------------------------------------------------- Clone: fix race condition that may skip some SST files (facebook#1274) Summary: A donor thread in the main copy loop took a file to send, released the donor state mutex, and then opened the file. If an ENOENT was received at this point, it was assumed that this was a stale SST file from an older checkpoint that was rolled since. Because the donor state mutex was released between taking of the file and opening it, the following race was possible: 1) Thread 1 takes the file 2) Thread 2 decides to roll the checkpoint, the old checkpoint is deleted 3) Thread 1 tries to open the file, gets ENOENT 4) Thread 2 creates the new checkpoint, the file re-appears, but it's too late. Rolling the checkpoint in a donor state mutex critical section is a possible fix, but such section would do a lot of I/O, serializing the parallel threads. Instead, fix by taking the file and opening it in the same critical section. Pull Request resolved: facebook#1274 Reviewed By: sunshine-Chun Differential Revision: D43629546 Pulled By: hermanlee fbshipit-source-id: 6e7f315
Summary: Pull Request resolved: facebook#1250 MyRocks Clone (1/n): make InnoDB clone MTR tests SE-agnostic The MyRocks clone MTR tests will reuse the InnoDB tests by including them. Adjust the tests so that they work this way, i.e. make them storage engine-agnostic by checking the default_storage_engine system variable and guarding InnoDB-specific tests this way. This includes the tests that artificially desync the InnoDB redo log position. MyRocks Clone (2/n): fix bug 109919 (Clone plugin does not support arbitrary length O_DIRECT files) The existing clone plugin I/O routines handle O_DIRECT files with the assumption that their sizes are always a multiple of a block size. This is too restrictive because in general O_DIRECT files can be of any size and MyRocks files actually are. Add a flag to Ha_clone_file to support arbitrary file lengths under O_DIRECT: o_direct_uneven_file_size. Pass it to clone_client.h/cc Client::get_aligned_buffer to allocate the buffer that bumps the length to the next CLONE_OS_ALIGN multiple if it's not already one. Introduce a pad_for_o_direct helper that performs this calculation for the passed length as needed. For writes, if O_DIRECT is enabled and the buffer length is not a multiple of CLONE_OS_ALIGN, pad it with zeros to the next multiple. Then check the actual bytes written, and if any padding bytes were actually written, call my_chsize to truncate the file to the needed size. MyRocks Clone (3/n): fix bug 109920 (SEs cannot add to clone data size estimates) In the current implementation, once the first SE sets the data size estimate for a particular stage, there is no way for other SEs to add to it. Address this by adding Ha_clone_cbk::add_to_data_size_estimate method. MyRocks Clone (4/n): fix bug 109921 (Cannot clone with synchronization across SEs) The existing clone implementation synchronizes InnoDB and binlog directly. There is no way to make other storage engines join this synchronization, thus replace it with synchronizing by a performance_schema.log_status query. - Add table_log_status::get_row method to allow doing the log_status query from inside the server. - Handlerton: introduce two new methods: clone_precopy - currently unused, which will handle any non-consistent-snapshot copying before the main consistent copy, and set_log_stop - which provides the SE the clone stop position information. - Plugin: introduce Ha_clone_common_cbk, which derives from Ha_clone_cbk, and from which derive all the concrete classes: Client_Cbk, Local_Cbk & Server_Cbk. - Ha_clone_common_cbk contains two methods: precopy and synchronize_engines. The former calls precopy for all the participating clone SE handlertons, if implemented. The latter, only to be called from the main clone thread, performs the performance_schema.LOG_STATUS query, splits its result per-SE and calls handlerton set_log_stop for all SEs. - The pfs.log_status query still must be called by InnoDB at a certain point in clone. That means that InnoDB is still a special SE in that it must always come first in the clone locator vectors. This will also handle donor waits on clone restarts, as that is implemented in InnoDB-specific code. To ensure this, add a helper clone_hton.cc:make_innodb_first which shuffles the vectors as needed. Call it in all the clone-starting code paths, add asserts that InnoDB comes first. - Since hton_clone_copy will call into InnoDB clone copy, which at certain point will synchronize engines, calling into the plugin again, and the calling into SEs again, introduce a backup locator vector Ha_clone_cbk::all_locators, and its setters/getters set_all_locators, reset_all_locators, get_all_locators. Use them in the re-entrant hton_clone_copy code. - Introduce a new InnoDB clone state named SST COPY, which calls precopy and has zero files and chunks set to copy in the regular InnoDB clone way, i.e. all it does it the precopy. - The redo log archiving keeps running throughout the precopy stage. - On stage transition from the precopy to the redo log copy. call synchronize. - Implement innodb_clone_set_log_stop by parsing out the desired clone stop LSN from the passed JSON from the log_status query. - Make InnoDB redo log archiving stop at that LSN instead of on reaching the end of redo log. - Support stopping redo log archiving in a middle of a log block. As the old implementation may encounter this case only at the end of the redo log, it took that block from the log buffer in memory. This no longer works with the new implementation, where the needed block might reside on disk only. Address this by forcing the whole block containing the target LSN to disk, reading it from there, and patching its memory copy to stop at the required LSN before giving it for the clone to copy. - To implement redo log block reading on a running server, adapt crash recovery function recv_read_log_seg to work both for crash recovery and on a running server. This is similar to what Percona Server does for its changed page tracking feature. - Make sure not to archive not-yet-written log: pfs.log_status query gets the log system LSN, which is the value of highest log reservation. Thus in an archiving iteration, archive up to the lower values of log archiving LSN, and write_lsn. Set the log archiving LSN to LSN_MAX at clone initialization, so that clone archiving may proceed until the stop LSN is set. - The above implementation results in cloned LSN != log end LSN for the cloned instance, thus adjust crash recovery to work with that. The existing crash recovery throws away any crash-recovered metadata and reads its snapshot from the data files, whereas the snapshot was not written in any consistency with the clone LSN. Fix by no longer throwing away the crash-recovered InnoDB dynamic metadata and by not re-reading the the last metadata snapshot. This forces also not to re-serialize the metadata in the crash recovery as that writes redo log which is to be avoided during clone crash recovery. - Additionally, at the start of the redo log copy the old implementation requested all InnoDB updates to be flushed to disk immediately, to avoid clone recovery generating new redo log. This is also disabled as it becomes redundant with the fix. This issue is similar to Percona XtraBackup https://jira.percona.com/browse/PXB-1634 - Disable clone.local_xa test because the new synchronization no longer blocks on XA transactions. - Re-record two clone-related tests in the component_keyring_file MTR suite. - Temporarily workaround the MyRocks bug where log_status query hangs if RocksDB file deletions were not disabled first. Note that this change breaks some of the clone-related group replication MTR tests. As they are not tested in fb-mysql and have been already broken, this patch makes no effort to fix them. MyRocks Clone (5/n): fix bug 109922 (SEs do not coordinate clone rollbacks on instance startup) If InnoDB clone decides to rollback during its startup, other storage engines are not aware of this and proceed as usual, resulting in inconsistent instance between storage engines. Ideally this should be fixed through a 2PC-like design where all the SEs vote to rollback the clone or to continue during the startup, but for MyRocks clone a simpler solution suffices: make InnoDB write a special marker file if it decides to rollback on clone startup. MyRocks Clone (6/n): on remote clone error, always send COM_ACK to InnoDB too In the case of multithreaded clone, the waits for the donor threads for the current clone stage to complete are implemented fully in InnoDB, even though that, and error handling, should be plugin-level features. With two storage engines participating in the clone this allows the situation where the other storage engine client reports an error with COM_ACK to one donor thread, while some other threads are in InnoDB clone state change waits. Those waits can abort if InnoDB clone gets an error from the client, but they know nothing about other SE errors, and will be aborted only by timeouts. As a minimal fix that does not involve pulling the whole error handling and state change waiting logic to the plugin layer, make the clone client always send an error-containing COM_ACK to InnoDB too, whenever it sends one to another storage engine. MyRocks Clone (7/n): MyRocks clone implementation - Cleanup any stale RocksDB checkpoint directories on server shutdown too. - For MyRocks clone network protocol, define four packet types (clone/common.h): - LOCATOR_V1: MyRocks-specific clone locator, consisting of a monotonically increasing clone session ID for the donor with an optional applier state (an array of file IDs and applied offsets) for restarts, or a zero in which case it is a version check empty locator used for handshakes. - FILE_NAME_MAP_V1: establishes a relation between a file ID, its name and target size. - FILE_CHUNK_V1: cloned file chunk data - ADD_ESTIMATE_V1: how many bytes should be added to the estimated clone data size. - All packet types share the common package header, which stores the version, the payload length without the header, and the packet type. - (clone/common.h): classes locator, file_metadata, chunk_metadata, & estimate_metadata correspond to the four packet types. They are subclassed on the donor side to add serialization routines and other donor-specific code and on the client side to add deserialization routines and other client-specific code. Re. serialization/deserialization, an exception is the locator, whose routines are implemented in the common code, because it may be passed both - ways. - (clone/common.h): Introduce a session class that handles the task ID set and errors, which is common for both donor and client. For errors, allow saving one error, with an optional associated path and errno, that can be set by any thread and later reported to the clone plugin. - (clone/common.cc) Setup MyRocks clone capabilities as HA_CLONE_HYBRID (not - applicable but matches InnoDB), HA_CLONE_MULTI_TASK (can clone with multiple - threads), & HA_CLONE_RESTART (can restart after a network I/O error). - (clone/common.cc) Enable O_DIRECT for clone file I/O if, on client if rocksdb_use_direct_io_for_flush_and_compaction is enabled, and on donor if rocksdb_use_direct_reads is enabled. - On the donor side (clone/donor.cc), encapsulate RocksDB checkpoints in a rdb_checkpoint class. Each checkpoint gets its own ID, which is also a donor session ID. The ID is constant for rolling checkpoints. - Subclass the common file_metadata to donor_file_metadata, chunk_metadata to donor_chunk_metadata, estimate_metadata to donor_estimate. - Implement donor going through a sequence of states: NONE, ROLLING, FINAL, FINAL_WITH_LOGS. To support multiple clone donor sessions, implement a manager class that looks up the donor instance by locator. - Implement MyRocks clone donor interface: - rocksdb_clone_start: handle the possible start modes: - HA_CLONE_MODE_START: create and register a new donor instance, register its master task ID. - HA_CLONE_MODE_ADD_TASK: lookup the existing donor instance, register a new task ID. - HA_CLONE_MODE_RESTART: lookup the existing donor instance, wait for the old master session to drop it by master task ID reference count, then process the locator applier state to reset the donor state. - rocksdb_clone_precopy: lookup the donor instance, create a RocksDB snapshot if state == NONE, otherwise send files by chunks and roll the checkpoint as needed. - rocksdb_clone_set_log_stop: add the WAL files to the clone file to copy set. - rocksdb_clone_copy: copy the clone files in any donor state. - rocksdb_clone_ack: acknowledge errors sent by client. - rocksdb_clone_end: for master task ID, stop accepting new worker join requests, wait until existing workers quit, and, in the case of restartable error, wait for any restarts to happen. Then decrement ref count for the master task ID (which will handle both the cases of restarted and not restarted) clone. For all other task IDs, simply unregister them. - On the client side (clone/client.cc), have file_in_progress class to track each individual file progress during application, and completed_file for fully-applied files. - Subclass the common file_metadata to client_file_metadata, implement its reading from the net buffer. Likewise for client_chunk_metadata subclassing chunk_metadata, and client_estimate for estimate_metadata. - TODO client class. - Implement MyRocks clone client interface: - rocksdb_clone_apply_begin: handle the possible start modes: - HA_CLONE_MODE_START: initialize the client instance for a new clone - HA_CLONE_MODE_ADD_TASK: register a new task ID for an ongoing clone - HA_CLONE_MODE_RESTART: restart the interrupted clone session - HA_CLONE_MODE_VERSION: prepare new version locator - rocksdb_clone_apply: if passed an error code, save it. Otherwise verify the passed buffer and dispatch ADD_ESTIMATE_V1, FILE_NAME_MAP_V1, & FILE_CHUNK_V1 packets accordingly. - rocksdb_clone_apply_end: for master task, wait for all other tasks to complete and finish the clone. For both master and non-master tasks unregister their IDs. - Add rocksdb_sys_vars tests for the new clone-related system variables. They are included in the current patch and not the MTR one in order not to break the existing rocksdb_sys_vars.all_vars test. - On instance startup, do clone fixup (clone/common.cc). It consists of: - Check whether InnoDB clone rollback marker is present. If yes, force rollback of MyRocks clone. - If the cloned temporary datadir contains clone-in-progress marker, assume half-completed clone and abort. - Move the cloned datadirs and WAL dirs in-place. For the datadir, do a directory-level rename with a save and later delete of the old datadir. For the WAL dir do not try to do parent directory-level renames because the mysqld process may not have permissions to delete that directory. Instead, rename all the old files individually, move new files in, delete the old files. - For all the rename operations, handle them returning EXDEV error (a rename would cross a filesystem boundary) by falling back to copy and delete. As we expect that it is the cloned RocksDB log files that are most affected by this, the total copy I/O wouldn't be large, and so the default MySQL copy routine my_copy is used, without extra copy throughput optimizations. Add a fault injection point for MTR to test this code paths. - Remove any RocksDB clone checkpoints found. - On instance startup, always move WALs to the target directory if they are found in the rocksdb_datadir yet the rocksdb_wal_dir is different. This handles the case of setting rocksdb_wal_dir while instance is shutdown, both for clone fixup and regular operation. - Add PFS instrumentation for all the instrumentable entities in this patch (mutexes, rwlocks, file I/O). MyRocks Clone (8/n): MyRocks clone MTR tests - Add a new MTR test suite rocksdb_clone - Support new variable MTR clone/include/clone_command.inc: clone_single_threaded, setting which forces single-threaded clone operation. - Reuse a big part of InnoDB clone tests by including their test files, with the exception of tests involving DDL, as concurrent DDL is not supported by MyRocks clone at the moment, and debug tests that adjust cloned instance LSN, because they desynchronize storage engines. The reused tests are: local_basic, local_boundary, local_dml, local_dml_auto_tune, local_dml_no_redo, local_insert, local_stage, monitor_progress, remote_basic, remote_basic_replace, remote_boundary, remote_dml, remote_dml_auto_tune, remote_dml_gtid, remote_dml_no_binlog, remote_dml_no_ssl, remote_dml_recovery, remote_dml_replace, remote_dml_ssl, remote_dml_ssl_certificate, remote_dml_upgrade, remote_insert, remote_monitor_progress, remote_restart, remote_skip_validation, remote_stage, remote_stage_restart. - As a part of the above, add MyRocks support to local_basic and monitor_progress MTR tests. - Besides the reused InnoDB clone tests, add some new MyRocks-specific tests: - local_minimal_1_thread, which is the simplest possible local clone test. - local_minimal_multithreaded, the simplest possible but multithreaded local clone test. - local_minimal_direct_io, which is the simplest possible local clone test in all combinations of rocksdb_use_direct_reads and rocksdb_use_direct_io_for_flush_and_compaction settings, which are the O_DIRECT-enabling options. - local_dml_o_direct_donor, local_dml_o_direct_client, local_dml_o_direct_both that reuse InnoDB local_dml test, but combined with either or both of the O_DIRECT-enabling options. - local_minimal_wal_dir, which clones with various rocksdb_wal_dir settings on donor and client. - local_minimal_wal_dir_copy_delete, which uses a debug injection to test the move-across-filesystem-boundary code path in clone startup. - remote_minimal_direct_io_{1, 2, 3, 4} tests for all the combinations of O_DIRECT-enabling options for donor and client instances. - remote_fail_apply, which uses a debug injection to fail the clone client during remote clone. - remote_inplace_restart, which performs a remote in-place clone with a simulated transient error on the client. - remote_kill_donor, which kill the donor instance at various points of remote cloning. - rolling_checkpoint for testing various combinations of clone rolling checkpoint tuning options. - three_se_basic to test a simple clone across InnoDB, MyRocks, & Blackhole engines. Fix moving RocksDB WALs on startup across file system boundaries Reviewed By: sunshine-Chun Differential Revision: D43160162 Pulled By: hermanlee fbshipit-source-id: 5358a57 -------------------------------------------------------------------------------- Rdb clone fixes (facebook#1271) Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 2245706 -------------------------------------------------------------------------------- Clone: fix race condition that may skip some SST files (facebook#1274) Summary: A donor thread in the main copy loop took a file to send, released the donor state mutex, and then opened the file. If an ENOENT was received at this point, it was assumed that this was a stale SST file from an older checkpoint that was rolled since. Because the donor state mutex was released between taking of the file and opening it, the following race was possible: 1) Thread 1 takes the file 2) Thread 2 decides to roll the checkpoint, the old checkpoint is deleted 3) Thread 1 tries to open the file, gets ENOENT 4) Thread 2 creates the new checkpoint, the file re-appears, but it's too late. Rolling the checkpoint in a donor state mutex critical section is a possible fix, but such section would do a lot of I/O, serializing the parallel threads. Instead, fix by taking the file and opening it in the same critical section. Pull Request resolved: facebook#1274 Reviewed By: sunshine-Chun Differential Revision: D43629546 Pulled By: hermanlee fbshipit-source-id: 6e7f315
Summary: Pull Request resolved: facebook#1250 MyRocks Clone (1/n): make InnoDB clone MTR tests SE-agnostic The MyRocks clone MTR tests will reuse the InnoDB tests by including them. Adjust the tests so that they work this way, i.e. make them storage engine-agnostic by checking the default_storage_engine system variable and guarding InnoDB-specific tests this way. This includes the tests that artificially desync the InnoDB redo log position. MyRocks Clone (2/n): fix bug 109919 (Clone plugin does not support arbitrary length O_DIRECT files) The existing clone plugin I/O routines handle O_DIRECT files with the assumption that their sizes are always a multiple of a block size. This is too restrictive because in general O_DIRECT files can be of any size and MyRocks files actually are. Add a flag to Ha_clone_file to support arbitrary file lengths under O_DIRECT: o_direct_uneven_file_size. Pass it to clone_client.h/cc Client::get_aligned_buffer to allocate the buffer that bumps the length to the next CLONE_OS_ALIGN multiple if it's not already one. Introduce a pad_for_o_direct helper that performs this calculation for the passed length as needed. For writes, if O_DIRECT is enabled and the buffer length is not a multiple of CLONE_OS_ALIGN, pad it with zeros to the next multiple. Then check the actual bytes written, and if any padding bytes were actually written, call my_chsize to truncate the file to the needed size. MyRocks Clone (3/n): fix bug 109920 (SEs cannot add to clone data size estimates) In the current implementation, once the first SE sets the data size estimate for a particular stage, there is no way for other SEs to add to it. Address this by adding Ha_clone_cbk::add_to_data_size_estimate method. MyRocks Clone (4/n): fix bug 109921 (Cannot clone with synchronization across SEs) The existing clone implementation synchronizes InnoDB and binlog directly. There is no way to make other storage engines join this synchronization, thus replace it with synchronizing by a performance_schema.log_status query. - Add table_log_status::get_row method to allow doing the log_status query from inside the server. - Handlerton: introduce two new methods: clone_precopy - currently unused, which will handle any non-consistent-snapshot copying before the main consistent copy, and set_log_stop - which provides the SE the clone stop position information. - Plugin: introduce Ha_clone_common_cbk, which derives from Ha_clone_cbk, and from which derive all the concrete classes: Client_Cbk, Local_Cbk & Server_Cbk. - Ha_clone_common_cbk contains two methods: precopy and synchronize_engines. The former calls precopy for all the participating clone SE handlertons, if implemented. The latter, only to be called from the main clone thread, performs the performance_schema.LOG_STATUS query, splits its result per-SE and calls handlerton set_log_stop for all SEs. - The pfs.log_status query still must be called by InnoDB at a certain point in clone. That means that InnoDB is still a special SE in that it must always come first in the clone locator vectors. This will also handle donor waits on clone restarts, as that is implemented in InnoDB-specific code. To ensure this, add a helper clone_hton.cc:make_innodb_first which shuffles the vectors as needed. Call it in all the clone-starting code paths, add asserts that InnoDB comes first. - Since hton_clone_copy will call into InnoDB clone copy, which at certain point will synchronize engines, calling into the plugin again, and the calling into SEs again, introduce a backup locator vector Ha_clone_cbk::all_locators, and its setters/getters set_all_locators, reset_all_locators, get_all_locators. Use them in the re-entrant hton_clone_copy code. - Introduce a new InnoDB clone state named SST COPY, which calls precopy and has zero files and chunks set to copy in the regular InnoDB clone way, i.e. all it does it the precopy. - The redo log archiving keeps running throughout the precopy stage. - On stage transition from the precopy to the redo log copy. call synchronize. - Implement innodb_clone_set_log_stop by parsing out the desired clone stop LSN from the passed JSON from the log_status query. - Make InnoDB redo log archiving stop at that LSN instead of on reaching the end of redo log. - Support stopping redo log archiving in a middle of a log block. As the old implementation may encounter this case only at the end of the redo log, it took that block from the log buffer in memory. This no longer works with the new implementation, where the needed block might reside on disk only. Address this by forcing the whole block containing the target LSN to disk, reading it from there, and patching its memory copy to stop at the required LSN before giving it for the clone to copy. - To implement redo log block reading on a running server, adapt crash recovery function recv_read_log_seg to work both for crash recovery and on a running server. This is similar to what Percona Server does for its changed page tracking feature. - Make sure not to archive not-yet-written log: pfs.log_status query gets the log system LSN, which is the value of highest log reservation. Thus in an archiving iteration, archive up to the lower values of log archiving LSN, and write_lsn. Set the log archiving LSN to LSN_MAX at clone initialization, so that clone archiving may proceed until the stop LSN is set. - The above implementation results in cloned LSN != log end LSN for the cloned instance, thus adjust crash recovery to work with that. The existing crash recovery throws away any crash-recovered metadata and reads its snapshot from the data files, whereas the snapshot was not written in any consistency with the clone LSN. Fix by no longer throwing away the crash-recovered InnoDB dynamic metadata and by not re-reading the the last metadata snapshot. This forces also not to re-serialize the metadata in the crash recovery as that writes redo log which is to be avoided during clone crash recovery. - Additionally, at the start of the redo log copy the old implementation requested all InnoDB updates to be flushed to disk immediately, to avoid clone recovery generating new redo log. This is also disabled as it becomes redundant with the fix. This issue is similar to Percona XtraBackup https://jira.percona.com/browse/PXB-1634 - Disable clone.local_xa test because the new synchronization no longer blocks on XA transactions. - Re-record two clone-related tests in the component_keyring_file MTR suite. - Temporarily workaround the MyRocks bug where log_status query hangs if RocksDB file deletions were not disabled first. Note that this change breaks some of the clone-related group replication MTR tests. As they are not tested in fb-mysql and have been already broken, this patch makes no effort to fix them. MyRocks Clone (5/n): fix bug 109922 (SEs do not coordinate clone rollbacks on instance startup) If InnoDB clone decides to rollback during its startup, other storage engines are not aware of this and proceed as usual, resulting in inconsistent instance between storage engines. Ideally this should be fixed through a 2PC-like design where all the SEs vote to rollback the clone or to continue during the startup, but for MyRocks clone a simpler solution suffices: make InnoDB write a special marker file if it decides to rollback on clone startup. MyRocks Clone (6/n): on remote clone error, always send COM_ACK to InnoDB too In the case of multithreaded clone, the waits for the donor threads for the current clone stage to complete are implemented fully in InnoDB, even though that, and error handling, should be plugin-level features. With two storage engines participating in the clone this allows the situation where the other storage engine client reports an error with COM_ACK to one donor thread, while some other threads are in InnoDB clone state change waits. Those waits can abort if InnoDB clone gets an error from the client, but they know nothing about other SE errors, and will be aborted only by timeouts. As a minimal fix that does not involve pulling the whole error handling and state change waiting logic to the plugin layer, make the clone client always send an error-containing COM_ACK to InnoDB too, whenever it sends one to another storage engine. MyRocks Clone (7/n): MyRocks clone implementation - Cleanup any stale RocksDB checkpoint directories on server shutdown too. - For MyRocks clone network protocol, define four packet types (clone/common.h): - LOCATOR_V1: MyRocks-specific clone locator, consisting of a monotonically increasing clone session ID for the donor with an optional applier state (an array of file IDs and applied offsets) for restarts, or a zero in which case it is a version check empty locator used for handshakes. - FILE_NAME_MAP_V1: establishes a relation between a file ID, its name and target size. - FILE_CHUNK_V1: cloned file chunk data - ADD_ESTIMATE_V1: how many bytes should be added to the estimated clone data size. - All packet types share the common package header, which stores the version, the payload length without the header, and the packet type. - (clone/common.h): classes locator, file_metadata, chunk_metadata, & estimate_metadata correspond to the four packet types. They are subclassed on the donor side to add serialization routines and other donor-specific code and on the client side to add deserialization routines and other client-specific code. Re. serialization/deserialization, an exception is the locator, whose routines are implemented in the common code, because it may be passed both - ways. - (clone/common.h): Introduce a session class that handles the task ID set and errors, which is common for both donor and client. For errors, allow saving one error, with an optional associated path and errno, that can be set by any thread and later reported to the clone plugin. - (clone/common.cc) Setup MyRocks clone capabilities as HA_CLONE_HYBRID (not - applicable but matches InnoDB), HA_CLONE_MULTI_TASK (can clone with multiple - threads), & HA_CLONE_RESTART (can restart after a network I/O error). - (clone/common.cc) Enable O_DIRECT for clone file I/O if, on client if rocksdb_use_direct_io_for_flush_and_compaction is enabled, and on donor if rocksdb_use_direct_reads is enabled. - On the donor side (clone/donor.cc), encapsulate RocksDB checkpoints in a rdb_checkpoint class. Each checkpoint gets its own ID, which is also a donor session ID. The ID is constant for rolling checkpoints. - Subclass the common file_metadata to donor_file_metadata, chunk_metadata to donor_chunk_metadata, estimate_metadata to donor_estimate. - Implement donor going through a sequence of states: NONE, ROLLING, FINAL, FINAL_WITH_LOGS. To support multiple clone donor sessions, implement a manager class that looks up the donor instance by locator. - Implement MyRocks clone donor interface: - rocksdb_clone_start: handle the possible start modes: - HA_CLONE_MODE_START: create and register a new donor instance, register its master task ID. - HA_CLONE_MODE_ADD_TASK: lookup the existing donor instance, register a new task ID. - HA_CLONE_MODE_RESTART: lookup the existing donor instance, wait for the old master session to drop it by master task ID reference count, then process the locator applier state to reset the donor state. - rocksdb_clone_precopy: lookup the donor instance, create a RocksDB snapshot if state == NONE, otherwise send files by chunks and roll the checkpoint as needed. - rocksdb_clone_set_log_stop: add the WAL files to the clone file to copy set. - rocksdb_clone_copy: copy the clone files in any donor state. - rocksdb_clone_ack: acknowledge errors sent by client. - rocksdb_clone_end: for master task ID, stop accepting new worker join requests, wait until existing workers quit, and, in the case of restartable error, wait for any restarts to happen. Then decrement ref count for the master task ID (which will handle both the cases of restarted and not restarted) clone. For all other task IDs, simply unregister them. - On the client side (clone/client.cc), have file_in_progress class to track each individual file progress during application, and completed_file for fully-applied files. - Subclass the common file_metadata to client_file_metadata, implement its reading from the net buffer. Likewise for client_chunk_metadata subclassing chunk_metadata, and client_estimate for estimate_metadata. - TODO client class. - Implement MyRocks clone client interface: - rocksdb_clone_apply_begin: handle the possible start modes: - HA_CLONE_MODE_START: initialize the client instance for a new clone - HA_CLONE_MODE_ADD_TASK: register a new task ID for an ongoing clone - HA_CLONE_MODE_RESTART: restart the interrupted clone session - HA_CLONE_MODE_VERSION: prepare new version locator - rocksdb_clone_apply: if passed an error code, save it. Otherwise verify the passed buffer and dispatch ADD_ESTIMATE_V1, FILE_NAME_MAP_V1, & FILE_CHUNK_V1 packets accordingly. - rocksdb_clone_apply_end: for master task, wait for all other tasks to complete and finish the clone. For both master and non-master tasks unregister their IDs. - Add rocksdb_sys_vars tests for the new clone-related system variables. They are included in the current patch and not the MTR one in order not to break the existing rocksdb_sys_vars.all_vars test. - On instance startup, do clone fixup (clone/common.cc). It consists of: - Check whether InnoDB clone rollback marker is present. If yes, force rollback of MyRocks clone. - If the cloned temporary datadir contains clone-in-progress marker, assume half-completed clone and abort. - Move the cloned datadirs and WAL dirs in-place. For the datadir, do a directory-level rename with a save and later delete of the old datadir. For the WAL dir do not try to do parent directory-level renames because the mysqld process may not have permissions to delete that directory. Instead, rename all the old files individually, move new files in, delete the old files. - For all the rename operations, handle them returning EXDEV error (a rename would cross a filesystem boundary) by falling back to copy and delete. As we expect that it is the cloned RocksDB log files that are most affected by this, the total copy I/O wouldn't be large, and so the default MySQL copy routine my_copy is used, without extra copy throughput optimizations. Add a fault injection point for MTR to test this code paths. - Remove any RocksDB clone checkpoints found. - On instance startup, always move WALs to the target directory if they are found in the rocksdb_datadir yet the rocksdb_wal_dir is different. This handles the case of setting rocksdb_wal_dir while instance is shutdown, both for clone fixup and regular operation. - Add PFS instrumentation for all the instrumentable entities in this patch (mutexes, rwlocks, file I/O). MyRocks Clone (8/n): MyRocks clone MTR tests - Add a new MTR test suite rocksdb_clone - Support new variable MTR clone/include/clone_command.inc: clone_single_threaded, setting which forces single-threaded clone operation. - Reuse a big part of InnoDB clone tests by including their test files, with the exception of tests involving DDL, as concurrent DDL is not supported by MyRocks clone at the moment, and debug tests that adjust cloned instance LSN, because they desynchronize storage engines. The reused tests are: local_basic, local_boundary, local_dml, local_dml_auto_tune, local_dml_no_redo, local_insert, local_stage, monitor_progress, remote_basic, remote_basic_replace, remote_boundary, remote_dml, remote_dml_auto_tune, remote_dml_gtid, remote_dml_no_binlog, remote_dml_no_ssl, remote_dml_recovery, remote_dml_replace, remote_dml_ssl, remote_dml_ssl_certificate, remote_dml_upgrade, remote_insert, remote_monitor_progress, remote_restart, remote_skip_validation, remote_stage, remote_stage_restart. - As a part of the above, add MyRocks support to local_basic and monitor_progress MTR tests. - Besides the reused InnoDB clone tests, add some new MyRocks-specific tests: - local_minimal_1_thread, which is the simplest possible local clone test. - local_minimal_multithreaded, the simplest possible but multithreaded local clone test. - local_minimal_direct_io, which is the simplest possible local clone test in all combinations of rocksdb_use_direct_reads and rocksdb_use_direct_io_for_flush_and_compaction settings, which are the O_DIRECT-enabling options. - local_dml_o_direct_donor, local_dml_o_direct_client, local_dml_o_direct_both that reuse InnoDB local_dml test, but combined with either or both of the O_DIRECT-enabling options. - local_minimal_wal_dir, which clones with various rocksdb_wal_dir settings on donor and client. - local_minimal_wal_dir_copy_delete, which uses a debug injection to test the move-across-filesystem-boundary code path in clone startup. - remote_minimal_direct_io_{1, 2, 3, 4} tests for all the combinations of O_DIRECT-enabling options for donor and client instances. - remote_fail_apply, which uses a debug injection to fail the clone client during remote clone. - remote_inplace_restart, which performs a remote in-place clone with a simulated transient error on the client. - remote_kill_donor, which kill the donor instance at various points of remote cloning. - rolling_checkpoint for testing various combinations of clone rolling checkpoint tuning options. - three_se_basic to test a simple clone across InnoDB, MyRocks, & Blackhole engines. Fix moving RocksDB WALs on startup across file system boundaries Reviewed By: sunshine-Chun Differential Revision: D43160162 Pulled By: hermanlee fbshipit-source-id: 5358a57 -------------------------------------------------------------------------------- Rdb clone fixes (facebook#1271) Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 2245706 -------------------------------------------------------------------------------- Clone: fix race condition that may skip some SST files (facebook#1274) Summary: A donor thread in the main copy loop took a file to send, released the donor state mutex, and then opened the file. If an ENOENT was received at this point, it was assumed that this was a stale SST file from an older checkpoint that was rolled since. Because the donor state mutex was released between taking of the file and opening it, the following race was possible: 1) Thread 1 takes the file 2) Thread 2 decides to roll the checkpoint, the old checkpoint is deleted 3) Thread 1 tries to open the file, gets ENOENT 4) Thread 2 creates the new checkpoint, the file re-appears, but it's too late. Rolling the checkpoint in a donor state mutex critical section is a possible fix, but such section would do a lot of I/O, serializing the parallel threads. Instead, fix by taking the file and opening it in the same critical section. Pull Request resolved: facebook#1274 Reviewed By: sunshine-Chun Differential Revision: D43629546 Pulled By: hermanlee fbshipit-source-id: 6e7f315
Summary: Pull Request resolved: facebook#1250 MyRocks Clone (1/n): make InnoDB clone MTR tests SE-agnostic The MyRocks clone MTR tests will reuse the InnoDB tests by including them. Adjust the tests so that they work this way, i.e. make them storage engine-agnostic by checking the default_storage_engine system variable and guarding InnoDB-specific tests this way. This includes the tests that artificially desync the InnoDB redo log position. MyRocks Clone (2/n): fix bug 109919 (Clone plugin does not support arbitrary length O_DIRECT files) The existing clone plugin I/O routines handle O_DIRECT files with the assumption that their sizes are always a multiple of a block size. This is too restrictive because in general O_DIRECT files can be of any size and MyRocks files actually are. Add a flag to Ha_clone_file to support arbitrary file lengths under O_DIRECT: o_direct_uneven_file_size. Pass it to clone_client.h/cc Client::get_aligned_buffer to allocate the buffer that bumps the length to the next CLONE_OS_ALIGN multiple if it's not already one. Introduce a pad_for_o_direct helper that performs this calculation for the passed length as needed. For writes, if O_DIRECT is enabled and the buffer length is not a multiple of CLONE_OS_ALIGN, pad it with zeros to the next multiple. Then check the actual bytes written, and if any padding bytes were actually written, call my_chsize to truncate the file to the needed size. MyRocks Clone (3/n): fix bug 109920 (SEs cannot add to clone data size estimates) In the current implementation, once the first SE sets the data size estimate for a particular stage, there is no way for other SEs to add to it. Address this by adding Ha_clone_cbk::add_to_data_size_estimate method. MyRocks Clone (4/n): fix bug 109921 (Cannot clone with synchronization across SEs) The existing clone implementation synchronizes InnoDB and binlog directly. There is no way to make other storage engines join this synchronization, thus replace it with synchronizing by a performance_schema.log_status query. - Add table_log_status::get_row method to allow doing the log_status query from inside the server. - Handlerton: introduce two new methods: clone_precopy - currently unused, which will handle any non-consistent-snapshot copying before the main consistent copy, and set_log_stop - which provides the SE the clone stop position information. - Plugin: introduce Ha_clone_common_cbk, which derives from Ha_clone_cbk, and from which derive all the concrete classes: Client_Cbk, Local_Cbk & Server_Cbk. - Ha_clone_common_cbk contains two methods: precopy and synchronize_engines. The former calls precopy for all the participating clone SE handlertons, if implemented. The latter, only to be called from the main clone thread, performs the performance_schema.LOG_STATUS query, splits its result per-SE and calls handlerton set_log_stop for all SEs. - The pfs.log_status query still must be called by InnoDB at a certain point in clone. That means that InnoDB is still a special SE in that it must always come first in the clone locator vectors. This will also handle donor waits on clone restarts, as that is implemented in InnoDB-specific code. To ensure this, add a helper clone_hton.cc:make_innodb_first which shuffles the vectors as needed. Call it in all the clone-starting code paths, add asserts that InnoDB comes first. - Since hton_clone_copy will call into InnoDB clone copy, which at certain point will synchronize engines, calling into the plugin again, and the calling into SEs again, introduce a backup locator vector Ha_clone_cbk::all_locators, and its setters/getters set_all_locators, reset_all_locators, get_all_locators. Use them in the re-entrant hton_clone_copy code. - Introduce a new InnoDB clone state named SST COPY, which calls precopy and has zero files and chunks set to copy in the regular InnoDB clone way, i.e. all it does it the precopy. - The redo log archiving keeps running throughout the precopy stage. - On stage transition from the precopy to the redo log copy. call synchronize. - Implement innodb_clone_set_log_stop by parsing out the desired clone stop LSN from the passed JSON from the log_status query. - Make InnoDB redo log archiving stop at that LSN instead of on reaching the end of redo log. - Support stopping redo log archiving in a middle of a log block. As the old implementation may encounter this case only at the end of the redo log, it took that block from the log buffer in memory. This no longer works with the new implementation, where the needed block might reside on disk only. Address this by forcing the whole block containing the target LSN to disk, reading it from there, and patching its memory copy to stop at the required LSN before giving it for the clone to copy. - To implement redo log block reading on a running server, adapt crash recovery function recv_read_log_seg to work both for crash recovery and on a running server. This is similar to what Percona Server does for its changed page tracking feature. - Make sure not to archive not-yet-written log: pfs.log_status query gets the log system LSN, which is the value of highest log reservation. Thus in an archiving iteration, archive up to the lower values of log archiving LSN, and write_lsn. Set the log archiving LSN to LSN_MAX at clone initialization, so that clone archiving may proceed until the stop LSN is set. - The above implementation results in cloned LSN != log end LSN for the cloned instance, thus adjust crash recovery to work with that. The existing crash recovery throws away any crash-recovered metadata and reads its snapshot from the data files, whereas the snapshot was not written in any consistency with the clone LSN. Fix by no longer throwing away the crash-recovered InnoDB dynamic metadata and by not re-reading the the last metadata snapshot. This forces also not to re-serialize the metadata in the crash recovery as that writes redo log which is to be avoided during clone crash recovery. - Additionally, at the start of the redo log copy the old implementation requested all InnoDB updates to be flushed to disk immediately, to avoid clone recovery generating new redo log. This is also disabled as it becomes redundant with the fix. This issue is similar to Percona XtraBackup https://jira.percona.com/browse/PXB-1634 - Disable clone.local_xa test because the new synchronization no longer blocks on XA transactions. - Re-record two clone-related tests in the component_keyring_file MTR suite. - Temporarily workaround the MyRocks bug where log_status query hangs if RocksDB file deletions were not disabled first. Note that this change breaks some of the clone-related group replication MTR tests. As they are not tested in fb-mysql and have been already broken, this patch makes no effort to fix them. MyRocks Clone (5/n): fix bug 109922 (SEs do not coordinate clone rollbacks on instance startup) If InnoDB clone decides to rollback during its startup, other storage engines are not aware of this and proceed as usual, resulting in inconsistent instance between storage engines. Ideally this should be fixed through a 2PC-like design where all the SEs vote to rollback the clone or to continue during the startup, but for MyRocks clone a simpler solution suffices: make InnoDB write a special marker file if it decides to rollback on clone startup. MyRocks Clone (6/n): on remote clone error, always send COM_ACK to InnoDB too In the case of multithreaded clone, the waits for the donor threads for the current clone stage to complete are implemented fully in InnoDB, even though that, and error handling, should be plugin-level features. With two storage engines participating in the clone this allows the situation where the other storage engine client reports an error with COM_ACK to one donor thread, while some other threads are in InnoDB clone state change waits. Those waits can abort if InnoDB clone gets an error from the client, but they know nothing about other SE errors, and will be aborted only by timeouts. As a minimal fix that does not involve pulling the whole error handling and state change waiting logic to the plugin layer, make the clone client always send an error-containing COM_ACK to InnoDB too, whenever it sends one to another storage engine. MyRocks Clone (7/n): MyRocks clone implementation - Cleanup any stale RocksDB checkpoint directories on server shutdown too. - For MyRocks clone network protocol, define four packet types (clone/common.h): - LOCATOR_V1: MyRocks-specific clone locator, consisting of a monotonically increasing clone session ID for the donor with an optional applier state (an array of file IDs and applied offsets) for restarts, or a zero in which case it is a version check empty locator used for handshakes. - FILE_NAME_MAP_V1: establishes a relation between a file ID, its name and target size. - FILE_CHUNK_V1: cloned file chunk data - ADD_ESTIMATE_V1: how many bytes should be added to the estimated clone data size. - All packet types share the common package header, which stores the version, the payload length without the header, and the packet type. - (clone/common.h): classes locator, file_metadata, chunk_metadata, & estimate_metadata correspond to the four packet types. They are subclassed on the donor side to add serialization routines and other donor-specific code and on the client side to add deserialization routines and other client-specific code. Re. serialization/deserialization, an exception is the locator, whose routines are implemented in the common code, because it may be passed both - ways. - (clone/common.h): Introduce a session class that handles the task ID set and errors, which is common for both donor and client. For errors, allow saving one error, with an optional associated path and errno, that can be set by any thread and later reported to the clone plugin. - (clone/common.cc) Setup MyRocks clone capabilities as HA_CLONE_HYBRID (not - applicable but matches InnoDB), HA_CLONE_MULTI_TASK (can clone with multiple - threads), & HA_CLONE_RESTART (can restart after a network I/O error). - (clone/common.cc) Enable O_DIRECT for clone file I/O if, on client if rocksdb_use_direct_io_for_flush_and_compaction is enabled, and on donor if rocksdb_use_direct_reads is enabled. - On the donor side (clone/donor.cc), encapsulate RocksDB checkpoints in a rdb_checkpoint class. Each checkpoint gets its own ID, which is also a donor session ID. The ID is constant for rolling checkpoints. - Subclass the common file_metadata to donor_file_metadata, chunk_metadata to donor_chunk_metadata, estimate_metadata to donor_estimate. - Implement donor going through a sequence of states: NONE, ROLLING, FINAL, FINAL_WITH_LOGS. To support multiple clone donor sessions, implement a manager class that looks up the donor instance by locator. - Implement MyRocks clone donor interface: - rocksdb_clone_start: handle the possible start modes: - HA_CLONE_MODE_START: create and register a new donor instance, register its master task ID. - HA_CLONE_MODE_ADD_TASK: lookup the existing donor instance, register a new task ID. - HA_CLONE_MODE_RESTART: lookup the existing donor instance, wait for the old master session to drop it by master task ID reference count, then process the locator applier state to reset the donor state. - rocksdb_clone_precopy: lookup the donor instance, create a RocksDB snapshot if state == NONE, otherwise send files by chunks and roll the checkpoint as needed. - rocksdb_clone_set_log_stop: add the WAL files to the clone file to copy set. - rocksdb_clone_copy: copy the clone files in any donor state. - rocksdb_clone_ack: acknowledge errors sent by client. - rocksdb_clone_end: for master task ID, stop accepting new worker join requests, wait until existing workers quit, and, in the case of restartable error, wait for any restarts to happen. Then decrement ref count for the master task ID (which will handle both the cases of restarted and not restarted) clone. For all other task IDs, simply unregister them. - On the client side (clone/client.cc), have file_in_progress class to track each individual file progress during application, and completed_file for fully-applied files. - Subclass the common file_metadata to client_file_metadata, implement its reading from the net buffer. Likewise for client_chunk_metadata subclassing chunk_metadata, and client_estimate for estimate_metadata. - TODO client class. - Implement MyRocks clone client interface: - rocksdb_clone_apply_begin: handle the possible start modes: - HA_CLONE_MODE_START: initialize the client instance for a new clone - HA_CLONE_MODE_ADD_TASK: register a new task ID for an ongoing clone - HA_CLONE_MODE_RESTART: restart the interrupted clone session - HA_CLONE_MODE_VERSION: prepare new version locator - rocksdb_clone_apply: if passed an error code, save it. Otherwise verify the passed buffer and dispatch ADD_ESTIMATE_V1, FILE_NAME_MAP_V1, & FILE_CHUNK_V1 packets accordingly. - rocksdb_clone_apply_end: for master task, wait for all other tasks to complete and finish the clone. For both master and non-master tasks unregister their IDs. - Add rocksdb_sys_vars tests for the new clone-related system variables. They are included in the current patch and not the MTR one in order not to break the existing rocksdb_sys_vars.all_vars test. - On instance startup, do clone fixup (clone/common.cc). It consists of: - Check whether InnoDB clone rollback marker is present. If yes, force rollback of MyRocks clone. - If the cloned temporary datadir contains clone-in-progress marker, assume half-completed clone and abort. - Move the cloned datadirs and WAL dirs in-place. For the datadir, do a directory-level rename with a save and later delete of the old datadir. For the WAL dir do not try to do parent directory-level renames because the mysqld process may not have permissions to delete that directory. Instead, rename all the old files individually, move new files in, delete the old files. - For all the rename operations, handle them returning EXDEV error (a rename would cross a filesystem boundary) by falling back to copy and delete. As we expect that it is the cloned RocksDB log files that are most affected by this, the total copy I/O wouldn't be large, and so the default MySQL copy routine my_copy is used, without extra copy throughput optimizations. Add a fault injection point for MTR to test this code paths. - Remove any RocksDB clone checkpoints found. - On instance startup, always move WALs to the target directory if they are found in the rocksdb_datadir yet the rocksdb_wal_dir is different. This handles the case of setting rocksdb_wal_dir while instance is shutdown, both for clone fixup and regular operation. - Add PFS instrumentation for all the instrumentable entities in this patch (mutexes, rwlocks, file I/O). MyRocks Clone (8/n): MyRocks clone MTR tests - Add a new MTR test suite rocksdb_clone - Support new variable MTR clone/include/clone_command.inc: clone_single_threaded, setting which forces single-threaded clone operation. - Reuse a big part of InnoDB clone tests by including their test files, with the exception of tests involving DDL, as concurrent DDL is not supported by MyRocks clone at the moment, and debug tests that adjust cloned instance LSN, because they desynchronize storage engines. The reused tests are: local_basic, local_boundary, local_dml, local_dml_auto_tune, local_dml_no_redo, local_insert, local_stage, monitor_progress, remote_basic, remote_basic_replace, remote_boundary, remote_dml, remote_dml_auto_tune, remote_dml_gtid, remote_dml_no_binlog, remote_dml_no_ssl, remote_dml_recovery, remote_dml_replace, remote_dml_ssl, remote_dml_ssl_certificate, remote_dml_upgrade, remote_insert, remote_monitor_progress, remote_restart, remote_skip_validation, remote_stage, remote_stage_restart. - As a part of the above, add MyRocks support to local_basic and monitor_progress MTR tests. - Besides the reused InnoDB clone tests, add some new MyRocks-specific tests: - local_minimal_1_thread, which is the simplest possible local clone test. - local_minimal_multithreaded, the simplest possible but multithreaded local clone test. - local_minimal_direct_io, which is the simplest possible local clone test in all combinations of rocksdb_use_direct_reads and rocksdb_use_direct_io_for_flush_and_compaction settings, which are the O_DIRECT-enabling options. - local_dml_o_direct_donor, local_dml_o_direct_client, local_dml_o_direct_both that reuse InnoDB local_dml test, but combined with either or both of the O_DIRECT-enabling options. - local_minimal_wal_dir, which clones with various rocksdb_wal_dir settings on donor and client. - local_minimal_wal_dir_copy_delete, which uses a debug injection to test the move-across-filesystem-boundary code path in clone startup. - remote_minimal_direct_io_{1, 2, 3, 4} tests for all the combinations of O_DIRECT-enabling options for donor and client instances. - remote_fail_apply, which uses a debug injection to fail the clone client during remote clone. - remote_inplace_restart, which performs a remote in-place clone with a simulated transient error on the client. - remote_kill_donor, which kill the donor instance at various points of remote cloning. - rolling_checkpoint for testing various combinations of clone rolling checkpoint tuning options. - three_se_basic to test a simple clone across InnoDB, MyRocks, & Blackhole engines. Fix moving RocksDB WALs on startup across file system boundaries Reviewed By: sunshine-Chun Differential Revision: D43160162 Pulled By: hermanlee fbshipit-source-id: 5358a57 -------------------------------------------------------------------------------- Rdb clone fixes (facebook#1271) Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 2245706 -------------------------------------------------------------------------------- Clone: fix race condition that may skip some SST files (facebook#1274) Summary: A donor thread in the main copy loop took a file to send, released the donor state mutex, and then opened the file. If an ENOENT was received at this point, it was assumed that this was a stale SST file from an older checkpoint that was rolled since. Because the donor state mutex was released between taking of the file and opening it, the following race was possible: 1) Thread 1 takes the file 2) Thread 2 decides to roll the checkpoint, the old checkpoint is deleted 3) Thread 1 tries to open the file, gets ENOENT 4) Thread 2 creates the new checkpoint, the file re-appears, but it's too late. Rolling the checkpoint in a donor state mutex critical section is a possible fix, but such section would do a lot of I/O, serializing the parallel threads. Instead, fix by taking the file and opening it in the same critical section. Pull Request resolved: facebook#1274 Reviewed By: sunshine-Chun Differential Revision: D43629546 Pulled By: hermanlee fbshipit-source-id: 6e7f315
Summary: Pull Request resolved: facebook#1250 MyRocks Clone (1/n): make InnoDB clone MTR tests SE-agnostic The MyRocks clone MTR tests will reuse the InnoDB tests by including them. Adjust the tests so that they work this way, i.e. make them storage engine-agnostic by checking the default_storage_engine system variable and guarding InnoDB-specific tests this way. This includes the tests that artificially desync the InnoDB redo log position. MyRocks Clone (2/n): fix bug 109919 (Clone plugin does not support arbitrary length O_DIRECT files) The existing clone plugin I/O routines handle O_DIRECT files with the assumption that their sizes are always a multiple of a block size. This is too restrictive because in general O_DIRECT files can be of any size and MyRocks files actually are. Add a flag to Ha_clone_file to support arbitrary file lengths under O_DIRECT: o_direct_uneven_file_size. Pass it to clone_client.h/cc Client::get_aligned_buffer to allocate the buffer that bumps the length to the next CLONE_OS_ALIGN multiple if it's not already one. Introduce a pad_for_o_direct helper that performs this calculation for the passed length as needed. For writes, if O_DIRECT is enabled and the buffer length is not a multiple of CLONE_OS_ALIGN, pad it with zeros to the next multiple. Then check the actual bytes written, and if any padding bytes were actually written, call my_chsize to truncate the file to the needed size. MyRocks Clone (3/n): fix bug 109920 (SEs cannot add to clone data size estimates) In the current implementation, once the first SE sets the data size estimate for a particular stage, there is no way for other SEs to add to it. Address this by adding Ha_clone_cbk::add_to_data_size_estimate method. MyRocks Clone (4/n): fix bug 109921 (Cannot clone with synchronization across SEs) The existing clone implementation synchronizes InnoDB and binlog directly. There is no way to make other storage engines join this synchronization, thus replace it with synchronizing by a performance_schema.log_status query. - Add table_log_status::get_row method to allow doing the log_status query from inside the server. - Handlerton: introduce two new methods: clone_precopy - currently unused, which will handle any non-consistent-snapshot copying before the main consistent copy, and set_log_stop - which provides the SE the clone stop position information. - Plugin: introduce Ha_clone_common_cbk, which derives from Ha_clone_cbk, and from which derive all the concrete classes: Client_Cbk, Local_Cbk & Server_Cbk. - Ha_clone_common_cbk contains two methods: precopy and synchronize_engines. The former calls precopy for all the participating clone SE handlertons, if implemented. The latter, only to be called from the main clone thread, performs the performance_schema.LOG_STATUS query, splits its result per-SE and calls handlerton set_log_stop for all SEs. - The pfs.log_status query still must be called by InnoDB at a certain point in clone. That means that InnoDB is still a special SE in that it must always come first in the clone locator vectors. This will also handle donor waits on clone restarts, as that is implemented in InnoDB-specific code. To ensure this, add a helper clone_hton.cc:make_innodb_first which shuffles the vectors as needed. Call it in all the clone-starting code paths, add asserts that InnoDB comes first. - Since hton_clone_copy will call into InnoDB clone copy, which at certain point will synchronize engines, calling into the plugin again, and the calling into SEs again, introduce a backup locator vector Ha_clone_cbk::all_locators, and its setters/getters set_all_locators, reset_all_locators, get_all_locators. Use them in the re-entrant hton_clone_copy code. - Introduce a new InnoDB clone state named SST COPY, which calls precopy and has zero files and chunks set to copy in the regular InnoDB clone way, i.e. all it does it the precopy. - The redo log archiving keeps running throughout the precopy stage. - On stage transition from the precopy to the redo log copy. call synchronize. - Implement innodb_clone_set_log_stop by parsing out the desired clone stop LSN from the passed JSON from the log_status query. - Make InnoDB redo log archiving stop at that LSN instead of on reaching the end of redo log. - Support stopping redo log archiving in a middle of a log block. As the old implementation may encounter this case only at the end of the redo log, it took that block from the log buffer in memory. This no longer works with the new implementation, where the needed block might reside on disk only. Address this by forcing the whole block containing the target LSN to disk, reading it from there, and patching its memory copy to stop at the required LSN before giving it for the clone to copy. - To implement redo log block reading on a running server, adapt crash recovery function recv_read_log_seg to work both for crash recovery and on a running server. This is similar to what Percona Server does for its changed page tracking feature. - Make sure not to archive not-yet-written log: pfs.log_status query gets the log system LSN, which is the value of highest log reservation. Thus in an archiving iteration, archive up to the lower values of log archiving LSN, and write_lsn. Set the log archiving LSN to LSN_MAX at clone initialization, so that clone archiving may proceed until the stop LSN is set. - The above implementation results in cloned LSN != log end LSN for the cloned instance, thus adjust crash recovery to work with that. The existing crash recovery throws away any crash-recovered metadata and reads its snapshot from the data files, whereas the snapshot was not written in any consistency with the clone LSN. Fix by no longer throwing away the crash-recovered InnoDB dynamic metadata and by not re-reading the the last metadata snapshot. This forces also not to re-serialize the metadata in the crash recovery as that writes redo log which is to be avoided during clone crash recovery. - Additionally, at the start of the redo log copy the old implementation requested all InnoDB updates to be flushed to disk immediately, to avoid clone recovery generating new redo log. This is also disabled as it becomes redundant with the fix. This issue is similar to Percona XtraBackup https://jira.percona.com/browse/PXB-1634 - Disable clone.local_xa test because the new synchronization no longer blocks on XA transactions. - Re-record two clone-related tests in the component_keyring_file MTR suite. - Temporarily workaround the MyRocks bug where log_status query hangs if RocksDB file deletions were not disabled first. Note that this change breaks some of the clone-related group replication MTR tests. As they are not tested in fb-mysql and have been already broken, this patch makes no effort to fix them. MyRocks Clone (5/n): fix bug 109922 (SEs do not coordinate clone rollbacks on instance startup) If InnoDB clone decides to rollback during its startup, other storage engines are not aware of this and proceed as usual, resulting in inconsistent instance between storage engines. Ideally this should be fixed through a 2PC-like design where all the SEs vote to rollback the clone or to continue during the startup, but for MyRocks clone a simpler solution suffices: make InnoDB write a special marker file if it decides to rollback on clone startup. MyRocks Clone (6/n): on remote clone error, always send COM_ACK to InnoDB too In the case of multithreaded clone, the waits for the donor threads for the current clone stage to complete are implemented fully in InnoDB, even though that, and error handling, should be plugin-level features. With two storage engines participating in the clone this allows the situation where the other storage engine client reports an error with COM_ACK to one donor thread, while some other threads are in InnoDB clone state change waits. Those waits can abort if InnoDB clone gets an error from the client, but they know nothing about other SE errors, and will be aborted only by timeouts. As a minimal fix that does not involve pulling the whole error handling and state change waiting logic to the plugin layer, make the clone client always send an error-containing COM_ACK to InnoDB too, whenever it sends one to another storage engine. MyRocks Clone (7/n): MyRocks clone implementation - Cleanup any stale RocksDB checkpoint directories on server shutdown too. - For MyRocks clone network protocol, define four packet types (clone/common.h): - LOCATOR_V1: MyRocks-specific clone locator, consisting of a monotonically increasing clone session ID for the donor with an optional applier state (an array of file IDs and applied offsets) for restarts, or a zero in which case it is a version check empty locator used for handshakes. - FILE_NAME_MAP_V1: establishes a relation between a file ID, its name and target size. - FILE_CHUNK_V1: cloned file chunk data - ADD_ESTIMATE_V1: how many bytes should be added to the estimated clone data size. - All packet types share the common package header, which stores the version, the payload length without the header, and the packet type. - (clone/common.h): classes locator, file_metadata, chunk_metadata, & estimate_metadata correspond to the four packet types. They are subclassed on the donor side to add serialization routines and other donor-specific code and on the client side to add deserialization routines and other client-specific code. Re. serialization/deserialization, an exception is the locator, whose routines are implemented in the common code, because it may be passed both - ways. - (clone/common.h): Introduce a session class that handles the task ID set and errors, which is common for both donor and client. For errors, allow saving one error, with an optional associated path and errno, that can be set by any thread and later reported to the clone plugin. - (clone/common.cc) Setup MyRocks clone capabilities as HA_CLONE_HYBRID (not - applicable but matches InnoDB), HA_CLONE_MULTI_TASK (can clone with multiple - threads), & HA_CLONE_RESTART (can restart after a network I/O error). - (clone/common.cc) Enable O_DIRECT for clone file I/O if, on client if rocksdb_use_direct_io_for_flush_and_compaction is enabled, and on donor if rocksdb_use_direct_reads is enabled. - On the donor side (clone/donor.cc), encapsulate RocksDB checkpoints in a rdb_checkpoint class. Each checkpoint gets its own ID, which is also a donor session ID. The ID is constant for rolling checkpoints. - Subclass the common file_metadata to donor_file_metadata, chunk_metadata to donor_chunk_metadata, estimate_metadata to donor_estimate. - Implement donor going through a sequence of states: NONE, ROLLING, FINAL, FINAL_WITH_LOGS. To support multiple clone donor sessions, implement a manager class that looks up the donor instance by locator. - Implement MyRocks clone donor interface: - rocksdb_clone_start: handle the possible start modes: - HA_CLONE_MODE_START: create and register a new donor instance, register its master task ID. - HA_CLONE_MODE_ADD_TASK: lookup the existing donor instance, register a new task ID. - HA_CLONE_MODE_RESTART: lookup the existing donor instance, wait for the old master session to drop it by master task ID reference count, then process the locator applier state to reset the donor state. - rocksdb_clone_precopy: lookup the donor instance, create a RocksDB snapshot if state == NONE, otherwise send files by chunks and roll the checkpoint as needed. - rocksdb_clone_set_log_stop: add the WAL files to the clone file to copy set. - rocksdb_clone_copy: copy the clone files in any donor state. - rocksdb_clone_ack: acknowledge errors sent by client. - rocksdb_clone_end: for master task ID, stop accepting new worker join requests, wait until existing workers quit, and, in the case of restartable error, wait for any restarts to happen. Then decrement ref count for the master task ID (which will handle both the cases of restarted and not restarted) clone. For all other task IDs, simply unregister them. - On the client side (clone/client.cc), have file_in_progress class to track each individual file progress during application, and completed_file for fully-applied files. - Subclass the common file_metadata to client_file_metadata, implement its reading from the net buffer. Likewise for client_chunk_metadata subclassing chunk_metadata, and client_estimate for estimate_metadata. - TODO client class. - Implement MyRocks clone client interface: - rocksdb_clone_apply_begin: handle the possible start modes: - HA_CLONE_MODE_START: initialize the client instance for a new clone - HA_CLONE_MODE_ADD_TASK: register a new task ID for an ongoing clone - HA_CLONE_MODE_RESTART: restart the interrupted clone session - HA_CLONE_MODE_VERSION: prepare new version locator - rocksdb_clone_apply: if passed an error code, save it. Otherwise verify the passed buffer and dispatch ADD_ESTIMATE_V1, FILE_NAME_MAP_V1, & FILE_CHUNK_V1 packets accordingly. - rocksdb_clone_apply_end: for master task, wait for all other tasks to complete and finish the clone. For both master and non-master tasks unregister their IDs. - Add rocksdb_sys_vars tests for the new clone-related system variables. They are included in the current patch and not the MTR one in order not to break the existing rocksdb_sys_vars.all_vars test. - On instance startup, do clone fixup (clone/common.cc). It consists of: - Check whether InnoDB clone rollback marker is present. If yes, force rollback of MyRocks clone. - If the cloned temporary datadir contains clone-in-progress marker, assume half-completed clone and abort. - Move the cloned datadirs and WAL dirs in-place. For the datadir, do a directory-level rename with a save and later delete of the old datadir. For the WAL dir do not try to do parent directory-level renames because the mysqld process may not have permissions to delete that directory. Instead, rename all the old files individually, move new files in, delete the old files. - For all the rename operations, handle them returning EXDEV error (a rename would cross a filesystem boundary) by falling back to copy and delete. As we expect that it is the cloned RocksDB log files that are most affected by this, the total copy I/O wouldn't be large, and so the default MySQL copy routine my_copy is used, without extra copy throughput optimizations. Add a fault injection point for MTR to test this code paths. - Remove any RocksDB clone checkpoints found. - On instance startup, always move WALs to the target directory if they are found in the rocksdb_datadir yet the rocksdb_wal_dir is different. This handles the case of setting rocksdb_wal_dir while instance is shutdown, both for clone fixup and regular operation. - Add PFS instrumentation for all the instrumentable entities in this patch (mutexes, rwlocks, file I/O). MyRocks Clone (8/n): MyRocks clone MTR tests - Add a new MTR test suite rocksdb_clone - Support new variable MTR clone/include/clone_command.inc: clone_single_threaded, setting which forces single-threaded clone operation. - Reuse a big part of InnoDB clone tests by including their test files, with the exception of tests involving DDL, as concurrent DDL is not supported by MyRocks clone at the moment, and debug tests that adjust cloned instance LSN, because they desynchronize storage engines. The reused tests are: local_basic, local_boundary, local_dml, local_dml_auto_tune, local_dml_no_redo, local_insert, local_stage, monitor_progress, remote_basic, remote_basic_replace, remote_boundary, remote_dml, remote_dml_auto_tune, remote_dml_gtid, remote_dml_no_binlog, remote_dml_no_ssl, remote_dml_recovery, remote_dml_replace, remote_dml_ssl, remote_dml_ssl_certificate, remote_dml_upgrade, remote_insert, remote_monitor_progress, remote_restart, remote_skip_validation, remote_stage, remote_stage_restart. - As a part of the above, add MyRocks support to local_basic and monitor_progress MTR tests. - Besides the reused InnoDB clone tests, add some new MyRocks-specific tests: - local_minimal_1_thread, which is the simplest possible local clone test. - local_minimal_multithreaded, the simplest possible but multithreaded local clone test. - local_minimal_direct_io, which is the simplest possible local clone test in all combinations of rocksdb_use_direct_reads and rocksdb_use_direct_io_for_flush_and_compaction settings, which are the O_DIRECT-enabling options. - local_dml_o_direct_donor, local_dml_o_direct_client, local_dml_o_direct_both that reuse InnoDB local_dml test, but combined with either or both of the O_DIRECT-enabling options. - local_minimal_wal_dir, which clones with various rocksdb_wal_dir settings on donor and client. - local_minimal_wal_dir_copy_delete, which uses a debug injection to test the move-across-filesystem-boundary code path in clone startup. - remote_minimal_direct_io_{1, 2, 3, 4} tests for all the combinations of O_DIRECT-enabling options for donor and client instances. - remote_fail_apply, which uses a debug injection to fail the clone client during remote clone. - remote_inplace_restart, which performs a remote in-place clone with a simulated transient error on the client. - remote_kill_donor, which kill the donor instance at various points of remote cloning. - rolling_checkpoint for testing various combinations of clone rolling checkpoint tuning options. - three_se_basic to test a simple clone across InnoDB, MyRocks, & Blackhole engines. Fix moving RocksDB WALs on startup across file system boundaries Reviewed By: sunshine-Chun Differential Revision: D43160162 Pulled By: hermanlee fbshipit-source-id: 5358a57 -------------------------------------------------------------------------------- Rdb clone fixes (facebook#1271) Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 2245706 -------------------------------------------------------------------------------- Clone: fix race condition that may skip some SST files (facebook#1274) Summary: A donor thread in the main copy loop took a file to send, released the donor state mutex, and then opened the file. If an ENOENT was received at this point, it was assumed that this was a stale SST file from an older checkpoint that was rolled since. Because the donor state mutex was released between taking of the file and opening it, the following race was possible: 1) Thread 1 takes the file 2) Thread 2 decides to roll the checkpoint, the old checkpoint is deleted 3) Thread 1 tries to open the file, gets ENOENT 4) Thread 2 creates the new checkpoint, the file re-appears, but it's too late. Rolling the checkpoint in a donor state mutex critical section is a possible fix, but such section would do a lot of I/O, serializing the parallel threads. Instead, fix by taking the file and opening it in the same critical section. Pull Request resolved: facebook#1274 Reviewed By: sunshine-Chun Differential Revision: D43629546 Pulled By: hermanlee fbshipit-source-id: 6e7f315
Summary: Pull Request resolved: facebook#1250 MyRocks Clone (1/n): make InnoDB clone MTR tests SE-agnostic The MyRocks clone MTR tests will reuse the InnoDB tests by including them. Adjust the tests so that they work this way, i.e. make them storage engine-agnostic by checking the default_storage_engine system variable and guarding InnoDB-specific tests this way. This includes the tests that artificially desync the InnoDB redo log position. MyRocks Clone (2/n): fix bug 109919 (Clone plugin does not support arbitrary length O_DIRECT files) The existing clone plugin I/O routines handle O_DIRECT files with the assumption that their sizes are always a multiple of a block size. This is too restrictive because in general O_DIRECT files can be of any size and MyRocks files actually are. Add a flag to Ha_clone_file to support arbitrary file lengths under O_DIRECT: o_direct_uneven_file_size. Pass it to clone_client.h/cc Client::get_aligned_buffer to allocate the buffer that bumps the length to the next CLONE_OS_ALIGN multiple if it's not already one. Introduce a pad_for_o_direct helper that performs this calculation for the passed length as needed. For writes, if O_DIRECT is enabled and the buffer length is not a multiple of CLONE_OS_ALIGN, pad it with zeros to the next multiple. Then check the actual bytes written, and if any padding bytes were actually written, call my_chsize to truncate the file to the needed size. MyRocks Clone (3/n): fix bug 109920 (SEs cannot add to clone data size estimates) In the current implementation, once the first SE sets the data size estimate for a particular stage, there is no way for other SEs to add to it. Address this by adding Ha_clone_cbk::add_to_data_size_estimate method. MyRocks Clone (4/n): fix bug 109921 (Cannot clone with synchronization across SEs) The existing clone implementation synchronizes InnoDB and binlog directly. There is no way to make other storage engines join this synchronization, thus replace it with synchronizing by a performance_schema.log_status query. - Add table_log_status::get_row method to allow doing the log_status query from inside the server. - Handlerton: introduce two new methods: clone_precopy - currently unused, which will handle any non-consistent-snapshot copying before the main consistent copy, and set_log_stop - which provides the SE the clone stop position information. - Plugin: introduce Ha_clone_common_cbk, which derives from Ha_clone_cbk, and from which derive all the concrete classes: Client_Cbk, Local_Cbk & Server_Cbk. - Ha_clone_common_cbk contains two methods: precopy and synchronize_engines. The former calls precopy for all the participating clone SE handlertons, if implemented. The latter, only to be called from the main clone thread, performs the performance_schema.LOG_STATUS query, splits its result per-SE and calls handlerton set_log_stop for all SEs. - The pfs.log_status query still must be called by InnoDB at a certain point in clone. That means that InnoDB is still a special SE in that it must always come first in the clone locator vectors. This will also handle donor waits on clone restarts, as that is implemented in InnoDB-specific code. To ensure this, add a helper clone_hton.cc:make_innodb_first which shuffles the vectors as needed. Call it in all the clone-starting code paths, add asserts that InnoDB comes first. - Since hton_clone_copy will call into InnoDB clone copy, which at certain point will synchronize engines, calling into the plugin again, and the calling into SEs again, introduce a backup locator vector Ha_clone_cbk::all_locators, and its setters/getters set_all_locators, reset_all_locators, get_all_locators. Use them in the re-entrant hton_clone_copy code. - Introduce a new InnoDB clone state named SST COPY, which calls precopy and has zero files and chunks set to copy in the regular InnoDB clone way, i.e. all it does it the precopy. - The redo log archiving keeps running throughout the precopy stage. - On stage transition from the precopy to the redo log copy. call synchronize. - Implement innodb_clone_set_log_stop by parsing out the desired clone stop LSN from the passed JSON from the log_status query. - Make InnoDB redo log archiving stop at that LSN instead of on reaching the end of redo log. - Support stopping redo log archiving in a middle of a log block. As the old implementation may encounter this case only at the end of the redo log, it took that block from the log buffer in memory. This no longer works with the new implementation, where the needed block might reside on disk only. Address this by forcing the whole block containing the target LSN to disk, reading it from there, and patching its memory copy to stop at the required LSN before giving it for the clone to copy. - To implement redo log block reading on a running server, adapt crash recovery function recv_read_log_seg to work both for crash recovery and on a running server. This is similar to what Percona Server does for its changed page tracking feature. - Make sure not to archive not-yet-written log: pfs.log_status query gets the log system LSN, which is the value of highest log reservation. Thus in an archiving iteration, archive up to the lower values of log archiving LSN, and write_lsn. Set the log archiving LSN to LSN_MAX at clone initialization, so that clone archiving may proceed until the stop LSN is set. - The above implementation results in cloned LSN != log end LSN for the cloned instance, thus adjust crash recovery to work with that. The existing crash recovery throws away any crash-recovered metadata and reads its snapshot from the data files, whereas the snapshot was not written in any consistency with the clone LSN. Fix by no longer throwing away the crash-recovered InnoDB dynamic metadata and by not re-reading the the last metadata snapshot. This forces also not to re-serialize the metadata in the crash recovery as that writes redo log which is to be avoided during clone crash recovery. - Additionally, at the start of the redo log copy the old implementation requested all InnoDB updates to be flushed to disk immediately, to avoid clone recovery generating new redo log. This is also disabled as it becomes redundant with the fix. This issue is similar to Percona XtraBackup https://jira.percona.com/browse/PXB-1634 - Disable clone.local_xa test because the new synchronization no longer blocks on XA transactions. - Re-record two clone-related tests in the component_keyring_file MTR suite. - Temporarily workaround the MyRocks bug where log_status query hangs if RocksDB file deletions were not disabled first. Note that this change breaks some of the clone-related group replication MTR tests. As they are not tested in fb-mysql and have been already broken, this patch makes no effort to fix them. MyRocks Clone (5/n): fix bug 109922 (SEs do not coordinate clone rollbacks on instance startup) If InnoDB clone decides to rollback during its startup, other storage engines are not aware of this and proceed as usual, resulting in inconsistent instance between storage engines. Ideally this should be fixed through a 2PC-like design where all the SEs vote to rollback the clone or to continue during the startup, but for MyRocks clone a simpler solution suffices: make InnoDB write a special marker file if it decides to rollback on clone startup. MyRocks Clone (6/n): on remote clone error, always send COM_ACK to InnoDB too In the case of multithreaded clone, the waits for the donor threads for the current clone stage to complete are implemented fully in InnoDB, even though that, and error handling, should be plugin-level features. With two storage engines participating in the clone this allows the situation where the other storage engine client reports an error with COM_ACK to one donor thread, while some other threads are in InnoDB clone state change waits. Those waits can abort if InnoDB clone gets an error from the client, but they know nothing about other SE errors, and will be aborted only by timeouts. As a minimal fix that does not involve pulling the whole error handling and state change waiting logic to the plugin layer, make the clone client always send an error-containing COM_ACK to InnoDB too, whenever it sends one to another storage engine. MyRocks Clone (7/n): MyRocks clone implementation - Cleanup any stale RocksDB checkpoint directories on server shutdown too. - For MyRocks clone network protocol, define four packet types (clone/common.h): - LOCATOR_V1: MyRocks-specific clone locator, consisting of a monotonically increasing clone session ID for the donor with an optional applier state (an array of file IDs and applied offsets) for restarts, or a zero in which case it is a version check empty locator used for handshakes. - FILE_NAME_MAP_V1: establishes a relation between a file ID, its name and target size. - FILE_CHUNK_V1: cloned file chunk data - ADD_ESTIMATE_V1: how many bytes should be added to the estimated clone data size. - All packet types share the common package header, which stores the version, the payload length without the header, and the packet type. - (clone/common.h): classes locator, file_metadata, chunk_metadata, & estimate_metadata correspond to the four packet types. They are subclassed on the donor side to add serialization routines and other donor-specific code and on the client side to add deserialization routines and other client-specific code. Re. serialization/deserialization, an exception is the locator, whose routines are implemented in the common code, because it may be passed both - ways. - (clone/common.h): Introduce a session class that handles the task ID set and errors, which is common for both donor and client. For errors, allow saving one error, with an optional associated path and errno, that can be set by any thread and later reported to the clone plugin. - (clone/common.cc) Setup MyRocks clone capabilities as HA_CLONE_HYBRID (not - applicable but matches InnoDB), HA_CLONE_MULTI_TASK (can clone with multiple - threads), & HA_CLONE_RESTART (can restart after a network I/O error). - (clone/common.cc) Enable O_DIRECT for clone file I/O if, on client if rocksdb_use_direct_io_for_flush_and_compaction is enabled, and on donor if rocksdb_use_direct_reads is enabled. - On the donor side (clone/donor.cc), encapsulate RocksDB checkpoints in a rdb_checkpoint class. Each checkpoint gets its own ID, which is also a donor session ID. The ID is constant for rolling checkpoints. - Subclass the common file_metadata to donor_file_metadata, chunk_metadata to donor_chunk_metadata, estimate_metadata to donor_estimate. - Implement donor going through a sequence of states: NONE, ROLLING, FINAL, FINAL_WITH_LOGS. To support multiple clone donor sessions, implement a manager class that looks up the donor instance by locator. - Implement MyRocks clone donor interface: - rocksdb_clone_start: handle the possible start modes: - HA_CLONE_MODE_START: create and register a new donor instance, register its master task ID. - HA_CLONE_MODE_ADD_TASK: lookup the existing donor instance, register a new task ID. - HA_CLONE_MODE_RESTART: lookup the existing donor instance, wait for the old master session to drop it by master task ID reference count, then process the locator applier state to reset the donor state. - rocksdb_clone_precopy: lookup the donor instance, create a RocksDB snapshot if state == NONE, otherwise send files by chunks and roll the checkpoint as needed. - rocksdb_clone_set_log_stop: add the WAL files to the clone file to copy set. - rocksdb_clone_copy: copy the clone files in any donor state. - rocksdb_clone_ack: acknowledge errors sent by client. - rocksdb_clone_end: for master task ID, stop accepting new worker join requests, wait until existing workers quit, and, in the case of restartable error, wait for any restarts to happen. Then decrement ref count for the master task ID (which will handle both the cases of restarted and not restarted) clone. For all other task IDs, simply unregister them. - On the client side (clone/client.cc), have file_in_progress class to track each individual file progress during application, and completed_file for fully-applied files. - Subclass the common file_metadata to client_file_metadata, implement its reading from the net buffer. Likewise for client_chunk_metadata subclassing chunk_metadata, and client_estimate for estimate_metadata. - TODO client class. - Implement MyRocks clone client interface: - rocksdb_clone_apply_begin: handle the possible start modes: - HA_CLONE_MODE_START: initialize the client instance for a new clone - HA_CLONE_MODE_ADD_TASK: register a new task ID for an ongoing clone - HA_CLONE_MODE_RESTART: restart the interrupted clone session - HA_CLONE_MODE_VERSION: prepare new version locator - rocksdb_clone_apply: if passed an error code, save it. Otherwise verify the passed buffer and dispatch ADD_ESTIMATE_V1, FILE_NAME_MAP_V1, & FILE_CHUNK_V1 packets accordingly. - rocksdb_clone_apply_end: for master task, wait for all other tasks to complete and finish the clone. For both master and non-master tasks unregister their IDs. - Add rocksdb_sys_vars tests for the new clone-related system variables. They are included in the current patch and not the MTR one in order not to break the existing rocksdb_sys_vars.all_vars test. - On instance startup, do clone fixup (clone/common.cc). It consists of: - Check whether InnoDB clone rollback marker is present. If yes, force rollback of MyRocks clone. - If the cloned temporary datadir contains clone-in-progress marker, assume half-completed clone and abort. - Move the cloned datadirs and WAL dirs in-place. For the datadir, do a directory-level rename with a save and later delete of the old datadir. For the WAL dir do not try to do parent directory-level renames because the mysqld process may not have permissions to delete that directory. Instead, rename all the old files individually, move new files in, delete the old files. - For all the rename operations, handle them returning EXDEV error (a rename would cross a filesystem boundary) by falling back to copy and delete. As we expect that it is the cloned RocksDB log files that are most affected by this, the total copy I/O wouldn't be large, and so the default MySQL copy routine my_copy is used, without extra copy throughput optimizations. Add a fault injection point for MTR to test this code paths. - Remove any RocksDB clone checkpoints found. - On instance startup, always move WALs to the target directory if they are found in the rocksdb_datadir yet the rocksdb_wal_dir is different. This handles the case of setting rocksdb_wal_dir while instance is shutdown, both for clone fixup and regular operation. - Add PFS instrumentation for all the instrumentable entities in this patch (mutexes, rwlocks, file I/O). MyRocks Clone (8/n): MyRocks clone MTR tests - Add a new MTR test suite rocksdb_clone - Support new variable MTR clone/include/clone_command.inc: clone_single_threaded, setting which forces single-threaded clone operation. - Reuse a big part of InnoDB clone tests by including their test files, with the exception of tests involving DDL, as concurrent DDL is not supported by MyRocks clone at the moment, and debug tests that adjust cloned instance LSN, because they desynchronize storage engines. The reused tests are: local_basic, local_boundary, local_dml, local_dml_auto_tune, local_dml_no_redo, local_insert, local_stage, monitor_progress, remote_basic, remote_basic_replace, remote_boundary, remote_dml, remote_dml_auto_tune, remote_dml_gtid, remote_dml_no_binlog, remote_dml_no_ssl, remote_dml_recovery, remote_dml_replace, remote_dml_ssl, remote_dml_ssl_certificate, remote_dml_upgrade, remote_insert, remote_monitor_progress, remote_restart, remote_skip_validation, remote_stage, remote_stage_restart. - As a part of the above, add MyRocks support to local_basic and monitor_progress MTR tests. - Besides the reused InnoDB clone tests, add some new MyRocks-specific tests: - local_minimal_1_thread, which is the simplest possible local clone test. - local_minimal_multithreaded, the simplest possible but multithreaded local clone test. - local_minimal_direct_io, which is the simplest possible local clone test in all combinations of rocksdb_use_direct_reads and rocksdb_use_direct_io_for_flush_and_compaction settings, which are the O_DIRECT-enabling options. - local_dml_o_direct_donor, local_dml_o_direct_client, local_dml_o_direct_both that reuse InnoDB local_dml test, but combined with either or both of the O_DIRECT-enabling options. - local_minimal_wal_dir, which clones with various rocksdb_wal_dir settings on donor and client. - local_minimal_wal_dir_copy_delete, which uses a debug injection to test the move-across-filesystem-boundary code path in clone startup. - remote_minimal_direct_io_{1, 2, 3, 4} tests for all the combinations of O_DIRECT-enabling options for donor and client instances. - remote_fail_apply, which uses a debug injection to fail the clone client during remote clone. - remote_inplace_restart, which performs a remote in-place clone with a simulated transient error on the client. - remote_kill_donor, which kill the donor instance at various points of remote cloning. - rolling_checkpoint for testing various combinations of clone rolling checkpoint tuning options. - three_se_basic to test a simple clone across InnoDB, MyRocks, & Blackhole engines. Fix moving RocksDB WALs on startup across file system boundaries Reviewed By: sunshine-Chun Differential Revision: D43160162 Pulled By: hermanlee fbshipit-source-id: 5358a57 -------------------------------------------------------------------------------- Rdb clone fixes (facebook#1271) Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 2245706 -------------------------------------------------------------------------------- Clone: fix race condition that may skip some SST files (facebook#1274) Summary: A donor thread in the main copy loop took a file to send, released the donor state mutex, and then opened the file. If an ENOENT was received at this point, it was assumed that this was a stale SST file from an older checkpoint that was rolled since. Because the donor state mutex was released between taking of the file and opening it, the following race was possible: 1) Thread 1 takes the file 2) Thread 2 decides to roll the checkpoint, the old checkpoint is deleted 3) Thread 1 tries to open the file, gets ENOENT 4) Thread 2 creates the new checkpoint, the file re-appears, but it's too late. Rolling the checkpoint in a donor state mutex critical section is a possible fix, but such section would do a lot of I/O, serializing the parallel threads. Instead, fix by taking the file and opening it in the same critical section. Pull Request resolved: facebook#1274 Reviewed By: sunshine-Chun Differential Revision: D43629546 Pulled By: hermanlee fbshipit-source-id: 6e7f315
Summary: Pull Request resolved: facebook#1250 MyRocks Clone (1/n): make InnoDB clone MTR tests SE-agnostic The MyRocks clone MTR tests will reuse the InnoDB tests by including them. Adjust the tests so that they work this way, i.e. make them storage engine-agnostic by checking the default_storage_engine system variable and guarding InnoDB-specific tests this way. This includes the tests that artificially desync the InnoDB redo log position. MyRocks Clone (2/n): fix bug 109919 (Clone plugin does not support arbitrary length O_DIRECT files) The existing clone plugin I/O routines handle O_DIRECT files with the assumption that their sizes are always a multiple of a block size. This is too restrictive because in general O_DIRECT files can be of any size and MyRocks files actually are. Add a flag to Ha_clone_file to support arbitrary file lengths under O_DIRECT: o_direct_uneven_file_size. Pass it to clone_client.h/cc Client::get_aligned_buffer to allocate the buffer that bumps the length to the next CLONE_OS_ALIGN multiple if it's not already one. Introduce a pad_for_o_direct helper that performs this calculation for the passed length as needed. For writes, if O_DIRECT is enabled and the buffer length is not a multiple of CLONE_OS_ALIGN, pad it with zeros to the next multiple. Then check the actual bytes written, and if any padding bytes were actually written, call my_chsize to truncate the file to the needed size. MyRocks Clone (3/n): fix bug 109920 (SEs cannot add to clone data size estimates) In the current implementation, once the first SE sets the data size estimate for a particular stage, there is no way for other SEs to add to it. Address this by adding Ha_clone_cbk::add_to_data_size_estimate method. MyRocks Clone (4/n): fix bug 109921 (Cannot clone with synchronization across SEs) The existing clone implementation synchronizes InnoDB and binlog directly. There is no way to make other storage engines join this synchronization, thus replace it with synchronizing by a performance_schema.log_status query. - Add table_log_status::get_row method to allow doing the log_status query from inside the server. - Handlerton: introduce two new methods: clone_precopy - currently unused, which will handle any non-consistent-snapshot copying before the main consistent copy, and set_log_stop - which provides the SE the clone stop position information. - Plugin: introduce Ha_clone_common_cbk, which derives from Ha_clone_cbk, and from which derive all the concrete classes: Client_Cbk, Local_Cbk & Server_Cbk. - Ha_clone_common_cbk contains two methods: precopy and synchronize_engines. The former calls precopy for all the participating clone SE handlertons, if implemented. The latter, only to be called from the main clone thread, performs the performance_schema.LOG_STATUS query, splits its result per-SE and calls handlerton set_log_stop for all SEs. - The pfs.log_status query still must be called by InnoDB at a certain point in clone. That means that InnoDB is still a special SE in that it must always come first in the clone locator vectors. This will also handle donor waits on clone restarts, as that is implemented in InnoDB-specific code. To ensure this, add a helper clone_hton.cc:make_innodb_first which shuffles the vectors as needed. Call it in all the clone-starting code paths, add asserts that InnoDB comes first. - Since hton_clone_copy will call into InnoDB clone copy, which at certain point will synchronize engines, calling into the plugin again, and the calling into SEs again, introduce a backup locator vector Ha_clone_cbk::all_locators, and its setters/getters set_all_locators, reset_all_locators, get_all_locators. Use them in the re-entrant hton_clone_copy code. - Introduce a new InnoDB clone state named SST COPY, which calls precopy and has zero files and chunks set to copy in the regular InnoDB clone way, i.e. all it does it the precopy. - The redo log archiving keeps running throughout the precopy stage. - On stage transition from the precopy to the redo log copy. call synchronize. - Implement innodb_clone_set_log_stop by parsing out the desired clone stop LSN from the passed JSON from the log_status query. - Make InnoDB redo log archiving stop at that LSN instead of on reaching the end of redo log. - Support stopping redo log archiving in a middle of a log block. As the old implementation may encounter this case only at the end of the redo log, it took that block from the log buffer in memory. This no longer works with the new implementation, where the needed block might reside on disk only. Address this by forcing the whole block containing the target LSN to disk, reading it from there, and patching its memory copy to stop at the required LSN before giving it for the clone to copy. - To implement redo log block reading on a running server, adapt crash recovery function recv_read_log_seg to work both for crash recovery and on a running server. This is similar to what Percona Server does for its changed page tracking feature. - Make sure not to archive not-yet-written log: pfs.log_status query gets the log system LSN, which is the value of highest log reservation. Thus in an archiving iteration, archive up to the lower values of log archiving LSN, and write_lsn. Set the log archiving LSN to LSN_MAX at clone initialization, so that clone archiving may proceed until the stop LSN is set. - The above implementation results in cloned LSN != log end LSN for the cloned instance, thus adjust crash recovery to work with that. The existing crash recovery throws away any crash-recovered metadata and reads its snapshot from the data files, whereas the snapshot was not written in any consistency with the clone LSN. Fix by no longer throwing away the crash-recovered InnoDB dynamic metadata and by not re-reading the the last metadata snapshot. This forces also not to re-serialize the metadata in the crash recovery as that writes redo log which is to be avoided during clone crash recovery. - Additionally, at the start of the redo log copy the old implementation requested all InnoDB updates to be flushed to disk immediately, to avoid clone recovery generating new redo log. This is also disabled as it becomes redundant with the fix. This issue is similar to Percona XtraBackup https://jira.percona.com/browse/PXB-1634 - Disable clone.local_xa test because the new synchronization no longer blocks on XA transactions. - Re-record two clone-related tests in the component_keyring_file MTR suite. - Temporarily workaround the MyRocks bug where log_status query hangs if RocksDB file deletions were not disabled first. Note that this change breaks some of the clone-related group replication MTR tests. As they are not tested in fb-mysql and have been already broken, this patch makes no effort to fix them. MyRocks Clone (5/n): fix bug 109922 (SEs do not coordinate clone rollbacks on instance startup) If InnoDB clone decides to rollback during its startup, other storage engines are not aware of this and proceed as usual, resulting in inconsistent instance between storage engines. Ideally this should be fixed through a 2PC-like design where all the SEs vote to rollback the clone or to continue during the startup, but for MyRocks clone a simpler solution suffices: make InnoDB write a special marker file if it decides to rollback on clone startup. MyRocks Clone (6/n): on remote clone error, always send COM_ACK to InnoDB too In the case of multithreaded clone, the waits for the donor threads for the current clone stage to complete are implemented fully in InnoDB, even though that, and error handling, should be plugin-level features. With two storage engines participating in the clone this allows the situation where the other storage engine client reports an error with COM_ACK to one donor thread, while some other threads are in InnoDB clone state change waits. Those waits can abort if InnoDB clone gets an error from the client, but they know nothing about other SE errors, and will be aborted only by timeouts. As a minimal fix that does not involve pulling the whole error handling and state change waiting logic to the plugin layer, make the clone client always send an error-containing COM_ACK to InnoDB too, whenever it sends one to another storage engine. MyRocks Clone (7/n): MyRocks clone implementation - Cleanup any stale RocksDB checkpoint directories on server shutdown too. - For MyRocks clone network protocol, define four packet types (clone/common.h): - LOCATOR_V1: MyRocks-specific clone locator, consisting of a monotonically increasing clone session ID for the donor with an optional applier state (an array of file IDs and applied offsets) for restarts, or a zero in which case it is a version check empty locator used for handshakes. - FILE_NAME_MAP_V1: establishes a relation between a file ID, its name and target size. - FILE_CHUNK_V1: cloned file chunk data - ADD_ESTIMATE_V1: how many bytes should be added to the estimated clone data size. - All packet types share the common package header, which stores the version, the payload length without the header, and the packet type. - (clone/common.h): classes locator, file_metadata, chunk_metadata, & estimate_metadata correspond to the four packet types. They are subclassed on the donor side to add serialization routines and other donor-specific code and on the client side to add deserialization routines and other client-specific code. Re. serialization/deserialization, an exception is the locator, whose routines are implemented in the common code, because it may be passed both - ways. - (clone/common.h): Introduce a session class that handles the task ID set and errors, which is common for both donor and client. For errors, allow saving one error, with an optional associated path and errno, that can be set by any thread and later reported to the clone plugin. - (clone/common.cc) Setup MyRocks clone capabilities as HA_CLONE_HYBRID (not - applicable but matches InnoDB), HA_CLONE_MULTI_TASK (can clone with multiple - threads), & HA_CLONE_RESTART (can restart after a network I/O error). - (clone/common.cc) Enable O_DIRECT for clone file I/O if, on client if rocksdb_use_direct_io_for_flush_and_compaction is enabled, and on donor if rocksdb_use_direct_reads is enabled. - On the donor side (clone/donor.cc), encapsulate RocksDB checkpoints in a rdb_checkpoint class. Each checkpoint gets its own ID, which is also a donor session ID. The ID is constant for rolling checkpoints. - Subclass the common file_metadata to donor_file_metadata, chunk_metadata to donor_chunk_metadata, estimate_metadata to donor_estimate. - Implement donor going through a sequence of states: NONE, ROLLING, FINAL, FINAL_WITH_LOGS. To support multiple clone donor sessions, implement a manager class that looks up the donor instance by locator. - Implement MyRocks clone donor interface: - rocksdb_clone_start: handle the possible start modes: - HA_CLONE_MODE_START: create and register a new donor instance, register its master task ID. - HA_CLONE_MODE_ADD_TASK: lookup the existing donor instance, register a new task ID. - HA_CLONE_MODE_RESTART: lookup the existing donor instance, wait for the old master session to drop it by master task ID reference count, then process the locator applier state to reset the donor state. - rocksdb_clone_precopy: lookup the donor instance, create a RocksDB snapshot if state == NONE, otherwise send files by chunks and roll the checkpoint as needed. - rocksdb_clone_set_log_stop: add the WAL files to the clone file to copy set. - rocksdb_clone_copy: copy the clone files in any donor state. - rocksdb_clone_ack: acknowledge errors sent by client. - rocksdb_clone_end: for master task ID, stop accepting new worker join requests, wait until existing workers quit, and, in the case of restartable error, wait for any restarts to happen. Then decrement ref count for the master task ID (which will handle both the cases of restarted and not restarted) clone. For all other task IDs, simply unregister them. - On the client side (clone/client.cc), have file_in_progress class to track each individual file progress during application, and completed_file for fully-applied files. - Subclass the common file_metadata to client_file_metadata, implement its reading from the net buffer. Likewise for client_chunk_metadata subclassing chunk_metadata, and client_estimate for estimate_metadata. - TODO client class. - Implement MyRocks clone client interface: - rocksdb_clone_apply_begin: handle the possible start modes: - HA_CLONE_MODE_START: initialize the client instance for a new clone - HA_CLONE_MODE_ADD_TASK: register a new task ID for an ongoing clone - HA_CLONE_MODE_RESTART: restart the interrupted clone session - HA_CLONE_MODE_VERSION: prepare new version locator - rocksdb_clone_apply: if passed an error code, save it. Otherwise verify the passed buffer and dispatch ADD_ESTIMATE_V1, FILE_NAME_MAP_V1, & FILE_CHUNK_V1 packets accordingly. - rocksdb_clone_apply_end: for master task, wait for all other tasks to complete and finish the clone. For both master and non-master tasks unregister their IDs. - Add rocksdb_sys_vars tests for the new clone-related system variables. They are included in the current patch and not the MTR one in order not to break the existing rocksdb_sys_vars.all_vars test. - On instance startup, do clone fixup (clone/common.cc). It consists of: - Check whether InnoDB clone rollback marker is present. If yes, force rollback of MyRocks clone. - If the cloned temporary datadir contains clone-in-progress marker, assume half-completed clone and abort. - Move the cloned datadirs and WAL dirs in-place. For the datadir, do a directory-level rename with a save and later delete of the old datadir. For the WAL dir do not try to do parent directory-level renames because the mysqld process may not have permissions to delete that directory. Instead, rename all the old files individually, move new files in, delete the old files. - For all the rename operations, handle them returning EXDEV error (a rename would cross a filesystem boundary) by falling back to copy and delete. As we expect that it is the cloned RocksDB log files that are most affected by this, the total copy I/O wouldn't be large, and so the default MySQL copy routine my_copy is used, without extra copy throughput optimizations. Add a fault injection point for MTR to test this code paths. - Remove any RocksDB clone checkpoints found. - On instance startup, always move WALs to the target directory if they are found in the rocksdb_datadir yet the rocksdb_wal_dir is different. This handles the case of setting rocksdb_wal_dir while instance is shutdown, both for clone fixup and regular operation. - Add PFS instrumentation for all the instrumentable entities in this patch (mutexes, rwlocks, file I/O). MyRocks Clone (8/n): MyRocks clone MTR tests - Add a new MTR test suite rocksdb_clone - Support new variable MTR clone/include/clone_command.inc: clone_single_threaded, setting which forces single-threaded clone operation. - Reuse a big part of InnoDB clone tests by including their test files, with the exception of tests involving DDL, as concurrent DDL is not supported by MyRocks clone at the moment, and debug tests that adjust cloned instance LSN, because they desynchronize storage engines. The reused tests are: local_basic, local_boundary, local_dml, local_dml_auto_tune, local_dml_no_redo, local_insert, local_stage, monitor_progress, remote_basic, remote_basic_replace, remote_boundary, remote_dml, remote_dml_auto_tune, remote_dml_gtid, remote_dml_no_binlog, remote_dml_no_ssl, remote_dml_recovery, remote_dml_replace, remote_dml_ssl, remote_dml_ssl_certificate, remote_dml_upgrade, remote_insert, remote_monitor_progress, remote_restart, remote_skip_validation, remote_stage, remote_stage_restart. - As a part of the above, add MyRocks support to local_basic and monitor_progress MTR tests. - Besides the reused InnoDB clone tests, add some new MyRocks-specific tests: - local_minimal_1_thread, which is the simplest possible local clone test. - local_minimal_multithreaded, the simplest possible but multithreaded local clone test. - local_minimal_direct_io, which is the simplest possible local clone test in all combinations of rocksdb_use_direct_reads and rocksdb_use_direct_io_for_flush_and_compaction settings, which are the O_DIRECT-enabling options. - local_dml_o_direct_donor, local_dml_o_direct_client, local_dml_o_direct_both that reuse InnoDB local_dml test, but combined with either or both of the O_DIRECT-enabling options. - local_minimal_wal_dir, which clones with various rocksdb_wal_dir settings on donor and client. - local_minimal_wal_dir_copy_delete, which uses a debug injection to test the move-across-filesystem-boundary code path in clone startup. - remote_minimal_direct_io_{1, 2, 3, 4} tests for all the combinations of O_DIRECT-enabling options for donor and client instances. - remote_fail_apply, which uses a debug injection to fail the clone client during remote clone. - remote_inplace_restart, which performs a remote in-place clone with a simulated transient error on the client. - remote_kill_donor, which kill the donor instance at various points of remote cloning. - rolling_checkpoint for testing various combinations of clone rolling checkpoint tuning options. - three_se_basic to test a simple clone across InnoDB, MyRocks, & Blackhole engines. Fix moving RocksDB WALs on startup across file system boundaries Reviewed By: sunshine-Chun Differential Revision: D43160162 Pulled By: hermanlee fbshipit-source-id: 5358a57 -------------------------------------------------------------------------------- Rdb clone fixes (facebook#1271) Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 2245706 -------------------------------------------------------------------------------- Clone: fix race condition that may skip some SST files (facebook#1274) Summary: A donor thread in the main copy loop took a file to send, released the donor state mutex, and then opened the file. If an ENOENT was received at this point, it was assumed that this was a stale SST file from an older checkpoint that was rolled since. Because the donor state mutex was released between taking of the file and opening it, the following race was possible: 1) Thread 1 takes the file 2) Thread 2 decides to roll the checkpoint, the old checkpoint is deleted 3) Thread 1 tries to open the file, gets ENOENT 4) Thread 2 creates the new checkpoint, the file re-appears, but it's too late. Rolling the checkpoint in a donor state mutex critical section is a possible fix, but such section would do a lot of I/O, serializing the parallel threads. Instead, fix by taking the file and opening it in the same critical section. Pull Request resolved: facebook#1274 Reviewed By: sunshine-Chun Differential Revision: D43629546 Pulled By: hermanlee fbshipit-source-id: 6e7f315 -------------------------------------------------------------------------------- MyRocks clone: add missing donor mutex unlocks on error exits (facebook#1317) Summary: Pull Request resolved: facebook#1317 Differential Revision: D46564079 fbshipit-source-id: 6fc1a4f
Summary: Pull Request resolved: facebook#1250 MyRocks Clone (1/n): make InnoDB clone MTR tests SE-agnostic The MyRocks clone MTR tests will reuse the InnoDB tests by including them. Adjust the tests so that they work this way, i.e. make them storage engine-agnostic by checking the default_storage_engine system variable and guarding InnoDB-specific tests this way. This includes the tests that artificially desync the InnoDB redo log position. MyRocks Clone (2/n): fix bug 109919 (Clone plugin does not support arbitrary length O_DIRECT files) The existing clone plugin I/O routines handle O_DIRECT files with the assumption that their sizes are always a multiple of a block size. This is too restrictive because in general O_DIRECT files can be of any size and MyRocks files actually are. Add a flag to Ha_clone_file to support arbitrary file lengths under O_DIRECT: o_direct_uneven_file_size. Pass it to clone_client.h/cc Client::get_aligned_buffer to allocate the buffer that bumps the length to the next CLONE_OS_ALIGN multiple if it's not already one. Introduce a pad_for_o_direct helper that performs this calculation for the passed length as needed. For writes, if O_DIRECT is enabled and the buffer length is not a multiple of CLONE_OS_ALIGN, pad it with zeros to the next multiple. Then check the actual bytes written, and if any padding bytes were actually written, call my_chsize to truncate the file to the needed size. MyRocks Clone (3/n): fix bug 109920 (SEs cannot add to clone data size estimates) In the current implementation, once the first SE sets the data size estimate for a particular stage, there is no way for other SEs to add to it. Address this by adding Ha_clone_cbk::add_to_data_size_estimate method. MyRocks Clone (4/n): fix bug 109921 (Cannot clone with synchronization across SEs) The existing clone implementation synchronizes InnoDB and binlog directly. There is no way to make other storage engines join this synchronization, thus replace it with synchronizing by a performance_schema.log_status query. - Add table_log_status::get_row method to allow doing the log_status query from inside the server. - Handlerton: introduce two new methods: clone_precopy - currently unused, which will handle any non-consistent-snapshot copying before the main consistent copy, and set_log_stop - which provides the SE the clone stop position information. - Plugin: introduce Ha_clone_common_cbk, which derives from Ha_clone_cbk, and from which derive all the concrete classes: Client_Cbk, Local_Cbk & Server_Cbk. - Ha_clone_common_cbk contains two methods: precopy and synchronize_engines. The former calls precopy for all the participating clone SE handlertons, if implemented. The latter, only to be called from the main clone thread, performs the performance_schema.LOG_STATUS query, splits its result per-SE and calls handlerton set_log_stop for all SEs. - The pfs.log_status query still must be called by InnoDB at a certain point in clone. That means that InnoDB is still a special SE in that it must always come first in the clone locator vectors. This will also handle donor waits on clone restarts, as that is implemented in InnoDB-specific code. To ensure this, add a helper clone_hton.cc:make_innodb_first which shuffles the vectors as needed. Call it in all the clone-starting code paths, add asserts that InnoDB comes first. - Since hton_clone_copy will call into InnoDB clone copy, which at certain point will synchronize engines, calling into the plugin again, and the calling into SEs again, introduce a backup locator vector Ha_clone_cbk::all_locators, and its setters/getters set_all_locators, reset_all_locators, get_all_locators. Use them in the re-entrant hton_clone_copy code. - Introduce a new InnoDB clone state named SST COPY, which calls precopy and has zero files and chunks set to copy in the regular InnoDB clone way, i.e. all it does it the precopy. - The redo log archiving keeps running throughout the precopy stage. - On stage transition from the precopy to the redo log copy. call synchronize. - Implement innodb_clone_set_log_stop by parsing out the desired clone stop LSN from the passed JSON from the log_status query. - Make InnoDB redo log archiving stop at that LSN instead of on reaching the end of redo log. - Support stopping redo log archiving in a middle of a log block. As the old implementation may encounter this case only at the end of the redo log, it took that block from the log buffer in memory. This no longer works with the new implementation, where the needed block might reside on disk only. Address this by forcing the whole block containing the target LSN to disk, reading it from there, and patching its memory copy to stop at the required LSN before giving it for the clone to copy. - To implement redo log block reading on a running server, adapt crash recovery function recv_read_log_seg to work both for crash recovery and on a running server. This is similar to what Percona Server does for its changed page tracking feature. - Make sure not to archive not-yet-written log: pfs.log_status query gets the log system LSN, which is the value of highest log reservation. Thus in an archiving iteration, archive up to the lower values of log archiving LSN, and write_lsn. Set the log archiving LSN to LSN_MAX at clone initialization, so that clone archiving may proceed until the stop LSN is set. - The above implementation results in cloned LSN != log end LSN for the cloned instance, thus adjust crash recovery to work with that. The existing crash recovery throws away any crash-recovered metadata and reads its snapshot from the data files, whereas the snapshot was not written in any consistency with the clone LSN. Fix by no longer throwing away the crash-recovered InnoDB dynamic metadata and by not re-reading the the last metadata snapshot. This forces also not to re-serialize the metadata in the crash recovery as that writes redo log which is to be avoided during clone crash recovery. - Additionally, at the start of the redo log copy the old implementation requested all InnoDB updates to be flushed to disk immediately, to avoid clone recovery generating new redo log. This is also disabled as it becomes redundant with the fix. This issue is similar to Percona XtraBackup https://jira.percona.com/browse/PXB-1634 - Disable clone.local_xa test because the new synchronization no longer blocks on XA transactions. - Re-record two clone-related tests in the component_keyring_file MTR suite. - Temporarily workaround the MyRocks bug where log_status query hangs if RocksDB file deletions were not disabled first. Note that this change breaks some of the clone-related group replication MTR tests. As they are not tested in fb-mysql and have been already broken, this patch makes no effort to fix them. MyRocks Clone (5/n): fix bug 109922 (SEs do not coordinate clone rollbacks on instance startup) If InnoDB clone decides to rollback during its startup, other storage engines are not aware of this and proceed as usual, resulting in inconsistent instance between storage engines. Ideally this should be fixed through a 2PC-like design where all the SEs vote to rollback the clone or to continue during the startup, but for MyRocks clone a simpler solution suffices: make InnoDB write a special marker file if it decides to rollback on clone startup. MyRocks Clone (6/n): on remote clone error, always send COM_ACK to InnoDB too In the case of multithreaded clone, the waits for the donor threads for the current clone stage to complete are implemented fully in InnoDB, even though that, and error handling, should be plugin-level features. With two storage engines participating in the clone this allows the situation where the other storage engine client reports an error with COM_ACK to one donor thread, while some other threads are in InnoDB clone state change waits. Those waits can abort if InnoDB clone gets an error from the client, but they know nothing about other SE errors, and will be aborted only by timeouts. As a minimal fix that does not involve pulling the whole error handling and state change waiting logic to the plugin layer, make the clone client always send an error-containing COM_ACK to InnoDB too, whenever it sends one to another storage engine. MyRocks Clone (7/n): MyRocks clone implementation - Cleanup any stale RocksDB checkpoint directories on server shutdown too. - For MyRocks clone network protocol, define four packet types (clone/common.h): - LOCATOR_V1: MyRocks-specific clone locator, consisting of a monotonically increasing clone session ID for the donor with an optional applier state (an array of file IDs and applied offsets) for restarts, or a zero in which case it is a version check empty locator used for handshakes. - FILE_NAME_MAP_V1: establishes a relation between a file ID, its name and target size. - FILE_CHUNK_V1: cloned file chunk data - ADD_ESTIMATE_V1: how many bytes should be added to the estimated clone data size. - All packet types share the common package header, which stores the version, the payload length without the header, and the packet type. - (clone/common.h): classes locator, file_metadata, chunk_metadata, & estimate_metadata correspond to the four packet types. They are subclassed on the donor side to add serialization routines and other donor-specific code and on the client side to add deserialization routines and other client-specific code. Re. serialization/deserialization, an exception is the locator, whose routines are implemented in the common code, because it may be passed both - ways. - (clone/common.h): Introduce a session class that handles the task ID set and errors, which is common for both donor and client. For errors, allow saving one error, with an optional associated path and errno, that can be set by any thread and later reported to the clone plugin. - (clone/common.cc) Setup MyRocks clone capabilities as HA_CLONE_HYBRID (not - applicable but matches InnoDB), HA_CLONE_MULTI_TASK (can clone with multiple - threads), & HA_CLONE_RESTART (can restart after a network I/O error). - (clone/common.cc) Enable O_DIRECT for clone file I/O if, on client if rocksdb_use_direct_io_for_flush_and_compaction is enabled, and on donor if rocksdb_use_direct_reads is enabled. - On the donor side (clone/donor.cc), encapsulate RocksDB checkpoints in a rdb_checkpoint class. Each checkpoint gets its own ID, which is also a donor session ID. The ID is constant for rolling checkpoints. - Subclass the common file_metadata to donor_file_metadata, chunk_metadata to donor_chunk_metadata, estimate_metadata to donor_estimate. - Implement donor going through a sequence of states: NONE, ROLLING, FINAL, FINAL_WITH_LOGS. To support multiple clone donor sessions, implement a manager class that looks up the donor instance by locator. - Implement MyRocks clone donor interface: - rocksdb_clone_start: handle the possible start modes: - HA_CLONE_MODE_START: create and register a new donor instance, register its master task ID. - HA_CLONE_MODE_ADD_TASK: lookup the existing donor instance, register a new task ID. - HA_CLONE_MODE_RESTART: lookup the existing donor instance, wait for the old master session to drop it by master task ID reference count, then process the locator applier state to reset the donor state. - rocksdb_clone_precopy: lookup the donor instance, create a RocksDB snapshot if state == NONE, otherwise send files by chunks and roll the checkpoint as needed. - rocksdb_clone_set_log_stop: add the WAL files to the clone file to copy set. - rocksdb_clone_copy: copy the clone files in any donor state. - rocksdb_clone_ack: acknowledge errors sent by client. - rocksdb_clone_end: for master task ID, stop accepting new worker join requests, wait until existing workers quit, and, in the case of restartable error, wait for any restarts to happen. Then decrement ref count for the master task ID (which will handle both the cases of restarted and not restarted) clone. For all other task IDs, simply unregister them. - On the client side (clone/client.cc), have file_in_progress class to track each individual file progress during application, and completed_file for fully-applied files. - Subclass the common file_metadata to client_file_metadata, implement its reading from the net buffer. Likewise for client_chunk_metadata subclassing chunk_metadata, and client_estimate for estimate_metadata. - TODO client class. - Implement MyRocks clone client interface: - rocksdb_clone_apply_begin: handle the possible start modes: - HA_CLONE_MODE_START: initialize the client instance for a new clone - HA_CLONE_MODE_ADD_TASK: register a new task ID for an ongoing clone - HA_CLONE_MODE_RESTART: restart the interrupted clone session - HA_CLONE_MODE_VERSION: prepare new version locator - rocksdb_clone_apply: if passed an error code, save it. Otherwise verify the passed buffer and dispatch ADD_ESTIMATE_V1, FILE_NAME_MAP_V1, & FILE_CHUNK_V1 packets accordingly. - rocksdb_clone_apply_end: for master task, wait for all other tasks to complete and finish the clone. For both master and non-master tasks unregister their IDs. - Add rocksdb_sys_vars tests for the new clone-related system variables. They are included in the current patch and not the MTR one in order not to break the existing rocksdb_sys_vars.all_vars test. - On instance startup, do clone fixup (clone/common.cc). It consists of: - Check whether InnoDB clone rollback marker is present. If yes, force rollback of MyRocks clone. - If the cloned temporary datadir contains clone-in-progress marker, assume half-completed clone and abort. - Move the cloned datadirs and WAL dirs in-place. For the datadir, do a directory-level rename with a save and later delete of the old datadir. For the WAL dir do not try to do parent directory-level renames because the mysqld process may not have permissions to delete that directory. Instead, rename all the old files individually, move new files in, delete the old files. - For all the rename operations, handle them returning EXDEV error (a rename would cross a filesystem boundary) by falling back to copy and delete. As we expect that it is the cloned RocksDB log files that are most affected by this, the total copy I/O wouldn't be large, and so the default MySQL copy routine my_copy is used, without extra copy throughput optimizations. Add a fault injection point for MTR to test this code paths. - Remove any RocksDB clone checkpoints found. - On instance startup, always move WALs to the target directory if they are found in the rocksdb_datadir yet the rocksdb_wal_dir is different. This handles the case of setting rocksdb_wal_dir while instance is shutdown, both for clone fixup and regular operation. - Add PFS instrumentation for all the instrumentable entities in this patch (mutexes, rwlocks, file I/O). MyRocks Clone (8/n): MyRocks clone MTR tests - Add a new MTR test suite rocksdb_clone - Support new variable MTR clone/include/clone_command.inc: clone_single_threaded, setting which forces single-threaded clone operation. - Reuse a big part of InnoDB clone tests by including their test files, with the exception of tests involving DDL, as concurrent DDL is not supported by MyRocks clone at the moment, and debug tests that adjust cloned instance LSN, because they desynchronize storage engines. The reused tests are: local_basic, local_boundary, local_dml, local_dml_auto_tune, local_dml_no_redo, local_insert, local_stage, monitor_progress, remote_basic, remote_basic_replace, remote_boundary, remote_dml, remote_dml_auto_tune, remote_dml_gtid, remote_dml_no_binlog, remote_dml_no_ssl, remote_dml_recovery, remote_dml_replace, remote_dml_ssl, remote_dml_ssl_certificate, remote_dml_upgrade, remote_insert, remote_monitor_progress, remote_restart, remote_skip_validation, remote_stage, remote_stage_restart. - As a part of the above, add MyRocks support to local_basic and monitor_progress MTR tests. - Besides the reused InnoDB clone tests, add some new MyRocks-specific tests: - local_minimal_1_thread, which is the simplest possible local clone test. - local_minimal_multithreaded, the simplest possible but multithreaded local clone test. - local_minimal_direct_io, which is the simplest possible local clone test in all combinations of rocksdb_use_direct_reads and rocksdb_use_direct_io_for_flush_and_compaction settings, which are the O_DIRECT-enabling options. - local_dml_o_direct_donor, local_dml_o_direct_client, local_dml_o_direct_both that reuse InnoDB local_dml test, but combined with either or both of the O_DIRECT-enabling options. - local_minimal_wal_dir, which clones with various rocksdb_wal_dir settings on donor and client. - local_minimal_wal_dir_copy_delete, which uses a debug injection to test the move-across-filesystem-boundary code path in clone startup. - remote_minimal_direct_io_{1, 2, 3, 4} tests for all the combinations of O_DIRECT-enabling options for donor and client instances. - remote_fail_apply, which uses a debug injection to fail the clone client during remote clone. - remote_inplace_restart, which performs a remote in-place clone with a simulated transient error on the client. - remote_kill_donor, which kill the donor instance at various points of remote cloning. - rolling_checkpoint for testing various combinations of clone rolling checkpoint tuning options. - three_se_basic to test a simple clone across InnoDB, MyRocks, & Blackhole engines. Fix moving RocksDB WALs on startup across file system boundaries Reviewed By: sunshine-Chun Differential Revision: D43160162 Pulled By: hermanlee fbshipit-source-id: 5358a57 -------------------------------------------------------------------------------- Rdb clone fixes (facebook#1271) Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 2245706 -------------------------------------------------------------------------------- Clone: fix race condition that may skip some SST files (facebook#1274) Summary: A donor thread in the main copy loop took a file to send, released the donor state mutex, and then opened the file. If an ENOENT was received at this point, it was assumed that this was a stale SST file from an older checkpoint that was rolled since. Because the donor state mutex was released between taking of the file and opening it, the following race was possible: 1) Thread 1 takes the file 2) Thread 2 decides to roll the checkpoint, the old checkpoint is deleted 3) Thread 1 tries to open the file, gets ENOENT 4) Thread 2 creates the new checkpoint, the file re-appears, but it's too late. Rolling the checkpoint in a donor state mutex critical section is a possible fix, but such section would do a lot of I/O, serializing the parallel threads. Instead, fix by taking the file and opening it in the same critical section. Pull Request resolved: facebook#1274 Reviewed By: sunshine-Chun Differential Revision: D43629546 Pulled By: hermanlee fbshipit-source-id: 6e7f315 -------------------------------------------------------------------------------- MyRocks clone: add missing donor mutex unlocks on error exits (facebook#1317) Summary: Pull Request resolved: facebook#1317 Differential Revision: D46564079 fbshipit-source-id: 6fc1a4f
Summary: Pull Request resolved: facebook#1250 MyRocks Clone (1/n): make InnoDB clone MTR tests SE-agnostic The MyRocks clone MTR tests will reuse the InnoDB tests by including them. Adjust the tests so that they work this way, i.e. make them storage engine-agnostic by checking the default_storage_engine system variable and guarding InnoDB-specific tests this way. This includes the tests that artificially desync the InnoDB redo log position. MyRocks Clone (2/n): fix bug 109919 (Clone plugin does not support arbitrary length O_DIRECT files) The existing clone plugin I/O routines handle O_DIRECT files with the assumption that their sizes are always a multiple of a block size. This is too restrictive because in general O_DIRECT files can be of any size and MyRocks files actually are. Add a flag to Ha_clone_file to support arbitrary file lengths under O_DIRECT: o_direct_uneven_file_size. Pass it to clone_client.h/cc Client::get_aligned_buffer to allocate the buffer that bumps the length to the next CLONE_OS_ALIGN multiple if it's not already one. Introduce a pad_for_o_direct helper that performs this calculation for the passed length as needed. For writes, if O_DIRECT is enabled and the buffer length is not a multiple of CLONE_OS_ALIGN, pad it with zeros to the next multiple. Then check the actual bytes written, and if any padding bytes were actually written, call my_chsize to truncate the file to the needed size. MyRocks Clone (3/n): fix bug 109920 (SEs cannot add to clone data size estimates) In the current implementation, once the first SE sets the data size estimate for a particular stage, there is no way for other SEs to add to it. Address this by adding Ha_clone_cbk::add_to_data_size_estimate method. MyRocks Clone (4/n): fix bug 109921 (Cannot clone with synchronization across SEs) The existing clone implementation synchronizes InnoDB and binlog directly. There is no way to make other storage engines join this synchronization, thus replace it with synchronizing by a performance_schema.log_status query. - Add table_log_status::get_row method to allow doing the log_status query from inside the server. - Handlerton: introduce two new methods: clone_precopy - currently unused, which will handle any non-consistent-snapshot copying before the main consistent copy, and set_log_stop - which provides the SE the clone stop position information. - Plugin: introduce Ha_clone_common_cbk, which derives from Ha_clone_cbk, and from which derive all the concrete classes: Client_Cbk, Local_Cbk & Server_Cbk. - Ha_clone_common_cbk contains two methods: precopy and synchronize_engines. The former calls precopy for all the participating clone SE handlertons, if implemented. The latter, only to be called from the main clone thread, performs the performance_schema.LOG_STATUS query, splits its result per-SE and calls handlerton set_log_stop for all SEs. - The pfs.log_status query still must be called by InnoDB at a certain point in clone. That means that InnoDB is still a special SE in that it must always come first in the clone locator vectors. This will also handle donor waits on clone restarts, as that is implemented in InnoDB-specific code. To ensure this, add a helper clone_hton.cc:make_innodb_first which shuffles the vectors as needed. Call it in all the clone-starting code paths, add asserts that InnoDB comes first. - Since hton_clone_copy will call into InnoDB clone copy, which at certain point will synchronize engines, calling into the plugin again, and the calling into SEs again, introduce a backup locator vector Ha_clone_cbk::all_locators, and its setters/getters set_all_locators, reset_all_locators, get_all_locators. Use them in the re-entrant hton_clone_copy code. - Introduce a new InnoDB clone state named SST COPY, which calls precopy and has zero files and chunks set to copy in the regular InnoDB clone way, i.e. all it does it the precopy. - The redo log archiving keeps running throughout the precopy stage. - On stage transition from the precopy to the redo log copy. call synchronize. - Implement innodb_clone_set_log_stop by parsing out the desired clone stop LSN from the passed JSON from the log_status query. - Make InnoDB redo log archiving stop at that LSN instead of on reaching the end of redo log. - Support stopping redo log archiving in a middle of a log block. As the old implementation may encounter this case only at the end of the redo log, it took that block from the log buffer in memory. This no longer works with the new implementation, where the needed block might reside on disk only. Address this by forcing the whole block containing the target LSN to disk, reading it from there, and patching its memory copy to stop at the required LSN before giving it for the clone to copy. - To implement redo log block reading on a running server, adapt crash recovery function recv_read_log_seg to work both for crash recovery and on a running server. This is similar to what Percona Server does for its changed page tracking feature. - Make sure not to archive not-yet-written log: pfs.log_status query gets the log system LSN, which is the value of highest log reservation. Thus in an archiving iteration, archive up to the lower values of log archiving LSN, and write_lsn. Set the log archiving LSN to LSN_MAX at clone initialization, so that clone archiving may proceed until the stop LSN is set. - The above implementation results in cloned LSN != log end LSN for the cloned instance, thus adjust crash recovery to work with that. The existing crash recovery throws away any crash-recovered metadata and reads its snapshot from the data files, whereas the snapshot was not written in any consistency with the clone LSN. Fix by no longer throwing away the crash-recovered InnoDB dynamic metadata and by not re-reading the the last metadata snapshot. This forces also not to re-serialize the metadata in the crash recovery as that writes redo log which is to be avoided during clone crash recovery. - Additionally, at the start of the redo log copy the old implementation requested all InnoDB updates to be flushed to disk immediately, to avoid clone recovery generating new redo log. This is also disabled as it becomes redundant with the fix. This issue is similar to Percona XtraBackup https://jira.percona.com/browse/PXB-1634 - Disable clone.local_xa test because the new synchronization no longer blocks on XA transactions. - Re-record two clone-related tests in the component_keyring_file MTR suite. - Temporarily workaround the MyRocks bug where log_status query hangs if RocksDB file deletions were not disabled first. Note that this change breaks some of the clone-related group replication MTR tests. As they are not tested in fb-mysql and have been already broken, this patch makes no effort to fix them. MyRocks Clone (5/n): fix bug 109922 (SEs do not coordinate clone rollbacks on instance startup) If InnoDB clone decides to rollback during its startup, other storage engines are not aware of this and proceed as usual, resulting in inconsistent instance between storage engines. Ideally this should be fixed through a 2PC-like design where all the SEs vote to rollback the clone or to continue during the startup, but for MyRocks clone a simpler solution suffices: make InnoDB write a special marker file if it decides to rollback on clone startup. MyRocks Clone (6/n): on remote clone error, always send COM_ACK to InnoDB too In the case of multithreaded clone, the waits for the donor threads for the current clone stage to complete are implemented fully in InnoDB, even though that, and error handling, should be plugin-level features. With two storage engines participating in the clone this allows the situation where the other storage engine client reports an error with COM_ACK to one donor thread, while some other threads are in InnoDB clone state change waits. Those waits can abort if InnoDB clone gets an error from the client, but they know nothing about other SE errors, and will be aborted only by timeouts. As a minimal fix that does not involve pulling the whole error handling and state change waiting logic to the plugin layer, make the clone client always send an error-containing COM_ACK to InnoDB too, whenever it sends one to another storage engine. MyRocks Clone (7/n): MyRocks clone implementation - Cleanup any stale RocksDB checkpoint directories on server shutdown too. - For MyRocks clone network protocol, define four packet types (clone/common.h): - LOCATOR_V1: MyRocks-specific clone locator, consisting of a monotonically increasing clone session ID for the donor with an optional applier state (an array of file IDs and applied offsets) for restarts, or a zero in which case it is a version check empty locator used for handshakes. - FILE_NAME_MAP_V1: establishes a relation between a file ID, its name and target size. - FILE_CHUNK_V1: cloned file chunk data - ADD_ESTIMATE_V1: how many bytes should be added to the estimated clone data size. - All packet types share the common package header, which stores the version, the payload length without the header, and the packet type. - (clone/common.h): classes locator, file_metadata, chunk_metadata, & estimate_metadata correspond to the four packet types. They are subclassed on the donor side to add serialization routines and other donor-specific code and on the client side to add deserialization routines and other client-specific code. Re. serialization/deserialization, an exception is the locator, whose routines are implemented in the common code, because it may be passed both - ways. - (clone/common.h): Introduce a session class that handles the task ID set and errors, which is common for both donor and client. For errors, allow saving one error, with an optional associated path and errno, that can be set by any thread and later reported to the clone plugin. - (clone/common.cc) Setup MyRocks clone capabilities as HA_CLONE_HYBRID (not - applicable but matches InnoDB), HA_CLONE_MULTI_TASK (can clone with multiple - threads), & HA_CLONE_RESTART (can restart after a network I/O error). - (clone/common.cc) Enable O_DIRECT for clone file I/O if, on client if rocksdb_use_direct_io_for_flush_and_compaction is enabled, and on donor if rocksdb_use_direct_reads is enabled. - On the donor side (clone/donor.cc), encapsulate RocksDB checkpoints in a rdb_checkpoint class. Each checkpoint gets its own ID, which is also a donor session ID. The ID is constant for rolling checkpoints. - Subclass the common file_metadata to donor_file_metadata, chunk_metadata to donor_chunk_metadata, estimate_metadata to donor_estimate. - Implement donor going through a sequence of states: NONE, ROLLING, FINAL, FINAL_WITH_LOGS. To support multiple clone donor sessions, implement a manager class that looks up the donor instance by locator. - Implement MyRocks clone donor interface: - rocksdb_clone_start: handle the possible start modes: - HA_CLONE_MODE_START: create and register a new donor instance, register its master task ID. - HA_CLONE_MODE_ADD_TASK: lookup the existing donor instance, register a new task ID. - HA_CLONE_MODE_RESTART: lookup the existing donor instance, wait for the old master session to drop it by master task ID reference count, then process the locator applier state to reset the donor state. - rocksdb_clone_precopy: lookup the donor instance, create a RocksDB snapshot if state == NONE, otherwise send files by chunks and roll the checkpoint as needed. - rocksdb_clone_set_log_stop: add the WAL files to the clone file to copy set. - rocksdb_clone_copy: copy the clone files in any donor state. - rocksdb_clone_ack: acknowledge errors sent by client. - rocksdb_clone_end: for master task ID, stop accepting new worker join requests, wait until existing workers quit, and, in the case of restartable error, wait for any restarts to happen. Then decrement ref count for the master task ID (which will handle both the cases of restarted and not restarted) clone. For all other task IDs, simply unregister them. - On the client side (clone/client.cc), have file_in_progress class to track each individual file progress during application, and completed_file for fully-applied files. - Subclass the common file_metadata to client_file_metadata, implement its reading from the net buffer. Likewise for client_chunk_metadata subclassing chunk_metadata, and client_estimate for estimate_metadata. - TODO client class. - Implement MyRocks clone client interface: - rocksdb_clone_apply_begin: handle the possible start modes: - HA_CLONE_MODE_START: initialize the client instance for a new clone - HA_CLONE_MODE_ADD_TASK: register a new task ID for an ongoing clone - HA_CLONE_MODE_RESTART: restart the interrupted clone session - HA_CLONE_MODE_VERSION: prepare new version locator - rocksdb_clone_apply: if passed an error code, save it. Otherwise verify the passed buffer and dispatch ADD_ESTIMATE_V1, FILE_NAME_MAP_V1, & FILE_CHUNK_V1 packets accordingly. - rocksdb_clone_apply_end: for master task, wait for all other tasks to complete and finish the clone. For both master and non-master tasks unregister their IDs. - Add rocksdb_sys_vars tests for the new clone-related system variables. They are included in the current patch and not the MTR one in order not to break the existing rocksdb_sys_vars.all_vars test. - On instance startup, do clone fixup (clone/common.cc). It consists of: - Check whether InnoDB clone rollback marker is present. If yes, force rollback of MyRocks clone. - If the cloned temporary datadir contains clone-in-progress marker, assume half-completed clone and abort. - Move the cloned datadirs and WAL dirs in-place. For the datadir, do a directory-level rename with a save and later delete of the old datadir. For the WAL dir do not try to do parent directory-level renames because the mysqld process may not have permissions to delete that directory. Instead, rename all the old files individually, move new files in, delete the old files. - For all the rename operations, handle them returning EXDEV error (a rename would cross a filesystem boundary) by falling back to copy and delete. As we expect that it is the cloned RocksDB log files that are most affected by this, the total copy I/O wouldn't be large, and so the default MySQL copy routine my_copy is used, without extra copy throughput optimizations. Add a fault injection point for MTR to test this code paths. - Remove any RocksDB clone checkpoints found. - On instance startup, always move WALs to the target directory if they are found in the rocksdb_datadir yet the rocksdb_wal_dir is different. This handles the case of setting rocksdb_wal_dir while instance is shutdown, both for clone fixup and regular operation. - Add PFS instrumentation for all the instrumentable entities in this patch (mutexes, rwlocks, file I/O). MyRocks Clone (8/n): MyRocks clone MTR tests - Add a new MTR test suite rocksdb_clone - Support new variable MTR clone/include/clone_command.inc: clone_single_threaded, setting which forces single-threaded clone operation. - Reuse a big part of InnoDB clone tests by including their test files, with the exception of tests involving DDL, as concurrent DDL is not supported by MyRocks clone at the moment, and debug tests that adjust cloned instance LSN, because they desynchronize storage engines. The reused tests are: local_basic, local_boundary, local_dml, local_dml_auto_tune, local_dml_no_redo, local_insert, local_stage, monitor_progress, remote_basic, remote_basic_replace, remote_boundary, remote_dml, remote_dml_auto_tune, remote_dml_gtid, remote_dml_no_binlog, remote_dml_no_ssl, remote_dml_recovery, remote_dml_replace, remote_dml_ssl, remote_dml_ssl_certificate, remote_dml_upgrade, remote_insert, remote_monitor_progress, remote_restart, remote_skip_validation, remote_stage, remote_stage_restart. - As a part of the above, add MyRocks support to local_basic and monitor_progress MTR tests. - Besides the reused InnoDB clone tests, add some new MyRocks-specific tests: - local_minimal_1_thread, which is the simplest possible local clone test. - local_minimal_multithreaded, the simplest possible but multithreaded local clone test. - local_minimal_direct_io, which is the simplest possible local clone test in all combinations of rocksdb_use_direct_reads and rocksdb_use_direct_io_for_flush_and_compaction settings, which are the O_DIRECT-enabling options. - local_dml_o_direct_donor, local_dml_o_direct_client, local_dml_o_direct_both that reuse InnoDB local_dml test, but combined with either or both of the O_DIRECT-enabling options. - local_minimal_wal_dir, which clones with various rocksdb_wal_dir settings on donor and client. - local_minimal_wal_dir_copy_delete, which uses a debug injection to test the move-across-filesystem-boundary code path in clone startup. - remote_minimal_direct_io_{1, 2, 3, 4} tests for all the combinations of O_DIRECT-enabling options for donor and client instances. - remote_fail_apply, which uses a debug injection to fail the clone client during remote clone. - remote_inplace_restart, which performs a remote in-place clone with a simulated transient error on the client. - remote_kill_donor, which kill the donor instance at various points of remote cloning. - rolling_checkpoint for testing various combinations of clone rolling checkpoint tuning options. - three_se_basic to test a simple clone across InnoDB, MyRocks, & Blackhole engines. Fix moving RocksDB WALs on startup across file system boundaries Reviewed By: sunshine-Chun Differential Revision: D43160162 Pulled By: hermanlee fbshipit-source-id: 5358a57 -------------------------------------------------------------------------------- Rdb clone fixes (facebook#1271) Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 2245706 -------------------------------------------------------------------------------- Clone: fix race condition that may skip some SST files (facebook#1274) Summary: A donor thread in the main copy loop took a file to send, released the donor state mutex, and then opened the file. If an ENOENT was received at this point, it was assumed that this was a stale SST file from an older checkpoint that was rolled since. Because the donor state mutex was released between taking of the file and opening it, the following race was possible: 1) Thread 1 takes the file 2) Thread 2 decides to roll the checkpoint, the old checkpoint is deleted 3) Thread 1 tries to open the file, gets ENOENT 4) Thread 2 creates the new checkpoint, the file re-appears, but it's too late. Rolling the checkpoint in a donor state mutex critical section is a possible fix, but such section would do a lot of I/O, serializing the parallel threads. Instead, fix by taking the file and opening it in the same critical section. Pull Request resolved: facebook#1274 Reviewed By: sunshine-Chun Differential Revision: D43629546 Pulled By: hermanlee fbshipit-source-id: 6e7f315 -------------------------------------------------------------------------------- MyRocks clone: add missing donor mutex unlocks on error exits (facebook#1317) Summary: Pull Request resolved: facebook#1317 Differential Revision: D46564079 fbshipit-source-id: 6fc1a4f
Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook/mysql-5.6#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook/mysql-5.6#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook/mysql-5.6#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook/mysql-5.6#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook/mysql-5.6#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook/mysql-5.6#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook/mysql-5.6#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook/mysql-5.6#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook/mysql-5.6#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook/mysql-5.6#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook/mysql-5.6#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook/mysql-5.6#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook/mysql-5.6#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook/mysql-5.6#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook/mysql-5.6#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook/mysql-5.6#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook/mysql-5.6#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook/mysql-5.6#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook/mysql-5.6#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook/mysql-5.6#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook/mysql-5.6#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook/mysql-5.6#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook/mysql-5.6#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook/mysql-5.6#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook/mysql-5.6#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook/mysql-5.6#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook/mysql-5.6#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook/mysql-5.6#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook/mysql-5.6#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook/mysql-5.6#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
Summary: Pull Request resolved: facebook#1250 MyRocks Clone (1/n): make InnoDB clone MTR tests SE-agnostic The MyRocks clone MTR tests will reuse the InnoDB tests by including them. Adjust the tests so that they work this way, i.e. make them storage engine-agnostic by checking the default_storage_engine system variable and guarding InnoDB-specific tests this way. This includes the tests that artificially desync the InnoDB redo log position. MyRocks Clone (2/n): fix bug 109919 (Clone plugin does not support arbitrary length O_DIRECT files) The existing clone plugin I/O routines handle O_DIRECT files with the assumption that their sizes are always a multiple of a block size. This is too restrictive because in general O_DIRECT files can be of any size and MyRocks files actually are. Add a flag to Ha_clone_file to support arbitrary file lengths under O_DIRECT: o_direct_uneven_file_size. Pass it to clone_client.h/cc Client::get_aligned_buffer to allocate the buffer that bumps the length to the next CLONE_OS_ALIGN multiple if it's not already one. Introduce a pad_for_o_direct helper that performs this calculation for the passed length as needed. For writes, if O_DIRECT is enabled and the buffer length is not a multiple of CLONE_OS_ALIGN, pad it with zeros to the next multiple. Then check the actual bytes written, and if any padding bytes were actually written, call my_chsize to truncate the file to the needed size. MyRocks Clone (3/n): fix bug 109920 (SEs cannot add to clone data size estimates) In the current implementation, once the first SE sets the data size estimate for a particular stage, there is no way for other SEs to add to it. Address this by adding Ha_clone_cbk::add_to_data_size_estimate method. MyRocks Clone (4/n): fix bug 109921 (Cannot clone with synchronization across SEs) The existing clone implementation synchronizes InnoDB and binlog directly. There is no way to make other storage engines join this synchronization, thus replace it with synchronizing by a performance_schema.log_status query. - Add table_log_status::get_row method to allow doing the log_status query from inside the server. - Handlerton: introduce two new methods: clone_precopy - currently unused, which will handle any non-consistent-snapshot copying before the main consistent copy, and set_log_stop - which provides the SE the clone stop position information. - Plugin: introduce Ha_clone_common_cbk, which derives from Ha_clone_cbk, and from which derive all the concrete classes: Client_Cbk, Local_Cbk & Server_Cbk. - Ha_clone_common_cbk contains two methods: precopy and synchronize_engines. The former calls precopy for all the participating clone SE handlertons, if implemented. The latter, only to be called from the main clone thread, performs the performance_schema.LOG_STATUS query, splits its result per-SE and calls handlerton set_log_stop for all SEs. - The pfs.log_status query still must be called by InnoDB at a certain point in clone. That means that InnoDB is still a special SE in that it must always come first in the clone locator vectors. This will also handle donor waits on clone restarts, as that is implemented in InnoDB-specific code. To ensure this, add a helper clone_hton.cc:make_innodb_first which shuffles the vectors as needed. Call it in all the clone-starting code paths, add asserts that InnoDB comes first. - Since hton_clone_copy will call into InnoDB clone copy, which at certain point will synchronize engines, calling into the plugin again, and the calling into SEs again, introduce a backup locator vector Ha_clone_cbk::all_locators, and its setters/getters set_all_locators, reset_all_locators, get_all_locators. Use them in the re-entrant hton_clone_copy code. - Introduce a new InnoDB clone state named SST COPY, which calls precopy and has zero files and chunks set to copy in the regular InnoDB clone way, i.e. all it does it the precopy. - The redo log archiving keeps running throughout the precopy stage. - On stage transition from the precopy to the redo log copy. call synchronize. - Implement innodb_clone_set_log_stop by parsing out the desired clone stop LSN from the passed JSON from the log_status query. - Make InnoDB redo log archiving stop at that LSN instead of on reaching the end of redo log. - Support stopping redo log archiving in a middle of a log block. As the old implementation may encounter this case only at the end of the redo log, it took that block from the log buffer in memory. This no longer works with the new implementation, where the needed block might reside on disk only. Address this by forcing the whole block containing the target LSN to disk, reading it from there, and patching its memory copy to stop at the required LSN before giving it for the clone to copy. - To implement redo log block reading on a running server, adapt crash recovery function recv_read_log_seg to work both for crash recovery and on a running server. This is similar to what Percona Server does for its changed page tracking feature. - Make sure not to archive not-yet-written log: pfs.log_status query gets the log system LSN, which is the value of highest log reservation. Thus in an archiving iteration, archive up to the lower values of log archiving LSN, and write_lsn. Set the log archiving LSN to LSN_MAX at clone initialization, so that clone archiving may proceed until the stop LSN is set. - The above implementation results in cloned LSN != log end LSN for the cloned instance, thus adjust crash recovery to work with that. The existing crash recovery throws away any crash-recovered metadata and reads its snapshot from the data files, whereas the snapshot was not written in any consistency with the clone LSN. Fix by no longer throwing away the crash-recovered InnoDB dynamic metadata and by not re-reading the the last metadata snapshot. This forces also not to re-serialize the metadata in the crash recovery as that writes redo log which is to be avoided during clone crash recovery. - Additionally, at the start of the redo log copy the old implementation requested all InnoDB updates to be flushed to disk immediately, to avoid clone recovery generating new redo log. This is also disabled as it becomes redundant with the fix. This issue is similar to Percona XtraBackup https://jira.percona.com/browse/PXB-1634 - Disable clone.local_xa test because the new synchronization no longer blocks on XA transactions. - Re-record two clone-related tests in the component_keyring_file MTR suite. - Temporarily workaround the MyRocks bug where log_status query hangs if RocksDB file deletions were not disabled first. Note that this change breaks some of the clone-related group replication MTR tests. As they are not tested in fb-mysql and have been already broken, this patch makes no effort to fix them. MyRocks Clone (5/n): fix bug 109922 (SEs do not coordinate clone rollbacks on instance startup) If InnoDB clone decides to rollback during its startup, other storage engines are not aware of this and proceed as usual, resulting in inconsistent instance between storage engines. Ideally this should be fixed through a 2PC-like design where all the SEs vote to rollback the clone or to continue during the startup, but for MyRocks clone a simpler solution suffices: make InnoDB write a special marker file if it decides to rollback on clone startup. MyRocks Clone (6/n): on remote clone error, always send COM_ACK to InnoDB too In the case of multithreaded clone, the waits for the donor threads for the current clone stage to complete are implemented fully in InnoDB, even though that, and error handling, should be plugin-level features. With two storage engines participating in the clone this allows the situation where the other storage engine client reports an error with COM_ACK to one donor thread, while some other threads are in InnoDB clone state change waits. Those waits can abort if InnoDB clone gets an error from the client, but they know nothing about other SE errors, and will be aborted only by timeouts. As a minimal fix that does not involve pulling the whole error handling and state change waiting logic to the plugin layer, make the clone client always send an error-containing COM_ACK to InnoDB too, whenever it sends one to another storage engine. MyRocks Clone (7/n): MyRocks clone implementation - Cleanup any stale RocksDB checkpoint directories on server shutdown too. - For MyRocks clone network protocol, define four packet types (clone/common.h): - LOCATOR_V1: MyRocks-specific clone locator, consisting of a monotonically increasing clone session ID for the donor with an optional applier state (an array of file IDs and applied offsets) for restarts, or a zero in which case it is a version check empty locator used for handshakes. - FILE_NAME_MAP_V1: establishes a relation between a file ID, its name and target size. - FILE_CHUNK_V1: cloned file chunk data - ADD_ESTIMATE_V1: how many bytes should be added to the estimated clone data size. - All packet types share the common package header, which stores the version, the payload length without the header, and the packet type. - (clone/common.h): classes locator, file_metadata, chunk_metadata, & estimate_metadata correspond to the four packet types. They are subclassed on the donor side to add serialization routines and other donor-specific code and on the client side to add deserialization routines and other client-specific code. Re. serialization/deserialization, an exception is the locator, whose routines are implemented in the common code, because it may be passed both - ways. - (clone/common.h): Introduce a session class that handles the task ID set and errors, which is common for both donor and client. For errors, allow saving one error, with an optional associated path and errno, that can be set by any thread and later reported to the clone plugin. - (clone/common.cc) Setup MyRocks clone capabilities as HA_CLONE_HYBRID (not - applicable but matches InnoDB), HA_CLONE_MULTI_TASK (can clone with multiple - threads), & HA_CLONE_RESTART (can restart after a network I/O error). - (clone/common.cc) Enable O_DIRECT for clone file I/O if, on client if rocksdb_use_direct_io_for_flush_and_compaction is enabled, and on donor if rocksdb_use_direct_reads is enabled. - On the donor side (clone/donor.cc), encapsulate RocksDB checkpoints in a rdb_checkpoint class. Each checkpoint gets its own ID, which is also a donor session ID. The ID is constant for rolling checkpoints. - Subclass the common file_metadata to donor_file_metadata, chunk_metadata to donor_chunk_metadata, estimate_metadata to donor_estimate. - Implement donor going through a sequence of states: NONE, ROLLING, FINAL, FINAL_WITH_LOGS. To support multiple clone donor sessions, implement a manager class that looks up the donor instance by locator. - Implement MyRocks clone donor interface: - rocksdb_clone_start: handle the possible start modes: - HA_CLONE_MODE_START: create and register a new donor instance, register its master task ID. - HA_CLONE_MODE_ADD_TASK: lookup the existing donor instance, register a new task ID. - HA_CLONE_MODE_RESTART: lookup the existing donor instance, wait for the old master session to drop it by master task ID reference count, then process the locator applier state to reset the donor state. - rocksdb_clone_precopy: lookup the donor instance, create a RocksDB snapshot if state == NONE, otherwise send files by chunks and roll the checkpoint as needed. - rocksdb_clone_set_log_stop: add the WAL files to the clone file to copy set. - rocksdb_clone_copy: copy the clone files in any donor state. - rocksdb_clone_ack: acknowledge errors sent by client. - rocksdb_clone_end: for master task ID, stop accepting new worker join requests, wait until existing workers quit, and, in the case of restartable error, wait for any restarts to happen. Then decrement ref count for the master task ID (which will handle both the cases of restarted and not restarted) clone. For all other task IDs, simply unregister them. - On the client side (clone/client.cc), have file_in_progress class to track each individual file progress during application, and completed_file for fully-applied files. - Subclass the common file_metadata to client_file_metadata, implement its reading from the net buffer. Likewise for client_chunk_metadata subclassing chunk_metadata, and client_estimate for estimate_metadata. - TODO client class. - Implement MyRocks clone client interface: - rocksdb_clone_apply_begin: handle the possible start modes: - HA_CLONE_MODE_START: initialize the client instance for a new clone - HA_CLONE_MODE_ADD_TASK: register a new task ID for an ongoing clone - HA_CLONE_MODE_RESTART: restart the interrupted clone session - HA_CLONE_MODE_VERSION: prepare new version locator - rocksdb_clone_apply: if passed an error code, save it. Otherwise verify the passed buffer and dispatch ADD_ESTIMATE_V1, FILE_NAME_MAP_V1, & FILE_CHUNK_V1 packets accordingly. - rocksdb_clone_apply_end: for master task, wait for all other tasks to complete and finish the clone. For both master and non-master tasks unregister their IDs. - Add rocksdb_sys_vars tests for the new clone-related system variables. They are included in the current patch and not the MTR one in order not to break the existing rocksdb_sys_vars.all_vars test. - On instance startup, do clone fixup (clone/common.cc). It consists of: - Check whether InnoDB clone rollback marker is present. If yes, force rollback of MyRocks clone. - If the cloned temporary datadir contains clone-in-progress marker, assume half-completed clone and abort. - Move the cloned datadirs and WAL dirs in-place. For the datadir, do a directory-level rename with a save and later delete of the old datadir. For the WAL dir do not try to do parent directory-level renames because the mysqld process may not have permissions to delete that directory. Instead, rename all the old files individually, move new files in, delete the old files. - For all the rename operations, handle them returning EXDEV error (a rename would cross a filesystem boundary) by falling back to copy and delete. As we expect that it is the cloned RocksDB log files that are most affected by this, the total copy I/O wouldn't be large, and so the default MySQL copy routine my_copy is used, without extra copy throughput optimizations. Add a fault injection point for MTR to test this code paths. - Remove any RocksDB clone checkpoints found. - On instance startup, always move WALs to the target directory if they are found in the rocksdb_datadir yet the rocksdb_wal_dir is different. This handles the case of setting rocksdb_wal_dir while instance is shutdown, both for clone fixup and regular operation. - Add PFS instrumentation for all the instrumentable entities in this patch (mutexes, rwlocks, file I/O). MyRocks Clone (8/n): MyRocks clone MTR tests - Add a new MTR test suite rocksdb_clone - Support new variable MTR clone/include/clone_command.inc: clone_single_threaded, setting which forces single-threaded clone operation. - Reuse a big part of InnoDB clone tests by including their test files, with the exception of tests involving DDL, as concurrent DDL is not supported by MyRocks clone at the moment, and debug tests that adjust cloned instance LSN, because they desynchronize storage engines. The reused tests are: local_basic, local_boundary, local_dml, local_dml_auto_tune, local_dml_no_redo, local_insert, local_stage, monitor_progress, remote_basic, remote_basic_replace, remote_boundary, remote_dml, remote_dml_auto_tune, remote_dml_gtid, remote_dml_no_binlog, remote_dml_no_ssl, remote_dml_recovery, remote_dml_replace, remote_dml_ssl, remote_dml_ssl_certificate, remote_dml_upgrade, remote_insert, remote_monitor_progress, remote_restart, remote_skip_validation, remote_stage, remote_stage_restart. - As a part of the above, add MyRocks support to local_basic and monitor_progress MTR tests. - Besides the reused InnoDB clone tests, add some new MyRocks-specific tests: - local_minimal_1_thread, which is the simplest possible local clone test. - local_minimal_multithreaded, the simplest possible but multithreaded local clone test. - local_minimal_direct_io, which is the simplest possible local clone test in all combinations of rocksdb_use_direct_reads and rocksdb_use_direct_io_for_flush_and_compaction settings, which are the O_DIRECT-enabling options. - local_dml_o_direct_donor, local_dml_o_direct_client, local_dml_o_direct_both that reuse InnoDB local_dml test, but combined with either or both of the O_DIRECT-enabling options. - local_minimal_wal_dir, which clones with various rocksdb_wal_dir settings on donor and client. - local_minimal_wal_dir_copy_delete, which uses a debug injection to test the move-across-filesystem-boundary code path in clone startup. - remote_minimal_direct_io_{1, 2, 3, 4} tests for all the combinations of O_DIRECT-enabling options for donor and client instances. - remote_fail_apply, which uses a debug injection to fail the clone client during remote clone. - remote_inplace_restart, which performs a remote in-place clone with a simulated transient error on the client. - remote_kill_donor, which kill the donor instance at various points of remote cloning. - rolling_checkpoint for testing various combinations of clone rolling checkpoint tuning options. - three_se_basic to test a simple clone across InnoDB, MyRocks, & Blackhole engines. Fix moving RocksDB WALs on startup across file system boundaries Reviewed By: sunshine-Chun Differential Revision: D43160162 Pulled By: hermanlee -------------------------------------------------------------------------------- Rdb clone fixes (facebook#1271) Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee -------------------------------------------------------------------------------- Clone: fix race condition that may skip some SST files (facebook#1274) Summary: A donor thread in the main copy loop took a file to send, released the donor state mutex, and then opened the file. If an ENOENT was received at this point, it was assumed that this was a stale SST file from an older checkpoint that was rolled since. Because the donor state mutex was released between taking of the file and opening it, the following race was possible: 1) Thread 1 takes the file 2) Thread 2 decides to roll the checkpoint, the old checkpoint is deleted 3) Thread 1 tries to open the file, gets ENOENT 4) Thread 2 creates the new checkpoint, the file re-appears, but it's too late. Rolling the checkpoint in a donor state mutex critical section is a possible fix, but such section would do a lot of I/O, serializing the parallel threads. Instead, fix by taking the file and opening it in the same critical section. Pull Request resolved: facebook#1274 Reviewed By: sunshine-Chun Differential Revision: D43629546 Pulled By: hermanlee -------------------------------------------------------------------------------- MyRocks clone: add missing donor mutex unlocks on error exits (facebook#1317) Summary: Pull Request resolved: facebook#1317 Differential Revision: D46564079
Summary: Pull Request resolved: facebook#1250 MyRocks Clone (1/n): make InnoDB clone MTR tests SE-agnostic The MyRocks clone MTR tests will reuse the InnoDB tests by including them. Adjust the tests so that they work this way, i.e. make them storage engine-agnostic by checking the default_storage_engine system variable and guarding InnoDB-specific tests this way. This includes the tests that artificially desync the InnoDB redo log position. MyRocks Clone (2/n): fix bug 109919 (Clone plugin does not support arbitrary length O_DIRECT files) The existing clone plugin I/O routines handle O_DIRECT files with the assumption that their sizes are always a multiple of a block size. This is too restrictive because in general O_DIRECT files can be of any size and MyRocks files actually are. Add a flag to Ha_clone_file to support arbitrary file lengths under O_DIRECT: o_direct_uneven_file_size. Pass it to clone_client.h/cc Client::get_aligned_buffer to allocate the buffer that bumps the length to the next CLONE_OS_ALIGN multiple if it's not already one. Introduce a pad_for_o_direct helper that performs this calculation for the passed length as needed. For writes, if O_DIRECT is enabled and the buffer length is not a multiple of CLONE_OS_ALIGN, pad it with zeros to the next multiple. Then check the actual bytes written, and if any padding bytes were actually written, call my_chsize to truncate the file to the needed size. MyRocks Clone (3/n): fix bug 109920 (SEs cannot add to clone data size estimates) In the current implementation, once the first SE sets the data size estimate for a particular stage, there is no way for other SEs to add to it. Address this by adding Ha_clone_cbk::add_to_data_size_estimate method. MyRocks Clone (4/n): fix bug 109921 (Cannot clone with synchronization across SEs) The existing clone implementation synchronizes InnoDB and binlog directly. There is no way to make other storage engines join this synchronization, thus replace it with synchronizing by a performance_schema.log_status query. - Add table_log_status::get_row method to allow doing the log_status query from inside the server. - Handlerton: introduce two new methods: clone_precopy - currently unused, which will handle any non-consistent-snapshot copying before the main consistent copy, and set_log_stop - which provides the SE the clone stop position information. - Plugin: introduce Ha_clone_common_cbk, which derives from Ha_clone_cbk, and from which derive all the concrete classes: Client_Cbk, Local_Cbk & Server_Cbk. - Ha_clone_common_cbk contains two methods: precopy and synchronize_engines. The former calls precopy for all the participating clone SE handlertons, if implemented. The latter, only to be called from the main clone thread, performs the performance_schema.LOG_STATUS query, splits its result per-SE and calls handlerton set_log_stop for all SEs. - The pfs.log_status query still must be called by InnoDB at a certain point in clone. That means that InnoDB is still a special SE in that it must always come first in the clone locator vectors. This will also handle donor waits on clone restarts, as that is implemented in InnoDB-specific code. To ensure this, add a helper clone_hton.cc:make_innodb_first which shuffles the vectors as needed. Call it in all the clone-starting code paths, add asserts that InnoDB comes first. - Since hton_clone_copy will call into InnoDB clone copy, which at certain point will synchronize engines, calling into the plugin again, and the calling into SEs again, introduce a backup locator vector Ha_clone_cbk::all_locators, and its setters/getters set_all_locators, reset_all_locators, get_all_locators. Use them in the re-entrant hton_clone_copy code. - Introduce a new InnoDB clone state named SST COPY, which calls precopy and has zero files and chunks set to copy in the regular InnoDB clone way, i.e. all it does it the precopy. - The redo log archiving keeps running throughout the precopy stage. - On stage transition from the precopy to the redo log copy. call synchronize. - Implement innodb_clone_set_log_stop by parsing out the desired clone stop LSN from the passed JSON from the log_status query. - Make InnoDB redo log archiving stop at that LSN instead of on reaching the end of redo log. - Support stopping redo log archiving in a middle of a log block. As the old implementation may encounter this case only at the end of the redo log, it took that block from the log buffer in memory. This no longer works with the new implementation, where the needed block might reside on disk only. Address this by forcing the whole block containing the target LSN to disk, reading it from there, and patching its memory copy to stop at the required LSN before giving it for the clone to copy. - To implement redo log block reading on a running server, adapt crash recovery function recv_read_log_seg to work both for crash recovery and on a running server. This is similar to what Percona Server does for its changed page tracking feature. - Make sure not to archive not-yet-written log: pfs.log_status query gets the log system LSN, which is the value of highest log reservation. Thus in an archiving iteration, archive up to the lower values of log archiving LSN, and write_lsn. Set the log archiving LSN to LSN_MAX at clone initialization, so that clone archiving may proceed until the stop LSN is set. - The above implementation results in cloned LSN != log end LSN for the cloned instance, thus adjust crash recovery to work with that. The existing crash recovery throws away any crash-recovered metadata and reads its snapshot from the data files, whereas the snapshot was not written in any consistency with the clone LSN. Fix by no longer throwing away the crash-recovered InnoDB dynamic metadata and by not re-reading the the last metadata snapshot. This forces also not to re-serialize the metadata in the crash recovery as that writes redo log which is to be avoided during clone crash recovery. - Additionally, at the start of the redo log copy the old implementation requested all InnoDB updates to be flushed to disk immediately, to avoid clone recovery generating new redo log. This is also disabled as it becomes redundant with the fix. This issue is similar to Percona XtraBackup https://jira.percona.com/browse/PXB-1634 - Disable clone.local_xa test because the new synchronization no longer blocks on XA transactions. - Re-record two clone-related tests in the component_keyring_file MTR suite. - Temporarily workaround the MyRocks bug where log_status query hangs if RocksDB file deletions were not disabled first. Note that this change breaks some of the clone-related group replication MTR tests. As they are not tested in fb-mysql and have been already broken, this patch makes no effort to fix them. MyRocks Clone (5/n): fix bug 109922 (SEs do not coordinate clone rollbacks on instance startup) If InnoDB clone decides to rollback during its startup, other storage engines are not aware of this and proceed as usual, resulting in inconsistent instance between storage engines. Ideally this should be fixed through a 2PC-like design where all the SEs vote to rollback the clone or to continue during the startup, but for MyRocks clone a simpler solution suffices: make InnoDB write a special marker file if it decides to rollback on clone startup. MyRocks Clone (6/n): on remote clone error, always send COM_ACK to InnoDB too In the case of multithreaded clone, the waits for the donor threads for the current clone stage to complete are implemented fully in InnoDB, even though that, and error handling, should be plugin-level features. With two storage engines participating in the clone this allows the situation where the other storage engine client reports an error with COM_ACK to one donor thread, while some other threads are in InnoDB clone state change waits. Those waits can abort if InnoDB clone gets an error from the client, but they know nothing about other SE errors, and will be aborted only by timeouts. As a minimal fix that does not involve pulling the whole error handling and state change waiting logic to the plugin layer, make the clone client always send an error-containing COM_ACK to InnoDB too, whenever it sends one to another storage engine. MyRocks Clone (7/n): MyRocks clone implementation - Cleanup any stale RocksDB checkpoint directories on server shutdown too. - For MyRocks clone network protocol, define four packet types (clone/common.h): - LOCATOR_V1: MyRocks-specific clone locator, consisting of a monotonically increasing clone session ID for the donor with an optional applier state (an array of file IDs and applied offsets) for restarts, or a zero in which case it is a version check empty locator used for handshakes. - FILE_NAME_MAP_V1: establishes a relation between a file ID, its name and target size. - FILE_CHUNK_V1: cloned file chunk data - ADD_ESTIMATE_V1: how many bytes should be added to the estimated clone data size. - All packet types share the common package header, which stores the version, the payload length without the header, and the packet type. - (clone/common.h): classes locator, file_metadata, chunk_metadata, & estimate_metadata correspond to the four packet types. They are subclassed on the donor side to add serialization routines and other donor-specific code and on the client side to add deserialization routines and other client-specific code. Re. serialization/deserialization, an exception is the locator, whose routines are implemented in the common code, because it may be passed both - ways. - (clone/common.h): Introduce a session class that handles the task ID set and errors, which is common for both donor and client. For errors, allow saving one error, with an optional associated path and errno, that can be set by any thread and later reported to the clone plugin. - (clone/common.cc) Setup MyRocks clone capabilities as HA_CLONE_HYBRID (not - applicable but matches InnoDB), HA_CLONE_MULTI_TASK (can clone with multiple - threads), & HA_CLONE_RESTART (can restart after a network I/O error). - (clone/common.cc) Enable O_DIRECT for clone file I/O if, on client if rocksdb_use_direct_io_for_flush_and_compaction is enabled, and on donor if rocksdb_use_direct_reads is enabled. - On the donor side (clone/donor.cc), encapsulate RocksDB checkpoints in a rdb_checkpoint class. Each checkpoint gets its own ID, which is also a donor session ID. The ID is constant for rolling checkpoints. - Subclass the common file_metadata to donor_file_metadata, chunk_metadata to donor_chunk_metadata, estimate_metadata to donor_estimate. - Implement donor going through a sequence of states: NONE, ROLLING, FINAL, FINAL_WITH_LOGS. To support multiple clone donor sessions, implement a manager class that looks up the donor instance by locator. - Implement MyRocks clone donor interface: - rocksdb_clone_start: handle the possible start modes: - HA_CLONE_MODE_START: create and register a new donor instance, register its master task ID. - HA_CLONE_MODE_ADD_TASK: lookup the existing donor instance, register a new task ID. - HA_CLONE_MODE_RESTART: lookup the existing donor instance, wait for the old master session to drop it by master task ID reference count, then process the locator applier state to reset the donor state. - rocksdb_clone_precopy: lookup the donor instance, create a RocksDB snapshot if state == NONE, otherwise send files by chunks and roll the checkpoint as needed. - rocksdb_clone_set_log_stop: add the WAL files to the clone file to copy set. - rocksdb_clone_copy: copy the clone files in any donor state. - rocksdb_clone_ack: acknowledge errors sent by client. - rocksdb_clone_end: for master task ID, stop accepting new worker join requests, wait until existing workers quit, and, in the case of restartable error, wait for any restarts to happen. Then decrement ref count for the master task ID (which will handle both the cases of restarted and not restarted) clone. For all other task IDs, simply unregister them. - On the client side (clone/client.cc), have file_in_progress class to track each individual file progress during application, and completed_file for fully-applied files. - Subclass the common file_metadata to client_file_metadata, implement its reading from the net buffer. Likewise for client_chunk_metadata subclassing chunk_metadata, and client_estimate for estimate_metadata. - TODO client class. - Implement MyRocks clone client interface: - rocksdb_clone_apply_begin: handle the possible start modes: - HA_CLONE_MODE_START: initialize the client instance for a new clone - HA_CLONE_MODE_ADD_TASK: register a new task ID for an ongoing clone - HA_CLONE_MODE_RESTART: restart the interrupted clone session - HA_CLONE_MODE_VERSION: prepare new version locator - rocksdb_clone_apply: if passed an error code, save it. Otherwise verify the passed buffer and dispatch ADD_ESTIMATE_V1, FILE_NAME_MAP_V1, & FILE_CHUNK_V1 packets accordingly. - rocksdb_clone_apply_end: for master task, wait for all other tasks to complete and finish the clone. For both master and non-master tasks unregister their IDs. - Add rocksdb_sys_vars tests for the new clone-related system variables. They are included in the current patch and not the MTR one in order not to break the existing rocksdb_sys_vars.all_vars test. - On instance startup, do clone fixup (clone/common.cc). It consists of: - Check whether InnoDB clone rollback marker is present. If yes, force rollback of MyRocks clone. - If the cloned temporary datadir contains clone-in-progress marker, assume half-completed clone and abort. - Move the cloned datadirs and WAL dirs in-place. For the datadir, do a directory-level rename with a save and later delete of the old datadir. For the WAL dir do not try to do parent directory-level renames because the mysqld process may not have permissions to delete that directory. Instead, rename all the old files individually, move new files in, delete the old files. - For all the rename operations, handle them returning EXDEV error (a rename would cross a filesystem boundary) by falling back to copy and delete. As we expect that it is the cloned RocksDB log files that are most affected by this, the total copy I/O wouldn't be large, and so the default MySQL copy routine my_copy is used, without extra copy throughput optimizations. Add a fault injection point for MTR to test this code paths. - Remove any RocksDB clone checkpoints found. - On instance startup, always move WALs to the target directory if they are found in the rocksdb_datadir yet the rocksdb_wal_dir is different. This handles the case of setting rocksdb_wal_dir while instance is shutdown, both for clone fixup and regular operation. - Add PFS instrumentation for all the instrumentable entities in this patch (mutexes, rwlocks, file I/O). MyRocks Clone (8/n): MyRocks clone MTR tests - Add a new MTR test suite rocksdb_clone - Support new variable MTR clone/include/clone_command.inc: clone_single_threaded, setting which forces single-threaded clone operation. - Reuse a big part of InnoDB clone tests by including their test files, with the exception of tests involving DDL, as concurrent DDL is not supported by MyRocks clone at the moment, and debug tests that adjust cloned instance LSN, because they desynchronize storage engines. The reused tests are: local_basic, local_boundary, local_dml, local_dml_auto_tune, local_dml_no_redo, local_insert, local_stage, monitor_progress, remote_basic, remote_basic_replace, remote_boundary, remote_dml, remote_dml_auto_tune, remote_dml_gtid, remote_dml_no_binlog, remote_dml_no_ssl, remote_dml_recovery, remote_dml_replace, remote_dml_ssl, remote_dml_ssl_certificate, remote_dml_upgrade, remote_insert, remote_monitor_progress, remote_restart, remote_skip_validation, remote_stage, remote_stage_restart. - As a part of the above, add MyRocks support to local_basic and monitor_progress MTR tests. - Besides the reused InnoDB clone tests, add some new MyRocks-specific tests: - local_minimal_1_thread, which is the simplest possible local clone test. - local_minimal_multithreaded, the simplest possible but multithreaded local clone test. - local_minimal_direct_io, which is the simplest possible local clone test in all combinations of rocksdb_use_direct_reads and rocksdb_use_direct_io_for_flush_and_compaction settings, which are the O_DIRECT-enabling options. - local_dml_o_direct_donor, local_dml_o_direct_client, local_dml_o_direct_both that reuse InnoDB local_dml test, but combined with either or both of the O_DIRECT-enabling options. - local_minimal_wal_dir, which clones with various rocksdb_wal_dir settings on donor and client. - local_minimal_wal_dir_copy_delete, which uses a debug injection to test the move-across-filesystem-boundary code path in clone startup. - remote_minimal_direct_io_{1, 2, 3, 4} tests for all the combinations of O_DIRECT-enabling options for donor and client instances. - remote_fail_apply, which uses a debug injection to fail the clone client during remote clone. - remote_inplace_restart, which performs a remote in-place clone with a simulated transient error on the client. - remote_kill_donor, which kill the donor instance at various points of remote cloning. - rolling_checkpoint for testing various combinations of clone rolling checkpoint tuning options. - three_se_basic to test a simple clone across InnoDB, MyRocks, & Blackhole engines. Fix moving RocksDB WALs on startup across file system boundaries Reviewed By: sunshine-Chun Differential Revision: D43160162 Pulled By: hermanlee -------------------------------------------------------------------------------- Rdb clone fixes (facebook#1271) Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee -------------------------------------------------------------------------------- Clone: fix race condition that may skip some SST files (facebook#1274) Summary: A donor thread in the main copy loop took a file to send, released the donor state mutex, and then opened the file. If an ENOENT was received at this point, it was assumed that this was a stale SST file from an older checkpoint that was rolled since. Because the donor state mutex was released between taking of the file and opening it, the following race was possible: 1) Thread 1 takes the file 2) Thread 2 decides to roll the checkpoint, the old checkpoint is deleted 3) Thread 1 tries to open the file, gets ENOENT 4) Thread 2 creates the new checkpoint, the file re-appears, but it's too late. Rolling the checkpoint in a donor state mutex critical section is a possible fix, but such section would do a lot of I/O, serializing the parallel threads. Instead, fix by taking the file and opening it in the same critical section. Pull Request resolved: facebook#1274 Reviewed By: sunshine-Chun Differential Revision: D43629546 Pulled By: hermanlee -------------------------------------------------------------------------------- MyRocks clone: add missing donor mutex unlocks on error exits (facebook#1317) Summary: Pull Request resolved: facebook#1317 Differential Revision: D46564079
Summary: Pull Request resolved: facebook#1250 MyRocks Clone (1/n): make InnoDB clone MTR tests SE-agnostic The MyRocks clone MTR tests will reuse the InnoDB tests by including them. Adjust the tests so that they work this way, i.e. make them storage engine-agnostic by checking the default_storage_engine system variable and guarding InnoDB-specific tests this way. This includes the tests that artificially desync the InnoDB redo log position. MyRocks Clone (2/n): fix bug 109919 (Clone plugin does not support arbitrary length O_DIRECT files) The existing clone plugin I/O routines handle O_DIRECT files with the assumption that their sizes are always a multiple of a block size. This is too restrictive because in general O_DIRECT files can be of any size and MyRocks files actually are. Add a flag to Ha_clone_file to support arbitrary file lengths under O_DIRECT: o_direct_uneven_file_size. Pass it to clone_client.h/cc Client::get_aligned_buffer to allocate the buffer that bumps the length to the next CLONE_OS_ALIGN multiple if it's not already one. Introduce a pad_for_o_direct helper that performs this calculation for the passed length as needed. For writes, if O_DIRECT is enabled and the buffer length is not a multiple of CLONE_OS_ALIGN, pad it with zeros to the next multiple. Then check the actual bytes written, and if any padding bytes were actually written, call my_chsize to truncate the file to the needed size. MyRocks Clone (3/n): fix bug 109920 (SEs cannot add to clone data size estimates) In the current implementation, once the first SE sets the data size estimate for a particular stage, there is no way for other SEs to add to it. Address this by adding Ha_clone_cbk::add_to_data_size_estimate method. MyRocks Clone (4/n): fix bug 109921 (Cannot clone with synchronization across SEs) The existing clone implementation synchronizes InnoDB and binlog directly. There is no way to make other storage engines join this synchronization, thus replace it with synchronizing by a performance_schema.log_status query. - Add table_log_status::get_row method to allow doing the log_status query from inside the server. - Handlerton: introduce two new methods: clone_precopy - currently unused, which will handle any non-consistent-snapshot copying before the main consistent copy, and set_log_stop - which provides the SE the clone stop position information. - Plugin: introduce Ha_clone_common_cbk, which derives from Ha_clone_cbk, and from which derive all the concrete classes: Client_Cbk, Local_Cbk & Server_Cbk. - Ha_clone_common_cbk contains two methods: precopy and synchronize_engines. The former calls precopy for all the participating clone SE handlertons, if implemented. The latter, only to be called from the main clone thread, performs the performance_schema.LOG_STATUS query, splits its result per-SE and calls handlerton set_log_stop for all SEs. - The pfs.log_status query still must be called by InnoDB at a certain point in clone. That means that InnoDB is still a special SE in that it must always come first in the clone locator vectors. This will also handle donor waits on clone restarts, as that is implemented in InnoDB-specific code. To ensure this, add a helper clone_hton.cc:make_innodb_first which shuffles the vectors as needed. Call it in all the clone-starting code paths, add asserts that InnoDB comes first. - Since hton_clone_copy will call into InnoDB clone copy, which at certain point will synchronize engines, calling into the plugin again, and the calling into SEs again, introduce a backup locator vector Ha_clone_cbk::all_locators, and its setters/getters set_all_locators, reset_all_locators, get_all_locators. Use them in the re-entrant hton_clone_copy code. - Introduce a new InnoDB clone state named SST COPY, which calls precopy and has zero files and chunks set to copy in the regular InnoDB clone way, i.e. all it does it the precopy. - The redo log archiving keeps running throughout the precopy stage. - On stage transition from the precopy to the redo log copy. call synchronize. - Implement innodb_clone_set_log_stop by parsing out the desired clone stop LSN from the passed JSON from the log_status query. - Make InnoDB redo log archiving stop at that LSN instead of on reaching the end of redo log. - Support stopping redo log archiving in a middle of a log block. As the old implementation may encounter this case only at the end of the redo log, it took that block from the log buffer in memory. This no longer works with the new implementation, where the needed block might reside on disk only. Address this by forcing the whole block containing the target LSN to disk, reading it from there, and patching its memory copy to stop at the required LSN before giving it for the clone to copy. - To implement redo log block reading on a running server, adapt crash recovery function recv_read_log_seg to work both for crash recovery and on a running server. This is similar to what Percona Server does for its changed page tracking feature. - Make sure not to archive not-yet-written log: pfs.log_status query gets the log system LSN, which is the value of highest log reservation. Thus in an archiving iteration, archive up to the lower values of log archiving LSN, and write_lsn. Set the log archiving LSN to LSN_MAX at clone initialization, so that clone archiving may proceed until the stop LSN is set. - The above implementation results in cloned LSN != log end LSN for the cloned instance, thus adjust crash recovery to work with that. The existing crash recovery throws away any crash-recovered metadata and reads its snapshot from the data files, whereas the snapshot was not written in any consistency with the clone LSN. Fix by no longer throwing away the crash-recovered InnoDB dynamic metadata and by not re-reading the the last metadata snapshot. This forces also not to re-serialize the metadata in the crash recovery as that writes redo log which is to be avoided during clone crash recovery. - Additionally, at the start of the redo log copy the old implementation requested all InnoDB updates to be flushed to disk immediately, to avoid clone recovery generating new redo log. This is also disabled as it becomes redundant with the fix. This issue is similar to Percona XtraBackup https://jira.percona.com/browse/PXB-1634 - Disable clone.local_xa test because the new synchronization no longer blocks on XA transactions. - Re-record two clone-related tests in the component_keyring_file MTR suite. - Temporarily workaround the MyRocks bug where log_status query hangs if RocksDB file deletions were not disabled first. Note that this change breaks some of the clone-related group replication MTR tests. As they are not tested in fb-mysql and have been already broken, this patch makes no effort to fix them. MyRocks Clone (5/n): fix bug 109922 (SEs do not coordinate clone rollbacks on instance startup) If InnoDB clone decides to rollback during its startup, other storage engines are not aware of this and proceed as usual, resulting in inconsistent instance between storage engines. Ideally this should be fixed through a 2PC-like design where all the SEs vote to rollback the clone or to continue during the startup, but for MyRocks clone a simpler solution suffices: make InnoDB write a special marker file if it decides to rollback on clone startup. MyRocks Clone (6/n): on remote clone error, always send COM_ACK to InnoDB too In the case of multithreaded clone, the waits for the donor threads for the current clone stage to complete are implemented fully in InnoDB, even though that, and error handling, should be plugin-level features. With two storage engines participating in the clone this allows the situation where the other storage engine client reports an error with COM_ACK to one donor thread, while some other threads are in InnoDB clone state change waits. Those waits can abort if InnoDB clone gets an error from the client, but they know nothing about other SE errors, and will be aborted only by timeouts. As a minimal fix that does not involve pulling the whole error handling and state change waiting logic to the plugin layer, make the clone client always send an error-containing COM_ACK to InnoDB too, whenever it sends one to another storage engine. MyRocks Clone (7/n): MyRocks clone implementation - Cleanup any stale RocksDB checkpoint directories on server shutdown too. - For MyRocks clone network protocol, define four packet types (clone/common.h): - LOCATOR_V1: MyRocks-specific clone locator, consisting of a monotonically increasing clone session ID for the donor with an optional applier state (an array of file IDs and applied offsets) for restarts, or a zero in which case it is a version check empty locator used for handshakes. - FILE_NAME_MAP_V1: establishes a relation between a file ID, its name and target size. - FILE_CHUNK_V1: cloned file chunk data - ADD_ESTIMATE_V1: how many bytes should be added to the estimated clone data size. - All packet types share the common package header, which stores the version, the payload length without the header, and the packet type. - (clone/common.h): classes locator, file_metadata, chunk_metadata, & estimate_metadata correspond to the four packet types. They are subclassed on the donor side to add serialization routines and other donor-specific code and on the client side to add deserialization routines and other client-specific code. Re. serialization/deserialization, an exception is the locator, whose routines are implemented in the common code, because it may be passed both - ways. - (clone/common.h): Introduce a session class that handles the task ID set and errors, which is common for both donor and client. For errors, allow saving one error, with an optional associated path and errno, that can be set by any thread and later reported to the clone plugin. - (clone/common.cc) Setup MyRocks clone capabilities as HA_CLONE_HYBRID (not - applicable but matches InnoDB), HA_CLONE_MULTI_TASK (can clone with multiple - threads), & HA_CLONE_RESTART (can restart after a network I/O error). - (clone/common.cc) Enable O_DIRECT for clone file I/O if, on client if rocksdb_use_direct_io_for_flush_and_compaction is enabled, and on donor if rocksdb_use_direct_reads is enabled. - On the donor side (clone/donor.cc), encapsulate RocksDB checkpoints in a rdb_checkpoint class. Each checkpoint gets its own ID, which is also a donor session ID. The ID is constant for rolling checkpoints. - Subclass the common file_metadata to donor_file_metadata, chunk_metadata to donor_chunk_metadata, estimate_metadata to donor_estimate. - Implement donor going through a sequence of states: NONE, ROLLING, FINAL, FINAL_WITH_LOGS. To support multiple clone donor sessions, implement a manager class that looks up the donor instance by locator. - Implement MyRocks clone donor interface: - rocksdb_clone_start: handle the possible start modes: - HA_CLONE_MODE_START: create and register a new donor instance, register its master task ID. - HA_CLONE_MODE_ADD_TASK: lookup the existing donor instance, register a new task ID. - HA_CLONE_MODE_RESTART: lookup the existing donor instance, wait for the old master session to drop it by master task ID reference count, then process the locator applier state to reset the donor state. - rocksdb_clone_precopy: lookup the donor instance, create a RocksDB snapshot if state == NONE, otherwise send files by chunks and roll the checkpoint as needed. - rocksdb_clone_set_log_stop: add the WAL files to the clone file to copy set. - rocksdb_clone_copy: copy the clone files in any donor state. - rocksdb_clone_ack: acknowledge errors sent by client. - rocksdb_clone_end: for master task ID, stop accepting new worker join requests, wait until existing workers quit, and, in the case of restartable error, wait for any restarts to happen. Then decrement ref count for the master task ID (which will handle both the cases of restarted and not restarted) clone. For all other task IDs, simply unregister them. - On the client side (clone/client.cc), have file_in_progress class to track each individual file progress during application, and completed_file for fully-applied files. - Subclass the common file_metadata to client_file_metadata, implement its reading from the net buffer. Likewise for client_chunk_metadata subclassing chunk_metadata, and client_estimate for estimate_metadata. - TODO client class. - Implement MyRocks clone client interface: - rocksdb_clone_apply_begin: handle the possible start modes: - HA_CLONE_MODE_START: initialize the client instance for a new clone - HA_CLONE_MODE_ADD_TASK: register a new task ID for an ongoing clone - HA_CLONE_MODE_RESTART: restart the interrupted clone session - HA_CLONE_MODE_VERSION: prepare new version locator - rocksdb_clone_apply: if passed an error code, save it. Otherwise verify the passed buffer and dispatch ADD_ESTIMATE_V1, FILE_NAME_MAP_V1, & FILE_CHUNK_V1 packets accordingly. - rocksdb_clone_apply_end: for master task, wait for all other tasks to complete and finish the clone. For both master and non-master tasks unregister their IDs. - Add rocksdb_sys_vars tests for the new clone-related system variables. They are included in the current patch and not the MTR one in order not to break the existing rocksdb_sys_vars.all_vars test. - On instance startup, do clone fixup (clone/common.cc). It consists of: - Check whether InnoDB clone rollback marker is present. If yes, force rollback of MyRocks clone. - If the cloned temporary datadir contains clone-in-progress marker, assume half-completed clone and abort. - Move the cloned datadirs and WAL dirs in-place. For the datadir, do a directory-level rename with a save and later delete of the old datadir. For the WAL dir do not try to do parent directory-level renames because the mysqld process may not have permissions to delete that directory. Instead, rename all the old files individually, move new files in, delete the old files. - For all the rename operations, handle them returning EXDEV error (a rename would cross a filesystem boundary) by falling back to copy and delete. As we expect that it is the cloned RocksDB log files that are most affected by this, the total copy I/O wouldn't be large, and so the default MySQL copy routine my_copy is used, without extra copy throughput optimizations. Add a fault injection point for MTR to test this code paths. - Remove any RocksDB clone checkpoints found. - On instance startup, always move WALs to the target directory if they are found in the rocksdb_datadir yet the rocksdb_wal_dir is different. This handles the case of setting rocksdb_wal_dir while instance is shutdown, both for clone fixup and regular operation. - Add PFS instrumentation for all the instrumentable entities in this patch (mutexes, rwlocks, file I/O). MyRocks Clone (8/n): MyRocks clone MTR tests - Add a new MTR test suite rocksdb_clone - Support new variable MTR clone/include/clone_command.inc: clone_single_threaded, setting which forces single-threaded clone operation. - Reuse a big part of InnoDB clone tests by including their test files, with the exception of tests involving DDL, as concurrent DDL is not supported by MyRocks clone at the moment, and debug tests that adjust cloned instance LSN, because they desynchronize storage engines. The reused tests are: local_basic, local_boundary, local_dml, local_dml_auto_tune, local_dml_no_redo, local_insert, local_stage, monitor_progress, remote_basic, remote_basic_replace, remote_boundary, remote_dml, remote_dml_auto_tune, remote_dml_gtid, remote_dml_no_binlog, remote_dml_no_ssl, remote_dml_recovery, remote_dml_replace, remote_dml_ssl, remote_dml_ssl_certificate, remote_dml_upgrade, remote_insert, remote_monitor_progress, remote_restart, remote_skip_validation, remote_stage, remote_stage_restart. - As a part of the above, add MyRocks support to local_basic and monitor_progress MTR tests. - Besides the reused InnoDB clone tests, add some new MyRocks-specific tests: - local_minimal_1_thread, which is the simplest possible local clone test. - local_minimal_multithreaded, the simplest possible but multithreaded local clone test. - local_minimal_direct_io, which is the simplest possible local clone test in all combinations of rocksdb_use_direct_reads and rocksdb_use_direct_io_for_flush_and_compaction settings, which are the O_DIRECT-enabling options. - local_dml_o_direct_donor, local_dml_o_direct_client, local_dml_o_direct_both that reuse InnoDB local_dml test, but combined with either or both of the O_DIRECT-enabling options. - local_minimal_wal_dir, which clones with various rocksdb_wal_dir settings on donor and client. - local_minimal_wal_dir_copy_delete, which uses a debug injection to test the move-across-filesystem-boundary code path in clone startup. - remote_minimal_direct_io_{1, 2, 3, 4} tests for all the combinations of O_DIRECT-enabling options for donor and client instances. - remote_fail_apply, which uses a debug injection to fail the clone client during remote clone. - remote_inplace_restart, which performs a remote in-place clone with a simulated transient error on the client. - remote_kill_donor, which kill the donor instance at various points of remote cloning. - rolling_checkpoint for testing various combinations of clone rolling checkpoint tuning options. - three_se_basic to test a simple clone across InnoDB, MyRocks, & Blackhole engines. Fix moving RocksDB WALs on startup across file system boundaries Reviewed By: sunshine-Chun Differential Revision: D43160162 Pulled By: hermanlee -------------------------------------------------------------------------------- Rdb clone fixes (facebook#1271) Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee -------------------------------------------------------------------------------- Clone: fix race condition that may skip some SST files (facebook#1274) Summary: A donor thread in the main copy loop took a file to send, released the donor state mutex, and then opened the file. If an ENOENT was received at this point, it was assumed that this was a stale SST file from an older checkpoint that was rolled since. Because the donor state mutex was released between taking of the file and opening it, the following race was possible: 1) Thread 1 takes the file 2) Thread 2 decides to roll the checkpoint, the old checkpoint is deleted 3) Thread 1 tries to open the file, gets ENOENT 4) Thread 2 creates the new checkpoint, the file re-appears, but it's too late. Rolling the checkpoint in a donor state mutex critical section is a possible fix, but such section would do a lot of I/O, serializing the parallel threads. Instead, fix by taking the file and opening it in the same critical section. Pull Request resolved: facebook#1274 Reviewed By: sunshine-Chun Differential Revision: D43629546 Pulled By: hermanlee -------------------------------------------------------------------------------- MyRocks clone: add missing donor mutex unlocks on error exits (facebook#1317) Summary: Pull Request resolved: facebook#1317 Differential Revision: D46564079
Summary: Pull Request resolved: facebook#1250 MyRocks Clone (1/n): make InnoDB clone MTR tests SE-agnostic The MyRocks clone MTR tests will reuse the InnoDB tests by including them. Adjust the tests so that they work this way, i.e. make them storage engine-agnostic by checking the default_storage_engine system variable and guarding InnoDB-specific tests this way. This includes the tests that artificially desync the InnoDB redo log position. MyRocks Clone (2/n): fix bug 109919 (Clone plugin does not support arbitrary length O_DIRECT files) The existing clone plugin I/O routines handle O_DIRECT files with the assumption that their sizes are always a multiple of a block size. This is too restrictive because in general O_DIRECT files can be of any size and MyRocks files actually are. Add a flag to Ha_clone_file to support arbitrary file lengths under O_DIRECT: o_direct_uneven_file_size. Pass it to clone_client.h/cc Client::get_aligned_buffer to allocate the buffer that bumps the length to the next CLONE_OS_ALIGN multiple if it's not already one. Introduce a pad_for_o_direct helper that performs this calculation for the passed length as needed. For writes, if O_DIRECT is enabled and the buffer length is not a multiple of CLONE_OS_ALIGN, pad it with zeros to the next multiple. Then check the actual bytes written, and if any padding bytes were actually written, call my_chsize to truncate the file to the needed size. MyRocks Clone (3/n): fix bug 109920 (SEs cannot add to clone data size estimates) In the current implementation, once the first SE sets the data size estimate for a particular stage, there is no way for other SEs to add to it. Address this by adding Ha_clone_cbk::add_to_data_size_estimate method. MyRocks Clone (4/n): fix bug 109921 (Cannot clone with synchronization across SEs) The existing clone implementation synchronizes InnoDB and binlog directly. There is no way to make other storage engines join this synchronization, thus replace it with synchronizing by a performance_schema.log_status query. - Add table_log_status::get_row method to allow doing the log_status query from inside the server. - Handlerton: introduce two new methods: clone_precopy - currently unused, which will handle any non-consistent-snapshot copying before the main consistent copy, and set_log_stop - which provides the SE the clone stop position information. - Plugin: introduce Ha_clone_common_cbk, which derives from Ha_clone_cbk, and from which derive all the concrete classes: Client_Cbk, Local_Cbk & Server_Cbk. - Ha_clone_common_cbk contains two methods: precopy and synchronize_engines. The former calls precopy for all the participating clone SE handlertons, if implemented. The latter, only to be called from the main clone thread, performs the performance_schema.LOG_STATUS query, splits its result per-SE and calls handlerton set_log_stop for all SEs. - The pfs.log_status query still must be called by InnoDB at a certain point in clone. That means that InnoDB is still a special SE in that it must always come first in the clone locator vectors. This will also handle donor waits on clone restarts, as that is implemented in InnoDB-specific code. To ensure this, add a helper clone_hton.cc:make_innodb_first which shuffles the vectors as needed. Call it in all the clone-starting code paths, add asserts that InnoDB comes first. - Since hton_clone_copy will call into InnoDB clone copy, which at certain point will synchronize engines, calling into the plugin again, and the calling into SEs again, introduce a backup locator vector Ha_clone_cbk::all_locators, and its setters/getters set_all_locators, reset_all_locators, get_all_locators. Use them in the re-entrant hton_clone_copy code. - Introduce a new InnoDB clone state named SST COPY, which calls precopy and has zero files and chunks set to copy in the regular InnoDB clone way, i.e. all it does it the precopy. - The redo log archiving keeps running throughout the precopy stage. - On stage transition from the precopy to the redo log copy. call synchronize. - Implement innodb_clone_set_log_stop by parsing out the desired clone stop LSN from the passed JSON from the log_status query. - Make InnoDB redo log archiving stop at that LSN instead of on reaching the end of redo log. - Support stopping redo log archiving in a middle of a log block. As the old implementation may encounter this case only at the end of the redo log, it took that block from the log buffer in memory. This no longer works with the new implementation, where the needed block might reside on disk only. Address this by forcing the whole block containing the target LSN to disk, reading it from there, and patching its memory copy to stop at the required LSN before giving it for the clone to copy. - To implement redo log block reading on a running server, adapt crash recovery function recv_read_log_seg to work both for crash recovery and on a running server. This is similar to what Percona Server does for its changed page tracking feature. - Make sure not to archive not-yet-written log: pfs.log_status query gets the log system LSN, which is the value of highest log reservation. Thus in an archiving iteration, archive up to the lower values of log archiving LSN, and write_lsn. Set the log archiving LSN to LSN_MAX at clone initialization, so that clone archiving may proceed until the stop LSN is set. - The above implementation results in cloned LSN != log end LSN for the cloned instance, thus adjust crash recovery to work with that. The existing crash recovery throws away any crash-recovered metadata and reads its snapshot from the data files, whereas the snapshot was not written in any consistency with the clone LSN. Fix by no longer throwing away the crash-recovered InnoDB dynamic metadata and by not re-reading the the last metadata snapshot. This forces also not to re-serialize the metadata in the crash recovery as that writes redo log which is to be avoided during clone crash recovery. - Additionally, at the start of the redo log copy the old implementation requested all InnoDB updates to be flushed to disk immediately, to avoid clone recovery generating new redo log. This is also disabled as it becomes redundant with the fix. This issue is similar to Percona XtraBackup https://jira.percona.com/browse/PXB-1634 - Disable clone.local_xa test because the new synchronization no longer blocks on XA transactions. - Re-record two clone-related tests in the component_keyring_file MTR suite. - Temporarily workaround the MyRocks bug where log_status query hangs if RocksDB file deletions were not disabled first. Note that this change breaks some of the clone-related group replication MTR tests. As they are not tested in fb-mysql and have been already broken, this patch makes no effort to fix them. MyRocks Clone (5/n): fix bug 109922 (SEs do not coordinate clone rollbacks on instance startup) If InnoDB clone decides to rollback during its startup, other storage engines are not aware of this and proceed as usual, resulting in inconsistent instance between storage engines. Ideally this should be fixed through a 2PC-like design where all the SEs vote to rollback the clone or to continue during the startup, but for MyRocks clone a simpler solution suffices: make InnoDB write a special marker file if it decides to rollback on clone startup. MyRocks Clone (6/n): on remote clone error, always send COM_ACK to InnoDB too In the case of multithreaded clone, the waits for the donor threads for the current clone stage to complete are implemented fully in InnoDB, even though that, and error handling, should be plugin-level features. With two storage engines participating in the clone this allows the situation where the other storage engine client reports an error with COM_ACK to one donor thread, while some other threads are in InnoDB clone state change waits. Those waits can abort if InnoDB clone gets an error from the client, but they know nothing about other SE errors, and will be aborted only by timeouts. As a minimal fix that does not involve pulling the whole error handling and state change waiting logic to the plugin layer, make the clone client always send an error-containing COM_ACK to InnoDB too, whenever it sends one to another storage engine. MyRocks Clone (7/n): MyRocks clone implementation - Cleanup any stale RocksDB checkpoint directories on server shutdown too. - For MyRocks clone network protocol, define four packet types (clone/common.h): - LOCATOR_V1: MyRocks-specific clone locator, consisting of a monotonically increasing clone session ID for the donor with an optional applier state (an array of file IDs and applied offsets) for restarts, or a zero in which case it is a version check empty locator used for handshakes. - FILE_NAME_MAP_V1: establishes a relation between a file ID, its name and target size. - FILE_CHUNK_V1: cloned file chunk data - ADD_ESTIMATE_V1: how many bytes should be added to the estimated clone data size. - All packet types share the common package header, which stores the version, the payload length without the header, and the packet type. - (clone/common.h): classes locator, file_metadata, chunk_metadata, & estimate_metadata correspond to the four packet types. They are subclassed on the donor side to add serialization routines and other donor-specific code and on the client side to add deserialization routines and other client-specific code. Re. serialization/deserialization, an exception is the locator, whose routines are implemented in the common code, because it may be passed both - ways. - (clone/common.h): Introduce a session class that handles the task ID set and errors, which is common for both donor and client. For errors, allow saving one error, with an optional associated path and errno, that can be set by any thread and later reported to the clone plugin. - (clone/common.cc) Setup MyRocks clone capabilities as HA_CLONE_HYBRID (not - applicable but matches InnoDB), HA_CLONE_MULTI_TASK (can clone with multiple - threads), & HA_CLONE_RESTART (can restart after a network I/O error). - (clone/common.cc) Enable O_DIRECT for clone file I/O if, on client if rocksdb_use_direct_io_for_flush_and_compaction is enabled, and on donor if rocksdb_use_direct_reads is enabled. - On the donor side (clone/donor.cc), encapsulate RocksDB checkpoints in a rdb_checkpoint class. Each checkpoint gets its own ID, which is also a donor session ID. The ID is constant for rolling checkpoints. - Subclass the common file_metadata to donor_file_metadata, chunk_metadata to donor_chunk_metadata, estimate_metadata to donor_estimate. - Implement donor going through a sequence of states: NONE, ROLLING, FINAL, FINAL_WITH_LOGS. To support multiple clone donor sessions, implement a manager class that looks up the donor instance by locator. - Implement MyRocks clone donor interface: - rocksdb_clone_start: handle the possible start modes: - HA_CLONE_MODE_START: create and register a new donor instance, register its master task ID. - HA_CLONE_MODE_ADD_TASK: lookup the existing donor instance, register a new task ID. - HA_CLONE_MODE_RESTART: lookup the existing donor instance, wait for the old master session to drop it by master task ID reference count, then process the locator applier state to reset the donor state. - rocksdb_clone_precopy: lookup the donor instance, create a RocksDB snapshot if state == NONE, otherwise send files by chunks and roll the checkpoint as needed. - rocksdb_clone_set_log_stop: add the WAL files to the clone file to copy set. - rocksdb_clone_copy: copy the clone files in any donor state. - rocksdb_clone_ack: acknowledge errors sent by client. - rocksdb_clone_end: for master task ID, stop accepting new worker join requests, wait until existing workers quit, and, in the case of restartable error, wait for any restarts to happen. Then decrement ref count for the master task ID (which will handle both the cases of restarted and not restarted) clone. For all other task IDs, simply unregister them. - On the client side (clone/client.cc), have file_in_progress class to track each individual file progress during application, and completed_file for fully-applied files. - Subclass the common file_metadata to client_file_metadata, implement its reading from the net buffer. Likewise for client_chunk_metadata subclassing chunk_metadata, and client_estimate for estimate_metadata. - TODO client class. - Implement MyRocks clone client interface: - rocksdb_clone_apply_begin: handle the possible start modes: - HA_CLONE_MODE_START: initialize the client instance for a new clone - HA_CLONE_MODE_ADD_TASK: register a new task ID for an ongoing clone - HA_CLONE_MODE_RESTART: restart the interrupted clone session - HA_CLONE_MODE_VERSION: prepare new version locator - rocksdb_clone_apply: if passed an error code, save it. Otherwise verify the passed buffer and dispatch ADD_ESTIMATE_V1, FILE_NAME_MAP_V1, & FILE_CHUNK_V1 packets accordingly. - rocksdb_clone_apply_end: for master task, wait for all other tasks to complete and finish the clone. For both master and non-master tasks unregister their IDs. - Add rocksdb_sys_vars tests for the new clone-related system variables. They are included in the current patch and not the MTR one in order not to break the existing rocksdb_sys_vars.all_vars test. - On instance startup, do clone fixup (clone/common.cc). It consists of: - Check whether InnoDB clone rollback marker is present. If yes, force rollback of MyRocks clone. - If the cloned temporary datadir contains clone-in-progress marker, assume half-completed clone and abort. - Move the cloned datadirs and WAL dirs in-place. For the datadir, do a directory-level rename with a save and later delete of the old datadir. For the WAL dir do not try to do parent directory-level renames because the mysqld process may not have permissions to delete that directory. Instead, rename all the old files individually, move new files in, delete the old files. - For all the rename operations, handle them returning EXDEV error (a rename would cross a filesystem boundary) by falling back to copy and delete. As we expect that it is the cloned RocksDB log files that are most affected by this, the total copy I/O wouldn't be large, and so the default MySQL copy routine my_copy is used, without extra copy throughput optimizations. Add a fault injection point for MTR to test this code paths. - Remove any RocksDB clone checkpoints found. - On instance startup, always move WALs to the target directory if they are found in the rocksdb_datadir yet the rocksdb_wal_dir is different. This handles the case of setting rocksdb_wal_dir while instance is shutdown, both for clone fixup and regular operation. - Add PFS instrumentation for all the instrumentable entities in this patch (mutexes, rwlocks, file I/O). MyRocks Clone (8/n): MyRocks clone MTR tests - Add a new MTR test suite rocksdb_clone - Support new variable MTR clone/include/clone_command.inc: clone_single_threaded, setting which forces single-threaded clone operation. - Reuse a big part of InnoDB clone tests by including their test files, with the exception of tests involving DDL, as concurrent DDL is not supported by MyRocks clone at the moment, and debug tests that adjust cloned instance LSN, because they desynchronize storage engines. The reused tests are: local_basic, local_boundary, local_dml, local_dml_auto_tune, local_dml_no_redo, local_insert, local_stage, monitor_progress, remote_basic, remote_basic_replace, remote_boundary, remote_dml, remote_dml_auto_tune, remote_dml_gtid, remote_dml_no_binlog, remote_dml_no_ssl, remote_dml_recovery, remote_dml_replace, remote_dml_ssl, remote_dml_ssl_certificate, remote_dml_upgrade, remote_insert, remote_monitor_progress, remote_restart, remote_skip_validation, remote_stage, remote_stage_restart. - As a part of the above, add MyRocks support to local_basic and monitor_progress MTR tests. - Besides the reused InnoDB clone tests, add some new MyRocks-specific tests: - local_minimal_1_thread, which is the simplest possible local clone test. - local_minimal_multithreaded, the simplest possible but multithreaded local clone test. - local_minimal_direct_io, which is the simplest possible local clone test in all combinations of rocksdb_use_direct_reads and rocksdb_use_direct_io_for_flush_and_compaction settings, which are the O_DIRECT-enabling options. - local_dml_o_direct_donor, local_dml_o_direct_client, local_dml_o_direct_both that reuse InnoDB local_dml test, but combined with either or both of the O_DIRECT-enabling options. - local_minimal_wal_dir, which clones with various rocksdb_wal_dir settings on donor and client. - local_minimal_wal_dir_copy_delete, which uses a debug injection to test the move-across-filesystem-boundary code path in clone startup. - remote_minimal_direct_io_{1, 2, 3, 4} tests for all the combinations of O_DIRECT-enabling options for donor and client instances. - remote_fail_apply, which uses a debug injection to fail the clone client during remote clone. - remote_inplace_restart, which performs a remote in-place clone with a simulated transient error on the client. - remote_kill_donor, which kill the donor instance at various points of remote cloning. - rolling_checkpoint for testing various combinations of clone rolling checkpoint tuning options. - three_se_basic to test a simple clone across InnoDB, MyRocks, & Blackhole engines. Fix moving RocksDB WALs on startup across file system boundaries Reviewed By: sunshine-Chun Differential Revision: D43160162 Pulled By: hermanlee -------------------------------------------------------------------------------- Rdb clone fixes (facebook#1271) Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee -------------------------------------------------------------------------------- Clone: fix race condition that may skip some SST files (facebook#1274) Summary: A donor thread in the main copy loop took a file to send, released the donor state mutex, and then opened the file. If an ENOENT was received at this point, it was assumed that this was a stale SST file from an older checkpoint that was rolled since. Because the donor state mutex was released between taking of the file and opening it, the following race was possible: 1) Thread 1 takes the file 2) Thread 2 decides to roll the checkpoint, the old checkpoint is deleted 3) Thread 1 tries to open the file, gets ENOENT 4) Thread 2 creates the new checkpoint, the file re-appears, but it's too late. Rolling the checkpoint in a donor state mutex critical section is a possible fix, but such section would do a lot of I/O, serializing the parallel threads. Instead, fix by taking the file and opening it in the same critical section. Pull Request resolved: facebook#1274 Reviewed By: sunshine-Chun Differential Revision: D43629546 Pulled By: hermanlee -------------------------------------------------------------------------------- MyRocks clone: add missing donor mutex unlocks on error exits (facebook#1317) Summary: Pull Request resolved: facebook#1317 Differential Revision: D46564079
Summary: Pull Request resolved: facebook#1250 MyRocks Clone (1/n): make InnoDB clone MTR tests SE-agnostic The MyRocks clone MTR tests will reuse the InnoDB tests by including them. Adjust the tests so that they work this way, i.e. make them storage engine-agnostic by checking the default_storage_engine system variable and guarding InnoDB-specific tests this way. This includes the tests that artificially desync the InnoDB redo log position. MyRocks Clone (2/n): fix bug 109919 (Clone plugin does not support arbitrary length O_DIRECT files) The existing clone plugin I/O routines handle O_DIRECT files with the assumption that their sizes are always a multiple of a block size. This is too restrictive because in general O_DIRECT files can be of any size and MyRocks files actually are. Add a flag to Ha_clone_file to support arbitrary file lengths under O_DIRECT: o_direct_uneven_file_size. Pass it to clone_client.h/cc Client::get_aligned_buffer to allocate the buffer that bumps the length to the next CLONE_OS_ALIGN multiple if it's not already one. Introduce a pad_for_o_direct helper that performs this calculation for the passed length as needed. For writes, if O_DIRECT is enabled and the buffer length is not a multiple of CLONE_OS_ALIGN, pad it with zeros to the next multiple. Then check the actual bytes written, and if any padding bytes were actually written, call my_chsize to truncate the file to the needed size. MyRocks Clone (3/n): fix bug 109920 (SEs cannot add to clone data size estimates) In the current implementation, once the first SE sets the data size estimate for a particular stage, there is no way for other SEs to add to it. Address this by adding Ha_clone_cbk::add_to_data_size_estimate method. MyRocks Clone (4/n): fix bug 109921 (Cannot clone with synchronization across SEs) The existing clone implementation synchronizes InnoDB and binlog directly. There is no way to make other storage engines join this synchronization, thus replace it with synchronizing by a performance_schema.log_status query. - Add table_log_status::get_row method to allow doing the log_status query from inside the server. - Handlerton: introduce two new methods: clone_precopy - currently unused, which will handle any non-consistent-snapshot copying before the main consistent copy, and set_log_stop - which provides the SE the clone stop position information. - Plugin: introduce Ha_clone_common_cbk, which derives from Ha_clone_cbk, and from which derive all the concrete classes: Client_Cbk, Local_Cbk & Server_Cbk. - Ha_clone_common_cbk contains two methods: precopy and synchronize_engines. The former calls precopy for all the participating clone SE handlertons, if implemented. The latter, only to be called from the main clone thread, performs the performance_schema.LOG_STATUS query, splits its result per-SE and calls handlerton set_log_stop for all SEs. - The pfs.log_status query still must be called by InnoDB at a certain point in clone. That means that InnoDB is still a special SE in that it must always come first in the clone locator vectors. This will also handle donor waits on clone restarts, as that is implemented in InnoDB-specific code. To ensure this, add a helper clone_hton.cc:make_innodb_first which shuffles the vectors as needed. Call it in all the clone-starting code paths, add asserts that InnoDB comes first. - Since hton_clone_copy will call into InnoDB clone copy, which at certain point will synchronize engines, calling into the plugin again, and the calling into SEs again, introduce a backup locator vector Ha_clone_cbk::all_locators, and its setters/getters set_all_locators, reset_all_locators, get_all_locators. Use them in the re-entrant hton_clone_copy code. - Introduce a new InnoDB clone state named SST COPY, which calls precopy and has zero files and chunks set to copy in the regular InnoDB clone way, i.e. all it does it the precopy. - The redo log archiving keeps running throughout the precopy stage. - On stage transition from the precopy to the redo log copy. call synchronize. - Implement innodb_clone_set_log_stop by parsing out the desired clone stop LSN from the passed JSON from the log_status query. - Make InnoDB redo log archiving stop at that LSN instead of on reaching the end of redo log. - Support stopping redo log archiving in a middle of a log block. As the old implementation may encounter this case only at the end of the redo log, it took that block from the log buffer in memory. This no longer works with the new implementation, where the needed block might reside on disk only. Address this by forcing the whole block containing the target LSN to disk, reading it from there, and patching its memory copy to stop at the required LSN before giving it for the clone to copy. - To implement redo log block reading on a running server, adapt crash recovery function recv_read_log_seg to work both for crash recovery and on a running server. This is similar to what Percona Server does for its changed page tracking feature. - Make sure not to archive not-yet-written log: pfs.log_status query gets the log system LSN, which is the value of highest log reservation. Thus in an archiving iteration, archive up to the lower values of log archiving LSN, and write_lsn. Set the log archiving LSN to LSN_MAX at clone initialization, so that clone archiving may proceed until the stop LSN is set. - The above implementation results in cloned LSN != log end LSN for the cloned instance, thus adjust crash recovery to work with that. The existing crash recovery throws away any crash-recovered metadata and reads its snapshot from the data files, whereas the snapshot was not written in any consistency with the clone LSN. Fix by no longer throwing away the crash-recovered InnoDB dynamic metadata and by not re-reading the the last metadata snapshot. This forces also not to re-serialize the metadata in the crash recovery as that writes redo log which is to be avoided during clone crash recovery. - Additionally, at the start of the redo log copy the old implementation requested all InnoDB updates to be flushed to disk immediately, to avoid clone recovery generating new redo log. This is also disabled as it becomes redundant with the fix. This issue is similar to Percona XtraBackup https://jira.percona.com/browse/PXB-1634 - Disable clone.local_xa test because the new synchronization no longer blocks on XA transactions. - Re-record two clone-related tests in the component_keyring_file MTR suite. - Temporarily workaround the MyRocks bug where log_status query hangs if RocksDB file deletions were not disabled first. Note that this change breaks some of the clone-related group replication MTR tests. As they are not tested in fb-mysql and have been already broken, this patch makes no effort to fix them. MyRocks Clone (5/n): fix bug 109922 (SEs do not coordinate clone rollbacks on instance startup) If InnoDB clone decides to rollback during its startup, other storage engines are not aware of this and proceed as usual, resulting in inconsistent instance between storage engines. Ideally this should be fixed through a 2PC-like design where all the SEs vote to rollback the clone or to continue during the startup, but for MyRocks clone a simpler solution suffices: make InnoDB write a special marker file if it decides to rollback on clone startup. MyRocks Clone (6/n): on remote clone error, always send COM_ACK to InnoDB too In the case of multithreaded clone, the waits for the donor threads for the current clone stage to complete are implemented fully in InnoDB, even though that, and error handling, should be plugin-level features. With two storage engines participating in the clone this allows the situation where the other storage engine client reports an error with COM_ACK to one donor thread, while some other threads are in InnoDB clone state change waits. Those waits can abort if InnoDB clone gets an error from the client, but they know nothing about other SE errors, and will be aborted only by timeouts. As a minimal fix that does not involve pulling the whole error handling and state change waiting logic to the plugin layer, make the clone client always send an error-containing COM_ACK to InnoDB too, whenever it sends one to another storage engine. MyRocks Clone (7/n): MyRocks clone implementation - Cleanup any stale RocksDB checkpoint directories on server shutdown too. - For MyRocks clone network protocol, define four packet types (clone/common.h): - LOCATOR_V1: MyRocks-specific clone locator, consisting of a monotonically increasing clone session ID for the donor with an optional applier state (an array of file IDs and applied offsets) for restarts, or a zero in which case it is a version check empty locator used for handshakes. - FILE_NAME_MAP_V1: establishes a relation between a file ID, its name and target size. - FILE_CHUNK_V1: cloned file chunk data - ADD_ESTIMATE_V1: how many bytes should be added to the estimated clone data size. - All packet types share the common package header, which stores the version, the payload length without the header, and the packet type. - (clone/common.h): classes locator, file_metadata, chunk_metadata, & estimate_metadata correspond to the four packet types. They are subclassed on the donor side to add serialization routines and other donor-specific code and on the client side to add deserialization routines and other client-specific code. Re. serialization/deserialization, an exception is the locator, whose routines are implemented in the common code, because it may be passed both - ways. - (clone/common.h): Introduce a session class that handles the task ID set and errors, which is common for both donor and client. For errors, allow saving one error, with an optional associated path and errno, that can be set by any thread and later reported to the clone plugin. - (clone/common.cc) Setup MyRocks clone capabilities as HA_CLONE_HYBRID (not - applicable but matches InnoDB), HA_CLONE_MULTI_TASK (can clone with multiple - threads), & HA_CLONE_RESTART (can restart after a network I/O error). - (clone/common.cc) Enable O_DIRECT for clone file I/O if, on client if rocksdb_use_direct_io_for_flush_and_compaction is enabled, and on donor if rocksdb_use_direct_reads is enabled. - On the donor side (clone/donor.cc), encapsulate RocksDB checkpoints in a rdb_checkpoint class. Each checkpoint gets its own ID, which is also a donor session ID. The ID is constant for rolling checkpoints. - Subclass the common file_metadata to donor_file_metadata, chunk_metadata to donor_chunk_metadata, estimate_metadata to donor_estimate. - Implement donor going through a sequence of states: NONE, ROLLING, FINAL, FINAL_WITH_LOGS. To support multiple clone donor sessions, implement a manager class that looks up the donor instance by locator. - Implement MyRocks clone donor interface: - rocksdb_clone_start: handle the possible start modes: - HA_CLONE_MODE_START: create and register a new donor instance, register its master task ID. - HA_CLONE_MODE_ADD_TASK: lookup the existing donor instance, register a new task ID. - HA_CLONE_MODE_RESTART: lookup the existing donor instance, wait for the old master session to drop it by master task ID reference count, then process the locator applier state to reset the donor state. - rocksdb_clone_precopy: lookup the donor instance, create a RocksDB snapshot if state == NONE, otherwise send files by chunks and roll the checkpoint as needed. - rocksdb_clone_set_log_stop: add the WAL files to the clone file to copy set. - rocksdb_clone_copy: copy the clone files in any donor state. - rocksdb_clone_ack: acknowledge errors sent by client. - rocksdb_clone_end: for master task ID, stop accepting new worker join requests, wait until existing workers quit, and, in the case of restartable error, wait for any restarts to happen. Then decrement ref count for the master task ID (which will handle both the cases of restarted and not restarted) clone. For all other task IDs, simply unregister them. - On the client side (clone/client.cc), have file_in_progress class to track each individual file progress during application, and completed_file for fully-applied files. - Subclass the common file_metadata to client_file_metadata, implement its reading from the net buffer. Likewise for client_chunk_metadata subclassing chunk_metadata, and client_estimate for estimate_metadata. - TODO client class. - Implement MyRocks clone client interface: - rocksdb_clone_apply_begin: handle the possible start modes: - HA_CLONE_MODE_START: initialize the client instance for a new clone - HA_CLONE_MODE_ADD_TASK: register a new task ID for an ongoing clone - HA_CLONE_MODE_RESTART: restart the interrupted clone session - HA_CLONE_MODE_VERSION: prepare new version locator - rocksdb_clone_apply: if passed an error code, save it. Otherwise verify the passed buffer and dispatch ADD_ESTIMATE_V1, FILE_NAME_MAP_V1, & FILE_CHUNK_V1 packets accordingly. - rocksdb_clone_apply_end: for master task, wait for all other tasks to complete and finish the clone. For both master and non-master tasks unregister their IDs. - Add rocksdb_sys_vars tests for the new clone-related system variables. They are included in the current patch and not the MTR one in order not to break the existing rocksdb_sys_vars.all_vars test. - On instance startup, do clone fixup (clone/common.cc). It consists of: - Check whether InnoDB clone rollback marker is present. If yes, force rollback of MyRocks clone. - If the cloned temporary datadir contains clone-in-progress marker, assume half-completed clone and abort. - Move the cloned datadirs and WAL dirs in-place. For the datadir, do a directory-level rename with a save and later delete of the old datadir. For the WAL dir do not try to do parent directory-level renames because the mysqld process may not have permissions to delete that directory. Instead, rename all the old files individually, move new files in, delete the old files. - For all the rename operations, handle them returning EXDEV error (a rename would cross a filesystem boundary) by falling back to copy and delete. As we expect that it is the cloned RocksDB log files that are most affected by this, the total copy I/O wouldn't be large, and so the default MySQL copy routine my_copy is used, without extra copy throughput optimizations. Add a fault injection point for MTR to test this code paths. - Remove any RocksDB clone checkpoints found. - On instance startup, always move WALs to the target directory if they are found in the rocksdb_datadir yet the rocksdb_wal_dir is different. This handles the case of setting rocksdb_wal_dir while instance is shutdown, both for clone fixup and regular operation. - Add PFS instrumentation for all the instrumentable entities in this patch (mutexes, rwlocks, file I/O). MyRocks Clone (8/n): MyRocks clone MTR tests - Add a new MTR test suite rocksdb_clone - Support new variable MTR clone/include/clone_command.inc: clone_single_threaded, setting which forces single-threaded clone operation. - Reuse a big part of InnoDB clone tests by including their test files, with the exception of tests involving DDL, as concurrent DDL is not supported by MyRocks clone at the moment, and debug tests that adjust cloned instance LSN, because they desynchronize storage engines. The reused tests are: local_basic, local_boundary, local_dml, local_dml_auto_tune, local_dml_no_redo, local_insert, local_stage, monitor_progress, remote_basic, remote_basic_replace, remote_boundary, remote_dml, remote_dml_auto_tune, remote_dml_gtid, remote_dml_no_binlog, remote_dml_no_ssl, remote_dml_recovery, remote_dml_replace, remote_dml_ssl, remote_dml_ssl_certificate, remote_dml_upgrade, remote_insert, remote_monitor_progress, remote_restart, remote_skip_validation, remote_stage, remote_stage_restart. - As a part of the above, add MyRocks support to local_basic and monitor_progress MTR tests. - Besides the reused InnoDB clone tests, add some new MyRocks-specific tests: - local_minimal_1_thread, which is the simplest possible local clone test. - local_minimal_multithreaded, the simplest possible but multithreaded local clone test. - local_minimal_direct_io, which is the simplest possible local clone test in all combinations of rocksdb_use_direct_reads and rocksdb_use_direct_io_for_flush_and_compaction settings, which are the O_DIRECT-enabling options. - local_dml_o_direct_donor, local_dml_o_direct_client, local_dml_o_direct_both that reuse InnoDB local_dml test, but combined with either or both of the O_DIRECT-enabling options. - local_minimal_wal_dir, which clones with various rocksdb_wal_dir settings on donor and client. - local_minimal_wal_dir_copy_delete, which uses a debug injection to test the move-across-filesystem-boundary code path in clone startup. - remote_minimal_direct_io_{1, 2, 3, 4} tests for all the combinations of O_DIRECT-enabling options for donor and client instances. - remote_fail_apply, which uses a debug injection to fail the clone client during remote clone. - remote_inplace_restart, which performs a remote in-place clone with a simulated transient error on the client. - remote_kill_donor, which kill the donor instance at various points of remote cloning. - rolling_checkpoint for testing various combinations of clone rolling checkpoint tuning options. - three_se_basic to test a simple clone across InnoDB, MyRocks, & Blackhole engines. Fix moving RocksDB WALs on startup across file system boundaries Reviewed By: sunshine-Chun Differential Revision: D43160162 Pulled By: hermanlee -------------------------------------------------------------------------------- Rdb clone fixes (facebook#1271) Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee -------------------------------------------------------------------------------- Clone: fix race condition that may skip some SST files (facebook#1274) Summary: A donor thread in the main copy loop took a file to send, released the donor state mutex, and then opened the file. If an ENOENT was received at this point, it was assumed that this was a stale SST file from an older checkpoint that was rolled since. Because the donor state mutex was released between taking of the file and opening it, the following race was possible: 1) Thread 1 takes the file 2) Thread 2 decides to roll the checkpoint, the old checkpoint is deleted 3) Thread 1 tries to open the file, gets ENOENT 4) Thread 2 creates the new checkpoint, the file re-appears, but it's too late. Rolling the checkpoint in a donor state mutex critical section is a possible fix, but such section would do a lot of I/O, serializing the parallel threads. Instead, fix by taking the file and opening it in the same critical section. Pull Request resolved: facebook#1274 Reviewed By: sunshine-Chun Differential Revision: D43629546 Pulled By: hermanlee -------------------------------------------------------------------------------- MyRocks clone: add missing donor mutex unlocks on error exits (facebook#1317) Summary: Pull Request resolved: facebook#1317 Differential Revision: D46564079
Summary: Pull Request resolved: facebook#1250 MyRocks Clone (1/n): make InnoDB clone MTR tests SE-agnostic The MyRocks clone MTR tests will reuse the InnoDB tests by including them. Adjust the tests so that they work this way, i.e. make them storage engine-agnostic by checking the default_storage_engine system variable and guarding InnoDB-specific tests this way. This includes the tests that artificially desync the InnoDB redo log position. MyRocks Clone (2/n): fix bug 109919 (Clone plugin does not support arbitrary length O_DIRECT files) The existing clone plugin I/O routines handle O_DIRECT files with the assumption that their sizes are always a multiple of a block size. This is too restrictive because in general O_DIRECT files can be of any size and MyRocks files actually are. Add a flag to Ha_clone_file to support arbitrary file lengths under O_DIRECT: o_direct_uneven_file_size. Pass it to clone_client.h/cc Client::get_aligned_buffer to allocate the buffer that bumps the length to the next CLONE_OS_ALIGN multiple if it's not already one. Introduce a pad_for_o_direct helper that performs this calculation for the passed length as needed. For writes, if O_DIRECT is enabled and the buffer length is not a multiple of CLONE_OS_ALIGN, pad it with zeros to the next multiple. Then check the actual bytes written, and if any padding bytes were actually written, call my_chsize to truncate the file to the needed size. MyRocks Clone (3/n): fix bug 109920 (SEs cannot add to clone data size estimates) In the current implementation, once the first SE sets the data size estimate for a particular stage, there is no way for other SEs to add to it. Address this by adding Ha_clone_cbk::add_to_data_size_estimate method. MyRocks Clone (4/n): fix bug 109921 (Cannot clone with synchronization across SEs) The existing clone implementation synchronizes InnoDB and binlog directly. There is no way to make other storage engines join this synchronization, thus replace it with synchronizing by a performance_schema.log_status query. - Add table_log_status::get_row method to allow doing the log_status query from inside the server. - Handlerton: introduce two new methods: clone_precopy - currently unused, which will handle any non-consistent-snapshot copying before the main consistent copy, and set_log_stop - which provides the SE the clone stop position information. - Plugin: introduce Ha_clone_common_cbk, which derives from Ha_clone_cbk, and from which derive all the concrete classes: Client_Cbk, Local_Cbk & Server_Cbk. - Ha_clone_common_cbk contains two methods: precopy and synchronize_engines. The former calls precopy for all the participating clone SE handlertons, if implemented. The latter, only to be called from the main clone thread, performs the performance_schema.LOG_STATUS query, splits its result per-SE and calls handlerton set_log_stop for all SEs. - The pfs.log_status query still must be called by InnoDB at a certain point in clone. That means that InnoDB is still a special SE in that it must always come first in the clone locator vectors. This will also handle donor waits on clone restarts, as that is implemented in InnoDB-specific code. To ensure this, add a helper clone_hton.cc:make_innodb_first which shuffles the vectors as needed. Call it in all the clone-starting code paths, add asserts that InnoDB comes first. - Since hton_clone_copy will call into InnoDB clone copy, which at certain point will synchronize engines, calling into the plugin again, and the calling into SEs again, introduce a backup locator vector Ha_clone_cbk::all_locators, and its setters/getters set_all_locators, reset_all_locators, get_all_locators. Use them in the re-entrant hton_clone_copy code. - Introduce a new InnoDB clone state named SST COPY, which calls precopy and has zero files and chunks set to copy in the regular InnoDB clone way, i.e. all it does it the precopy. - The redo log archiving keeps running throughout the precopy stage. - On stage transition from the precopy to the redo log copy. call synchronize. - Implement innodb_clone_set_log_stop by parsing out the desired clone stop LSN from the passed JSON from the log_status query. - Make InnoDB redo log archiving stop at that LSN instead of on reaching the end of redo log. - Support stopping redo log archiving in a middle of a log block. As the old implementation may encounter this case only at the end of the redo log, it took that block from the log buffer in memory. This no longer works with the new implementation, where the needed block might reside on disk only. Address this by forcing the whole block containing the target LSN to disk, reading it from there, and patching its memory copy to stop at the required LSN before giving it for the clone to copy. - To implement redo log block reading on a running server, adapt crash recovery function recv_read_log_seg to work both for crash recovery and on a running server. This is similar to what Percona Server does for its changed page tracking feature. - Make sure not to archive not-yet-written log: pfs.log_status query gets the log system LSN, which is the value of highest log reservation. Thus in an archiving iteration, archive up to the lower values of log archiving LSN, and write_lsn. Set the log archiving LSN to LSN_MAX at clone initialization, so that clone archiving may proceed until the stop LSN is set. - The above implementation results in cloned LSN != log end LSN for the cloned instance, thus adjust crash recovery to work with that. The existing crash recovery throws away any crash-recovered metadata and reads its snapshot from the data files, whereas the snapshot was not written in any consistency with the clone LSN. Fix by no longer throwing away the crash-recovered InnoDB dynamic metadata and by not re-reading the the last metadata snapshot. This forces also not to re-serialize the metadata in the crash recovery as that writes redo log which is to be avoided during clone crash recovery. - Additionally, at the start of the redo log copy the old implementation requested all InnoDB updates to be flushed to disk immediately, to avoid clone recovery generating new redo log. This is also disabled as it becomes redundant with the fix. This issue is similar to Percona XtraBackup https://jira.percona.com/browse/PXB-1634 - Disable clone.local_xa test because the new synchronization no longer blocks on XA transactions. - Re-record two clone-related tests in the component_keyring_file MTR suite. - Temporarily workaround the MyRocks bug where log_status query hangs if RocksDB file deletions were not disabled first. Note that this change breaks some of the clone-related group replication MTR tests. As they are not tested in fb-mysql and have been already broken, this patch makes no effort to fix them. MyRocks Clone (5/n): fix bug 109922 (SEs do not coordinate clone rollbacks on instance startup) If InnoDB clone decides to rollback during its startup, other storage engines are not aware of this and proceed as usual, resulting in inconsistent instance between storage engines. Ideally this should be fixed through a 2PC-like design where all the SEs vote to rollback the clone or to continue during the startup, but for MyRocks clone a simpler solution suffices: make InnoDB write a special marker file if it decides to rollback on clone startup. MyRocks Clone (6/n): on remote clone error, always send COM_ACK to InnoDB too In the case of multithreaded clone, the waits for the donor threads for the current clone stage to complete are implemented fully in InnoDB, even though that, and error handling, should be plugin-level features. With two storage engines participating in the clone this allows the situation where the other storage engine client reports an error with COM_ACK to one donor thread, while some other threads are in InnoDB clone state change waits. Those waits can abort if InnoDB clone gets an error from the client, but they know nothing about other SE errors, and will be aborted only by timeouts. As a minimal fix that does not involve pulling the whole error handling and state change waiting logic to the plugin layer, make the clone client always send an error-containing COM_ACK to InnoDB too, whenever it sends one to another storage engine. MyRocks Clone (7/n): MyRocks clone implementation - Cleanup any stale RocksDB checkpoint directories on server shutdown too. - For MyRocks clone network protocol, define four packet types (clone/common.h): - LOCATOR_V1: MyRocks-specific clone locator, consisting of a monotonically increasing clone session ID for the donor with an optional applier state (an array of file IDs and applied offsets) for restarts, or a zero in which case it is a version check empty locator used for handshakes. - FILE_NAME_MAP_V1: establishes a relation between a file ID, its name and target size. - FILE_CHUNK_V1: cloned file chunk data - ADD_ESTIMATE_V1: how many bytes should be added to the estimated clone data size. - All packet types share the common package header, which stores the version, the payload length without the header, and the packet type. - (clone/common.h): classes locator, file_metadata, chunk_metadata, & estimate_metadata correspond to the four packet types. They are subclassed on the donor side to add serialization routines and other donor-specific code and on the client side to add deserialization routines and other client-specific code. Re. serialization/deserialization, an exception is the locator, whose routines are implemented in the common code, because it may be passed both - ways. - (clone/common.h): Introduce a session class that handles the task ID set and errors, which is common for both donor and client. For errors, allow saving one error, with an optional associated path and errno, that can be set by any thread and later reported to the clone plugin. - (clone/common.cc) Setup MyRocks clone capabilities as HA_CLONE_HYBRID (not - applicable but matches InnoDB), HA_CLONE_MULTI_TASK (can clone with multiple - threads), & HA_CLONE_RESTART (can restart after a network I/O error). - (clone/common.cc) Enable O_DIRECT for clone file I/O if, on client if rocksdb_use_direct_io_for_flush_and_compaction is enabled, and on donor if rocksdb_use_direct_reads is enabled. - On the donor side (clone/donor.cc), encapsulate RocksDB checkpoints in a rdb_checkpoint class. Each checkpoint gets its own ID, which is also a donor session ID. The ID is constant for rolling checkpoints. - Subclass the common file_metadata to donor_file_metadata, chunk_metadata to donor_chunk_metadata, estimate_metadata to donor_estimate. - Implement donor going through a sequence of states: NONE, ROLLING, FINAL, FINAL_WITH_LOGS. To support multiple clone donor sessions, implement a manager class that looks up the donor instance by locator. - Implement MyRocks clone donor interface: - rocksdb_clone_start: handle the possible start modes: - HA_CLONE_MODE_START: create and register a new donor instance, register its master task ID. - HA_CLONE_MODE_ADD_TASK: lookup the existing donor instance, register a new task ID. - HA_CLONE_MODE_RESTART: lookup the existing donor instance, wait for the old master session to drop it by master task ID reference count, then process the locator applier state to reset the donor state. - rocksdb_clone_precopy: lookup the donor instance, create a RocksDB snapshot if state == NONE, otherwise send files by chunks and roll the checkpoint as needed. - rocksdb_clone_set_log_stop: add the WAL files to the clone file to copy set. - rocksdb_clone_copy: copy the clone files in any donor state. - rocksdb_clone_ack: acknowledge errors sent by client. - rocksdb_clone_end: for master task ID, stop accepting new worker join requests, wait until existing workers quit, and, in the case of restartable error, wait for any restarts to happen. Then decrement ref count for the master task ID (which will handle both the cases of restarted and not restarted) clone. For all other task IDs, simply unregister them. - On the client side (clone/client.cc), have file_in_progress class to track each individual file progress during application, and completed_file for fully-applied files. - Subclass the common file_metadata to client_file_metadata, implement its reading from the net buffer. Likewise for client_chunk_metadata subclassing chunk_metadata, and client_estimate for estimate_metadata. - TODO client class. - Implement MyRocks clone client interface: - rocksdb_clone_apply_begin: handle the possible start modes: - HA_CLONE_MODE_START: initialize the client instance for a new clone - HA_CLONE_MODE_ADD_TASK: register a new task ID for an ongoing clone - HA_CLONE_MODE_RESTART: restart the interrupted clone session - HA_CLONE_MODE_VERSION: prepare new version locator - rocksdb_clone_apply: if passed an error code, save it. Otherwise verify the passed buffer and dispatch ADD_ESTIMATE_V1, FILE_NAME_MAP_V1, & FILE_CHUNK_V1 packets accordingly. - rocksdb_clone_apply_end: for master task, wait for all other tasks to complete and finish the clone. For both master and non-master tasks unregister their IDs. - Add rocksdb_sys_vars tests for the new clone-related system variables. They are included in the current patch and not the MTR one in order not to break the existing rocksdb_sys_vars.all_vars test. - On instance startup, do clone fixup (clone/common.cc). It consists of: - Check whether InnoDB clone rollback marker is present. If yes, force rollback of MyRocks clone. - If the cloned temporary datadir contains clone-in-progress marker, assume half-completed clone and abort. - Move the cloned datadirs and WAL dirs in-place. For the datadir, do a directory-level rename with a save and later delete of the old datadir. For the WAL dir do not try to do parent directory-level renames because the mysqld process may not have permissions to delete that directory. Instead, rename all the old files individually, move new files in, delete the old files. - For all the rename operations, handle them returning EXDEV error (a rename would cross a filesystem boundary) by falling back to copy and delete. As we expect that it is the cloned RocksDB log files that are most affected by this, the total copy I/O wouldn't be large, and so the default MySQL copy routine my_copy is used, without extra copy throughput optimizations. Add a fault injection point for MTR to test this code paths. - Remove any RocksDB clone checkpoints found. - On instance startup, always move WALs to the target directory if they are found in the rocksdb_datadir yet the rocksdb_wal_dir is different. This handles the case of setting rocksdb_wal_dir while instance is shutdown, both for clone fixup and regular operation. - Add PFS instrumentation for all the instrumentable entities in this patch (mutexes, rwlocks, file I/O). MyRocks Clone (8/n): MyRocks clone MTR tests - Add a new MTR test suite rocksdb_clone - Support new variable MTR clone/include/clone_command.inc: clone_single_threaded, setting which forces single-threaded clone operation. - Reuse a big part of InnoDB clone tests by including their test files, with the exception of tests involving DDL, as concurrent DDL is not supported by MyRocks clone at the moment, and debug tests that adjust cloned instance LSN, because they desynchronize storage engines. The reused tests are: local_basic, local_boundary, local_dml, local_dml_auto_tune, local_dml_no_redo, local_insert, local_stage, monitor_progress, remote_basic, remote_basic_replace, remote_boundary, remote_dml, remote_dml_auto_tune, remote_dml_gtid, remote_dml_no_binlog, remote_dml_no_ssl, remote_dml_recovery, remote_dml_replace, remote_dml_ssl, remote_dml_ssl_certificate, remote_dml_upgrade, remote_insert, remote_monitor_progress, remote_restart, remote_skip_validation, remote_stage, remote_stage_restart. - As a part of the above, add MyRocks support to local_basic and monitor_progress MTR tests. - Besides the reused InnoDB clone tests, add some new MyRocks-specific tests: - local_minimal_1_thread, which is the simplest possible local clone test. - local_minimal_multithreaded, the simplest possible but multithreaded local clone test. - local_minimal_direct_io, which is the simplest possible local clone test in all combinations of rocksdb_use_direct_reads and rocksdb_use_direct_io_for_flush_and_compaction settings, which are the O_DIRECT-enabling options. - local_dml_o_direct_donor, local_dml_o_direct_client, local_dml_o_direct_both that reuse InnoDB local_dml test, but combined with either or both of the O_DIRECT-enabling options. - local_minimal_wal_dir, which clones with various rocksdb_wal_dir settings on donor and client. - local_minimal_wal_dir_copy_delete, which uses a debug injection to test the move-across-filesystem-boundary code path in clone startup. - remote_minimal_direct_io_{1, 2, 3, 4} tests for all the combinations of O_DIRECT-enabling options for donor and client instances. - remote_fail_apply, which uses a debug injection to fail the clone client during remote clone. - remote_inplace_restart, which performs a remote in-place clone with a simulated transient error on the client. - remote_kill_donor, which kill the donor instance at various points of remote cloning. - rolling_checkpoint for testing various combinations of clone rolling checkpoint tuning options. - three_se_basic to test a simple clone across InnoDB, MyRocks, & Blackhole engines. Fix moving RocksDB WALs on startup across file system boundaries Reviewed By: sunshine-Chun Differential Revision: D43160162 Pulled By: hermanlee -------------------------------------------------------------------------------- Rdb clone fixes (facebook#1271) Summary: Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir Add asserts and __builtin_unreachable after exhaustive switch statements To help the compiler with the case where it cannot tell that the switch is actually exhaustive. This fixes, with GCC: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type] 295 | } | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type] 821 | } | ^ Remove redundant constexpr This fixes, under GCC: In file included from /usr/include/c++/11/cassert:44, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34, from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ 288 | assert(m_files_in_progress.empty()); | ~~~~~~~~~~~~~~~~~~~~~~~~~^~ In file included from /usr/include/c++/11/unordered_map:47, from /usr/include/c++/11/functional:61, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17: /usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here 305 | empty() const noexcept | ^~~~~ Factor out clone metadata deserialization buffer length checking Add an overflow check and a cast, extract resulting code to a new helper function. This fixes GCC warnings: Fix a cast "const" was redundant because it indicated that the pointer itself, not its contents, were const. At the same time replace the C style cast with a C++ one, and drop a redundant trailing semicolon. This fixes a GCC warning: In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’: /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers] 38 | rdb_netbuf_store_index((uchar *const)buf.data(), index_id); | ^~~~~~~~~~~~~~~~~~~~~~~~ Use GCC-compatible attribute((nonnull)) syntax Clang allows adding this attribute to individual function parameters, while GCC only allows annotating the whole declaration with relevant parameter indexes. Use the GCC-compatible syntax. This fixes In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32, from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24: /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’ 495 | int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes] 100 | #define MY_ATTRIBUTE(A) __attribute__(A) | ^ /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’ 496 | const char *const to MY_ATTRIBUTE((__nonnull__)), | ^~~~~~~~~~~~ Handle my_tell return value correctly. Fixes facebook#1270 The incorrect my_tell value handling was pointed out by GCC: [ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’: /home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits] 312 | if (file_size < 0) { | ~~~~~~~~~~^~~ cc1plus: all warnings being treated as errors Pull Request resolved: facebook#1271 Reviewed By: sunshine-Chun Differential Revision: D43259662 Pulled By: hermanlee -------------------------------------------------------------------------------- Clone: fix race condition that may skip some SST files (facebook#1274) Summary: A donor thread in the main copy loop took a file to send, released the donor state mutex, and then opened the file. If an ENOENT was received at this point, it was assumed that this was a stale SST file from an older checkpoint that was rolled since. Because the donor state mutex was released between taking of the file and opening it, the following race was possible: 1) Thread 1 takes the file 2) Thread 2 decides to roll the checkpoint, the old checkpoint is deleted 3) Thread 1 tries to open the file, gets ENOENT 4) Thread 2 creates the new checkpoint, the file re-appears, but it's too late. Rolling the checkpoint in a donor state mutex critical section is a possible fix, but such section would do a lot of I/O, serializing the parallel threads. Instead, fix by taking the file and opening it in the same critical section. Pull Request resolved: facebook#1274 Reviewed By: sunshine-Chun Differential Revision: D43629546 Pulled By: hermanlee -------------------------------------------------------------------------------- MyRocks clone: add missing donor mutex unlocks on error exits (facebook#1317) Summary: Pull Request resolved: facebook#1317 Differential Revision: D46564079
The joys of auto types, from git blame plugin/clone/src/clone_os.cc
The warning is:
The text was updated successfully, but these errors were encountered: