Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Warning from compiling clone_os.cc #1270

Closed
mdcallag opened this issue Feb 9, 2023 · 6 comments
Closed

Warning from compiling clone_os.cc #1270

mdcallag opened this issue Feb 9, 2023 · 6 comments

Comments

@mdcallag
Copy link
Contributor

mdcallag commented Feb 9, 2023

The joys of auto types, from git blame plugin/clone/src/clone_os.cc

d46fb7f495a1 (Laurynas Biveinis    2023-02-02 09:24:09 -0800 308)     if (actual_size > length) {
d46fb7f495a1 (Laurynas Biveinis    2023-02-02 09:24:09 -0800 309)       assert(block_pad_len > 0);
d46fb7f495a1 (Laurynas Biveinis    2023-02-02 09:24:09 -0800 310)       assert(to_file.o_direct_uneven_file_size);
d46fb7f495a1 (Laurynas Biveinis    2023-02-02 09:24:09 -0800 311)       const auto file_size = my_tell(to_file.file_desc, MYF(MY_WME));
d46fb7f495a1 (Laurynas Biveinis    2023-02-02 09:24:09 -0800 312)       if (file_size < 0) {
d46fb7f495a1 (Laurynas Biveinis    2023-02-02 09:24:09 -0800 313)         return ER_ERROR_ON_WRITE;
d46fb7f495a1 (Laurynas Biveinis    2023-02-02 09:24:09 -0800 314)       }

The warning is:

/home/mdcallag/git/fbmysql-56/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/mdcallag/git/fbmysql-56/plugin/clone/src/clone_os.cc:312:21: warning: comparison of unsigned expression in ‘< 0’ is always false [-Wtype-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
@mdcallag mdcallag changed the title Warning Warning from compiling clone_os.cc Feb 9, 2023
@mdcallag
Copy link
Contributor Author

mdcallag commented Feb 9, 2023

Another one, but an error rather than a warning

In file included from /home/mdcallag/git/fbmysql-56/storage/rocksdb/ha_rocksdb.cc:87:
/home/mdcallag/git/fbmysql-56/storage/rocksdb/./clone/common.h: In member function ‘constexpr bool myrocks::clone::locator::has_applier_status() const’:
/home/mdcallag/git/fbmysql-56/storage/rocksdb/./clone/common.h:282:35: error: call to non-‘constexpr’ function ‘bool std::vector<_Tp, _Alloc>::empty() const [with _Tp = myrocks::clone::locator::file_state; _Alloc = std::allocator<myrocks::clone::locator::file_state>]’
  282 |     return !m_applier_status.empty();
      |             ~~~~~~~~~~~~~~~~~~~~~~^~

@laurynas-biveinis
Copy link
Contributor

/home/mdcallag/git/fbmysql-56/plugin/clone/src/clone_os.cc:312:21: warning: comparison of unsigned expression in ‘< 0’ is always false [-Wtype-limits]
312 | if (file_size < 0) {
| ~~~~~~~~~~^~~

Should read file_size == MY_FILEPOS_ERROR, will PR as soon as the trunk status is clear

@laurynas-biveinis
Copy link
Contributor

/home/mdcallag/git/fbmysql-56/storage/rocksdb/./clone/common.h: In member function ‘constexpr bool myrocks::clone::locator::has_applier_status() const’:
/home/mdcallag/git/fbmysql-56/storage/rocksdb/./clone/common.h:282:35: error: call to non-‘constexpr’ function ‘bool std::vector<_Tp, _Alloc>::empty() const [with _Tp = myrocks::clone::locator::file_state; _Alloc = std::allocatormyrocks::clone::locator::file_state]’
282 | return !m_applier_status.empty();
| ~~~~~~~~~~~~~~~~~~~~~~^~

Was already fixed last week in the clone PR, apparently an older version had been merged instead

@hermanlee
Copy link
Contributor

I'm in the process of fixing this. A slightly older version of the PR was pushed, and the patch summaries were also missing. The fixed patch should be on its way via a forced update.

laurynas-biveinis added a commit to laurynas-biveinis/mysql-5.6 that referenced this issue Feb 10, 2023
laurynas-biveinis added a commit to laurynas-biveinis/mysql-5.6 that referenced this issue Feb 10, 2023
The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors
laurynas-biveinis added a commit to laurynas-biveinis/mysql-5.6 that referenced this issue Feb 15, 2023
The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors
facebook-github-bot pushed a commit that referenced this issue Feb 28, 2023
Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes #1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: #1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 2245706
alanliang pushed a commit to alanliang/mysql-5.6 that referenced this issue Mar 4, 2023
Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~


Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook#1271
GitHub Author: Laurynas Biveinis <laurynas.biveinis@gmail.com>

Test Plan: Imported from GitHub, without a `Test Plan:` line.

Reviewers: chni

Reviewed By: chni

Subscribers: webscalesql-eng@fb.com

Differential Revision: https://phabricator.intern.facebook.com/D43259662

Tags: accept2ship
@laurynas-biveinis
Copy link
Contributor

Should be fixed by 5728dd7

inikep pushed a commit to inikep/percona-server that referenced this issue May 9, 2023
Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook/mysql-5.6#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
inikep pushed a commit to inikep/percona-server that referenced this issue May 9, 2023
Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook/mysql-5.6#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
inikep pushed a commit to inikep/percona-server that referenced this issue May 10, 2023
Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook/mysql-5.6#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
inikep pushed a commit to inikep/mysql-5.6 that referenced this issue May 15, 2023
Summary: Pull Request resolved: facebook#1250

MyRocks Clone (1/n): make InnoDB clone MTR tests SE-agnostic

The MyRocks clone MTR tests will reuse the InnoDB tests by including them.
Adjust the tests so that they work this way, i.e. make them storage
engine-agnostic by checking the default_storage_engine system variable and
guarding InnoDB-specific tests this way. This includes the tests that
artificially desync the InnoDB redo log position.

MyRocks Clone (2/n): fix bug 109919 (Clone plugin does not support arbitrary
length O_DIRECT files)

The existing clone plugin I/O routines handle O_DIRECT files with the assumption
that their sizes are always a multiple of a block size. This is too restrictive
because in general O_DIRECT files can be of any size and MyRocks files actually are.

Add a flag to Ha_clone_file to support arbitrary file lengths under O_DIRECT:
o_direct_uneven_file_size. Pass it to clone_client.h/cc
Client::get_aligned_buffer to allocate the buffer that bumps the length to the
next CLONE_OS_ALIGN multiple if it's not already one. Introduce a
pad_for_o_direct helper that performs this calculation for the passed length as
needed.

For writes, if O_DIRECT is enabled and the buffer length is not a multiple of
CLONE_OS_ALIGN, pad it with zeros to the next multiple. Then check the actual
bytes written, and if any padding bytes were actually written, call my_chsize to
truncate the file to the needed size.

MyRocks Clone (3/n): fix bug 109920 (SEs cannot add to clone data size estimates)

In the current implementation, once the first SE sets the data size estimate for
a particular stage, there is no way for other SEs to add to it. Address this by
adding Ha_clone_cbk::add_to_data_size_estimate method.

MyRocks Clone (4/n): fix bug 109921 (Cannot clone with synchronization across SEs)

The existing clone implementation synchronizes InnoDB and binlog directly. There
is no way to make other storage engines join this synchronization, thus replace
it with synchronizing by a performance_schema.log_status query.

- Add table_log_status::get_row method to allow doing the log_status query from
  inside the server.
- Handlerton: introduce two new methods: clone_precopy - currently unused, which
  will handle any non-consistent-snapshot copying before the main consistent
  copy, and set_log_stop - which provides the SE the clone stop position
  information.
- Plugin: introduce Ha_clone_common_cbk, which derives from Ha_clone_cbk, and
  from which derive all the concrete classes: Client_Cbk, Local_Cbk &
  Server_Cbk.
- Ha_clone_common_cbk contains two methods: precopy and synchronize_engines. The
  former calls precopy for all the participating clone SE handlertons, if
  implemented. The latter, only to be called from the main clone thread,
  performs the performance_schema.LOG_STATUS query, splits its result per-SE and
  calls handlerton set_log_stop for all SEs.
- The pfs.log_status query still must be called by InnoDB at a certain point in
  clone. That means that InnoDB is still a special SE in that it must always
  come first in the clone locator vectors. This will also handle donor waits on
  clone restarts, as that is implemented in InnoDB-specific code. To ensure
  this, add a helper clone_hton.cc:make_innodb_first which shuffles the vectors
  as needed. Call it in all the clone-starting code paths, add asserts that
  InnoDB comes first.
- Since hton_clone_copy will call into InnoDB clone copy, which at certain point
  will synchronize engines, calling into the plugin again, and the calling into
  SEs again, introduce a backup locator vector Ha_clone_cbk::all_locators, and
  its setters/getters set_all_locators, reset_all_locators, get_all_locators.
  Use them in the re-entrant hton_clone_copy code.
- Introduce a new InnoDB clone state named SST COPY, which calls precopy and has
  zero files and chunks set to copy in the regular InnoDB clone way, i.e. all it
  does it the precopy.
- The redo log archiving keeps running throughout the precopy stage.
- On stage transition from the precopy to the redo log copy. call synchronize.
- Implement innodb_clone_set_log_stop by parsing out the desired clone stop LSN
  from the passed JSON from the log_status query.
- Make InnoDB redo log archiving stop at that LSN instead of on reaching the end
  of redo log.
- Support stopping redo log archiving in a middle of a log block. As the old
  implementation may encounter this case only at the end of the redo log, it
  took that block from the log buffer in memory. This no longer works with the
  new implementation, where the needed block might reside on disk only. Address
  this by forcing the whole block containing the target LSN to disk, reading it
  from there, and patching its memory copy to stop at the required LSN before
  giving it for the clone to copy.
- To implement redo log block reading on a running server, adapt crash recovery
  function recv_read_log_seg to work both for crash recovery and on a running
  server. This is similar to what Percona Server does for its changed page
  tracking feature.
- Make sure not to archive not-yet-written log: pfs.log_status query gets the
  log system LSN, which is the value of highest log reservation. Thus in an
  archiving iteration, archive up to the lower values of log archiving LSN, and
  write_lsn. Set the log archiving LSN to LSN_MAX at clone initialization, so
  that clone archiving may proceed until the stop LSN is set.
- The above implementation results in cloned LSN != log end LSN for the cloned
  instance, thus adjust crash recovery to work with that. The existing crash
  recovery throws away any crash-recovered metadata and reads its snapshot from
  the data files, whereas the snapshot was not written in any consistency with
  the clone LSN. Fix by no longer throwing away the crash-recovered InnoDB
  dynamic metadata and by not re-reading the the last metadata snapshot. This
  forces also not to re-serialize the metadata in the crash recovery as that
  writes redo log which is to be avoided during clone crash recovery.
- Additionally, at the start of the redo log copy the old implementation
  requested all InnoDB updates to be flushed to disk immediately, to avoid clone
  recovery generating new redo log. This is also disabled as it becomes
  redundant with the fix. This issue is similar to Percona XtraBackup
  https://jira.percona.com/browse/PXB-1634
- Disable clone.local_xa test because the new synchronization no longer blocks
  on XA transactions.
- Re-record two clone-related tests in the component_keyring_file MTR suite.
- Temporarily workaround the MyRocks bug where log_status query hangs if RocksDB
  file deletions were not disabled first.

Note that this change breaks some of the clone-related group replication MTR
tests. As they are not tested in fb-mysql and have been already broken, this
patch makes no effort to fix them.

MyRocks Clone (5/n): fix bug 109922 (SEs do not coordinate clone rollbacks on
instance startup)

If InnoDB clone decides to rollback during its startup, other storage engines
are not aware of this and proceed as usual, resulting in inconsistent instance
between storage engines. Ideally this should be fixed through a 2PC-like design
where all the SEs vote to rollback the clone or to continue during the startup,
but for MyRocks clone a simpler solution suffices: make InnoDB write a special
marker file if it decides to rollback on clone startup.

MyRocks Clone (6/n): on remote clone error, always send COM_ACK to InnoDB too

In the case of multithreaded clone, the waits for the donor threads for the
current clone stage to complete are implemented fully in InnoDB, even though
that, and error handling, should be plugin-level features. With two storage
engines participating in the clone this allows the situation where the other
storage engine client reports an error with COM_ACK to one donor thread, while
some other threads are in InnoDB clone state change waits. Those waits can abort
if InnoDB clone gets an error from the client, but they know nothing about other
SE errors, and will be aborted only by timeouts.

As a minimal fix that does not involve pulling the whole error handling and
state change waiting logic to the plugin layer, make the clone client always
send an error-containing COM_ACK to InnoDB too, whenever it sends one to another
storage engine.

MyRocks Clone (7/n): MyRocks clone implementation

- Cleanup any stale RocksDB checkpoint directories on server shutdown too.

- For MyRocks clone network protocol, define four packet types (clone/common.h):
  - LOCATOR_V1: MyRocks-specific clone locator, consisting of a monotonically
    increasing clone session ID for the donor with an optional applier
    state (an array of file IDs and applied offsets) for restarts, or a zero
    in which case it is a version check empty locator used for handshakes.
  - FILE_NAME_MAP_V1: establishes a relation between a file ID, its name and
    target size.
  - FILE_CHUNK_V1: cloned file chunk data
  - ADD_ESTIMATE_V1: how many bytes should be added to the estimated clone data
    size.
- All packet types share the common package header, which stores the version,
  the payload length without the header, and the packet type.
- (clone/common.h): classes locator, file_metadata, chunk_metadata, &
  estimate_metadata correspond to the four packet types. They are subclassed on
  the donor side to add serialization routines and other donor-specific code and
  on the client side to add deserialization routines and other client-specific
  code. Re. serialization/deserialization, an exception is the locator, whose
  routines are implemented in the common code, because it may be passed both
  - ways.
- (clone/common.h): Introduce a session class that handles the task ID set and
  errors, which is common for both donor and client. For errors, allow saving
  one error, with an optional associated path and errno, that can be set by any
  thread and later reported to the clone plugin.
- (clone/common.cc) Setup MyRocks clone capabilities as HA_CLONE_HYBRID (not
  - applicable but matches InnoDB), HA_CLONE_MULTI_TASK (can clone with multiple
  - threads), & HA_CLONE_RESTART (can restart after a network I/O error).
- (clone/common.cc) Enable O_DIRECT for clone file I/O if, on client if
  rocksdb_use_direct_io_for_flush_and_compaction is enabled, and on donor if
  rocksdb_use_direct_reads is enabled.
- On the donor side (clone/donor.cc), encapsulate RocksDB checkpoints in a
  rdb_checkpoint class. Each checkpoint gets its own ID, which is also a donor
  session ID. The ID is constant for rolling checkpoints.
- Subclass the common file_metadata to donor_file_metadata, chunk_metadata to
  donor_chunk_metadata, estimate_metadata to donor_estimate.
- Implement donor going through a sequence of states: NONE, ROLLING, FINAL,
  FINAL_WITH_LOGS. To support multiple clone donor sessions, implement a manager
  class that looks up the donor instance by locator.
- Implement MyRocks clone donor interface:
  - rocksdb_clone_start: handle the possible start modes:
    - HA_CLONE_MODE_START: create and register a new donor instance, register
      its master task ID.
    - HA_CLONE_MODE_ADD_TASK: lookup the existing donor instance, register a new
      task ID.
    - HA_CLONE_MODE_RESTART: lookup the existing donor instance, wait for the
      old master session to drop it by master task ID reference count, then
      process the locator applier state to reset the donor state.
  - rocksdb_clone_precopy: lookup the donor instance, create a RocksDB snapshot
    if state == NONE, otherwise send files by chunks and roll the checkpoint as
    needed.
  - rocksdb_clone_set_log_stop: add the WAL files to the clone file to copy set.
  - rocksdb_clone_copy: copy the clone files in any donor state.
  - rocksdb_clone_ack: acknowledge errors sent by client.
  - rocksdb_clone_end: for master task ID, stop accepting new worker join
    requests, wait until existing workers quit, and, in the case of restartable
    error, wait for any restarts to happen. Then decrement ref count for the
    master task ID (which will handle both the cases of restarted and not
    restarted) clone. For all other task IDs, simply unregister them.
- On the client side (clone/client.cc), have file_in_progress class to track
  each individual file progress during application, and completed_file for
  fully-applied files.
- Subclass the common file_metadata to client_file_metadata, implement its
  reading from the net buffer. Likewise for client_chunk_metadata subclassing
  chunk_metadata, and client_estimate for estimate_metadata.
- TODO client class.
- Implement MyRocks clone client interface:
  - rocksdb_clone_apply_begin: handle the possible start modes:
    - HA_CLONE_MODE_START: initialize the client instance for a new clone
    - HA_CLONE_MODE_ADD_TASK: register a new task ID for an ongoing clone
    - HA_CLONE_MODE_RESTART: restart the interrupted clone session
    - HA_CLONE_MODE_VERSION: prepare new version locator
  - rocksdb_clone_apply: if passed an error code, save it. Otherwise verify the
    passed buffer and dispatch ADD_ESTIMATE_V1, FILE_NAME_MAP_V1, &
    FILE_CHUNK_V1 packets accordingly.
  - rocksdb_clone_apply_end: for master task, wait for all other tasks to
    complete and finish the clone. For both master and non-master tasks
    unregister their IDs.
- Add rocksdb_sys_vars tests for the new clone-related system variables. They
  are included in the current patch and not the MTR one in order not to break
  the existing rocksdb_sys_vars.all_vars test.
- On instance startup, do clone fixup (clone/common.cc). It consists of:
  - Check whether InnoDB clone rollback marker is present. If yes, force
    rollback of MyRocks clone.
  - If the cloned temporary datadir contains clone-in-progress marker, assume
    half-completed clone and abort.
  - Move the cloned datadirs and WAL dirs in-place. For the datadir, do a
    directory-level rename with a save and later delete of the old datadir. For
    the WAL dir do not try to do parent directory-level renames because the
    mysqld process may not have  permissions to delete that directory. Instead,
    rename all the old files individually, move new files in, delete the old
    files.
  - For all the rename operations, handle them returning EXDEV error (a rename
    would cross a filesystem boundary) by falling back to copy and delete. As we
    expect that it is the cloned RocksDB log files that are most affected by
    this, the total copy I/O  wouldn't be large, and so the default MySQL copy
    routine my_copy is used, without extra copy throughput optimizations. Add a
    fault injection point for MTR to test this code paths.
  - Remove any RocksDB clone checkpoints found.
- On instance startup, always move WALs to the target directory if they are
  found in the rocksdb_datadir yet the rocksdb_wal_dir is different. This
  handles the case of setting rocksdb_wal_dir while instance is shutdown, both
  for clone fixup and regular operation.
- Add PFS instrumentation for all the instrumentable entities in this
  patch (mutexes, rwlocks, file I/O).

MyRocks Clone (8/n): MyRocks clone MTR tests

- Add a new MTR test suite rocksdb_clone
- Support new variable MTR clone/include/clone_command.inc:
  clone_single_threaded, setting which forces single-threaded clone operation.
- Reuse a big part of InnoDB clone tests by including their test files, with the
  exception of tests involving DDL, as concurrent DDL is not supported by
  MyRocks clone at the moment, and debug tests that adjust cloned instance LSN,
  because they desynchronize storage engines. The reused tests are: local_basic,
  local_boundary, local_dml, local_dml_auto_tune, local_dml_no_redo,
  local_insert, local_stage, monitor_progress, remote_basic,
  remote_basic_replace, remote_boundary, remote_dml, remote_dml_auto_tune,
  remote_dml_gtid, remote_dml_no_binlog, remote_dml_no_ssl, remote_dml_recovery,
  remote_dml_replace, remote_dml_ssl, remote_dml_ssl_certificate,
  remote_dml_upgrade, remote_insert, remote_monitor_progress, remote_restart,
  remote_skip_validation, remote_stage, remote_stage_restart.
- As a part of the above, add MyRocks support to local_basic and
  monitor_progress MTR tests.
- Besides the reused InnoDB clone tests, add some new MyRocks-specific tests:
- local_minimal_1_thread, which is the simplest possible local clone test.
- local_minimal_multithreaded, the simplest possible but multithreaded local
  clone test.
- local_minimal_direct_io, which is the simplest possible local clone test in
  all combinations of rocksdb_use_direct_reads and
  rocksdb_use_direct_io_for_flush_and_compaction settings, which are the
  O_DIRECT-enabling options.
- local_dml_o_direct_donor, local_dml_o_direct_client, local_dml_o_direct_both
  that reuse InnoDB local_dml test, but combined with either or both of the
  O_DIRECT-enabling options.
- local_minimal_wal_dir, which clones with various rocksdb_wal_dir settings on
  donor and client.
- local_minimal_wal_dir_copy_delete, which uses a debug injection to test the
  move-across-filesystem-boundary code path in clone startup.
- remote_minimal_direct_io_{1, 2, 3, 4} tests for all the combinations of
  O_DIRECT-enabling options for donor and client instances.
- remote_fail_apply, which uses a debug injection to fail the clone client
  during remote clone.
- remote_inplace_restart, which performs a remote in-place clone with a
  simulated transient error on the client.
- remote_kill_donor, which kill the donor instance at various points of remote
  cloning.
- rolling_checkpoint for testing various combinations of clone rolling
  checkpoint tuning options.
- three_se_basic to test a simple clone across InnoDB, MyRocks, & Blackhole
  engines.

Fix moving RocksDB WALs on startup across file system boundaries

Reviewed By: sunshine-Chun

Differential Revision: D43160162

Pulled By: hermanlee

fbshipit-source-id: 5358a57

--------------------------------------------------------------------------------

Rdb clone fixes (facebook#1271)

Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 2245706

--------------------------------------------------------------------------------

Clone: fix race condition that may skip some SST files (facebook#1274)

Summary:
A donor thread in the main copy loop took a file to send, released the donor state mutex, and then opened the file. If an ENOENT was received at this point, it was assumed that this was a stale SST file from an older checkpoint that was rolled since.

Because the donor state mutex was released between taking of the file and opening it, the following race was possible:
1) Thread 1 takes the file
2) Thread 2 decides to roll the checkpoint, the old checkpoint is deleted 3) Thread 1 tries to open the file, gets ENOENT
4) Thread 2 creates the new checkpoint, the file re-appears, but it's too late.

Rolling the checkpoint in a donor state mutex critical section is a possible fix, but such section would do a lot of I/O, serializing the parallel threads. Instead, fix by taking the file and opening it in the same critical section.

Pull Request resolved: facebook#1274

Reviewed By: sunshine-Chun

Differential Revision: D43629546

Pulled By: hermanlee

fbshipit-source-id: 6e7f315
inikep pushed a commit to inikep/mysql-5.6 that referenced this issue May 16, 2023
Summary: Pull Request resolved: facebook#1250

MyRocks Clone (1/n): make InnoDB clone MTR tests SE-agnostic

The MyRocks clone MTR tests will reuse the InnoDB tests by including them.
Adjust the tests so that they work this way, i.e. make them storage
engine-agnostic by checking the default_storage_engine system variable and
guarding InnoDB-specific tests this way. This includes the tests that
artificially desync the InnoDB redo log position.

MyRocks Clone (2/n): fix bug 109919 (Clone plugin does not support arbitrary
length O_DIRECT files)

The existing clone plugin I/O routines handle O_DIRECT files with the assumption
that their sizes are always a multiple of a block size. This is too restrictive
because in general O_DIRECT files can be of any size and MyRocks files actually are.

Add a flag to Ha_clone_file to support arbitrary file lengths under O_DIRECT:
o_direct_uneven_file_size. Pass it to clone_client.h/cc
Client::get_aligned_buffer to allocate the buffer that bumps the length to the
next CLONE_OS_ALIGN multiple if it's not already one. Introduce a
pad_for_o_direct helper that performs this calculation for the passed length as
needed.

For writes, if O_DIRECT is enabled and the buffer length is not a multiple of
CLONE_OS_ALIGN, pad it with zeros to the next multiple. Then check the actual
bytes written, and if any padding bytes were actually written, call my_chsize to
truncate the file to the needed size.

MyRocks Clone (3/n): fix bug 109920 (SEs cannot add to clone data size estimates)

In the current implementation, once the first SE sets the data size estimate for
a particular stage, there is no way for other SEs to add to it. Address this by
adding Ha_clone_cbk::add_to_data_size_estimate method.

MyRocks Clone (4/n): fix bug 109921 (Cannot clone with synchronization across SEs)

The existing clone implementation synchronizes InnoDB and binlog directly. There
is no way to make other storage engines join this synchronization, thus replace
it with synchronizing by a performance_schema.log_status query.

- Add table_log_status::get_row method to allow doing the log_status query from
  inside the server.
- Handlerton: introduce two new methods: clone_precopy - currently unused, which
  will handle any non-consistent-snapshot copying before the main consistent
  copy, and set_log_stop - which provides the SE the clone stop position
  information.
- Plugin: introduce Ha_clone_common_cbk, which derives from Ha_clone_cbk, and
  from which derive all the concrete classes: Client_Cbk, Local_Cbk &
  Server_Cbk.
- Ha_clone_common_cbk contains two methods: precopy and synchronize_engines. The
  former calls precopy for all the participating clone SE handlertons, if
  implemented. The latter, only to be called from the main clone thread,
  performs the performance_schema.LOG_STATUS query, splits its result per-SE and
  calls handlerton set_log_stop for all SEs.
- The pfs.log_status query still must be called by InnoDB at a certain point in
  clone. That means that InnoDB is still a special SE in that it must always
  come first in the clone locator vectors. This will also handle donor waits on
  clone restarts, as that is implemented in InnoDB-specific code. To ensure
  this, add a helper clone_hton.cc:make_innodb_first which shuffles the vectors
  as needed. Call it in all the clone-starting code paths, add asserts that
  InnoDB comes first.
- Since hton_clone_copy will call into InnoDB clone copy, which at certain point
  will synchronize engines, calling into the plugin again, and the calling into
  SEs again, introduce a backup locator vector Ha_clone_cbk::all_locators, and
  its setters/getters set_all_locators, reset_all_locators, get_all_locators.
  Use them in the re-entrant hton_clone_copy code.
- Introduce a new InnoDB clone state named SST COPY, which calls precopy and has
  zero files and chunks set to copy in the regular InnoDB clone way, i.e. all it
  does it the precopy.
- The redo log archiving keeps running throughout the precopy stage.
- On stage transition from the precopy to the redo log copy. call synchronize.
- Implement innodb_clone_set_log_stop by parsing out the desired clone stop LSN
  from the passed JSON from the log_status query.
- Make InnoDB redo log archiving stop at that LSN instead of on reaching the end
  of redo log.
- Support stopping redo log archiving in a middle of a log block. As the old
  implementation may encounter this case only at the end of the redo log, it
  took that block from the log buffer in memory. This no longer works with the
  new implementation, where the needed block might reside on disk only. Address
  this by forcing the whole block containing the target LSN to disk, reading it
  from there, and patching its memory copy to stop at the required LSN before
  giving it for the clone to copy.
- To implement redo log block reading on a running server, adapt crash recovery
  function recv_read_log_seg to work both for crash recovery and on a running
  server. This is similar to what Percona Server does for its changed page
  tracking feature.
- Make sure not to archive not-yet-written log: pfs.log_status query gets the
  log system LSN, which is the value of highest log reservation. Thus in an
  archiving iteration, archive up to the lower values of log archiving LSN, and
  write_lsn. Set the log archiving LSN to LSN_MAX at clone initialization, so
  that clone archiving may proceed until the stop LSN is set.
- The above implementation results in cloned LSN != log end LSN for the cloned
  instance, thus adjust crash recovery to work with that. The existing crash
  recovery throws away any crash-recovered metadata and reads its snapshot from
  the data files, whereas the snapshot was not written in any consistency with
  the clone LSN. Fix by no longer throwing away the crash-recovered InnoDB
  dynamic metadata and by not re-reading the the last metadata snapshot. This
  forces also not to re-serialize the metadata in the crash recovery as that
  writes redo log which is to be avoided during clone crash recovery.
- Additionally, at the start of the redo log copy the old implementation
  requested all InnoDB updates to be flushed to disk immediately, to avoid clone
  recovery generating new redo log. This is also disabled as it becomes
  redundant with the fix. This issue is similar to Percona XtraBackup
  https://jira.percona.com/browse/PXB-1634
- Disable clone.local_xa test because the new synchronization no longer blocks
  on XA transactions.
- Re-record two clone-related tests in the component_keyring_file MTR suite.
- Temporarily workaround the MyRocks bug where log_status query hangs if RocksDB
  file deletions were not disabled first.

Note that this change breaks some of the clone-related group replication MTR
tests. As they are not tested in fb-mysql and have been already broken, this
patch makes no effort to fix them.

MyRocks Clone (5/n): fix bug 109922 (SEs do not coordinate clone rollbacks on
instance startup)

If InnoDB clone decides to rollback during its startup, other storage engines
are not aware of this and proceed as usual, resulting in inconsistent instance
between storage engines. Ideally this should be fixed through a 2PC-like design
where all the SEs vote to rollback the clone or to continue during the startup,
but for MyRocks clone a simpler solution suffices: make InnoDB write a special
marker file if it decides to rollback on clone startup.

MyRocks Clone (6/n): on remote clone error, always send COM_ACK to InnoDB too

In the case of multithreaded clone, the waits for the donor threads for the
current clone stage to complete are implemented fully in InnoDB, even though
that, and error handling, should be plugin-level features. With two storage
engines participating in the clone this allows the situation where the other
storage engine client reports an error with COM_ACK to one donor thread, while
some other threads are in InnoDB clone state change waits. Those waits can abort
if InnoDB clone gets an error from the client, but they know nothing about other
SE errors, and will be aborted only by timeouts.

As a minimal fix that does not involve pulling the whole error handling and
state change waiting logic to the plugin layer, make the clone client always
send an error-containing COM_ACK to InnoDB too, whenever it sends one to another
storage engine.

MyRocks Clone (7/n): MyRocks clone implementation

- Cleanup any stale RocksDB checkpoint directories on server shutdown too.

- For MyRocks clone network protocol, define four packet types (clone/common.h):
  - LOCATOR_V1: MyRocks-specific clone locator, consisting of a monotonically
    increasing clone session ID for the donor with an optional applier
    state (an array of file IDs and applied offsets) for restarts, or a zero
    in which case it is a version check empty locator used for handshakes.
  - FILE_NAME_MAP_V1: establishes a relation between a file ID, its name and
    target size.
  - FILE_CHUNK_V1: cloned file chunk data
  - ADD_ESTIMATE_V1: how many bytes should be added to the estimated clone data
    size.
- All packet types share the common package header, which stores the version,
  the payload length without the header, and the packet type.
- (clone/common.h): classes locator, file_metadata, chunk_metadata, &
  estimate_metadata correspond to the four packet types. They are subclassed on
  the donor side to add serialization routines and other donor-specific code and
  on the client side to add deserialization routines and other client-specific
  code. Re. serialization/deserialization, an exception is the locator, whose
  routines are implemented in the common code, because it may be passed both
  - ways.
- (clone/common.h): Introduce a session class that handles the task ID set and
  errors, which is common for both donor and client. For errors, allow saving
  one error, with an optional associated path and errno, that can be set by any
  thread and later reported to the clone plugin.
- (clone/common.cc) Setup MyRocks clone capabilities as HA_CLONE_HYBRID (not
  - applicable but matches InnoDB), HA_CLONE_MULTI_TASK (can clone with multiple
  - threads), & HA_CLONE_RESTART (can restart after a network I/O error).
- (clone/common.cc) Enable O_DIRECT for clone file I/O if, on client if
  rocksdb_use_direct_io_for_flush_and_compaction is enabled, and on donor if
  rocksdb_use_direct_reads is enabled.
- On the donor side (clone/donor.cc), encapsulate RocksDB checkpoints in a
  rdb_checkpoint class. Each checkpoint gets its own ID, which is also a donor
  session ID. The ID is constant for rolling checkpoints.
- Subclass the common file_metadata to donor_file_metadata, chunk_metadata to
  donor_chunk_metadata, estimate_metadata to donor_estimate.
- Implement donor going through a sequence of states: NONE, ROLLING, FINAL,
  FINAL_WITH_LOGS. To support multiple clone donor sessions, implement a manager
  class that looks up the donor instance by locator.
- Implement MyRocks clone donor interface:
  - rocksdb_clone_start: handle the possible start modes:
    - HA_CLONE_MODE_START: create and register a new donor instance, register
      its master task ID.
    - HA_CLONE_MODE_ADD_TASK: lookup the existing donor instance, register a new
      task ID.
    - HA_CLONE_MODE_RESTART: lookup the existing donor instance, wait for the
      old master session to drop it by master task ID reference count, then
      process the locator applier state to reset the donor state.
  - rocksdb_clone_precopy: lookup the donor instance, create a RocksDB snapshot
    if state == NONE, otherwise send files by chunks and roll the checkpoint as
    needed.
  - rocksdb_clone_set_log_stop: add the WAL files to the clone file to copy set.
  - rocksdb_clone_copy: copy the clone files in any donor state.
  - rocksdb_clone_ack: acknowledge errors sent by client.
  - rocksdb_clone_end: for master task ID, stop accepting new worker join
    requests, wait until existing workers quit, and, in the case of restartable
    error, wait for any restarts to happen. Then decrement ref count for the
    master task ID (which will handle both the cases of restarted and not
    restarted) clone. For all other task IDs, simply unregister them.
- On the client side (clone/client.cc), have file_in_progress class to track
  each individual file progress during application, and completed_file for
  fully-applied files.
- Subclass the common file_metadata to client_file_metadata, implement its
  reading from the net buffer. Likewise for client_chunk_metadata subclassing
  chunk_metadata, and client_estimate for estimate_metadata.
- TODO client class.
- Implement MyRocks clone client interface:
  - rocksdb_clone_apply_begin: handle the possible start modes:
    - HA_CLONE_MODE_START: initialize the client instance for a new clone
    - HA_CLONE_MODE_ADD_TASK: register a new task ID for an ongoing clone
    - HA_CLONE_MODE_RESTART: restart the interrupted clone session
    - HA_CLONE_MODE_VERSION: prepare new version locator
  - rocksdb_clone_apply: if passed an error code, save it. Otherwise verify the
    passed buffer and dispatch ADD_ESTIMATE_V1, FILE_NAME_MAP_V1, &
    FILE_CHUNK_V1 packets accordingly.
  - rocksdb_clone_apply_end: for master task, wait for all other tasks to
    complete and finish the clone. For both master and non-master tasks
    unregister their IDs.
- Add rocksdb_sys_vars tests for the new clone-related system variables. They
  are included in the current patch and not the MTR one in order not to break
  the existing rocksdb_sys_vars.all_vars test.
- On instance startup, do clone fixup (clone/common.cc). It consists of:
  - Check whether InnoDB clone rollback marker is present. If yes, force
    rollback of MyRocks clone.
  - If the cloned temporary datadir contains clone-in-progress marker, assume
    half-completed clone and abort.
  - Move the cloned datadirs and WAL dirs in-place. For the datadir, do a
    directory-level rename with a save and later delete of the old datadir. For
    the WAL dir do not try to do parent directory-level renames because the
    mysqld process may not have  permissions to delete that directory. Instead,
    rename all the old files individually, move new files in, delete the old
    files.
  - For all the rename operations, handle them returning EXDEV error (a rename
    would cross a filesystem boundary) by falling back to copy and delete. As we
    expect that it is the cloned RocksDB log files that are most affected by
    this, the total copy I/O  wouldn't be large, and so the default MySQL copy
    routine my_copy is used, without extra copy throughput optimizations. Add a
    fault injection point for MTR to test this code paths.
  - Remove any RocksDB clone checkpoints found.
- On instance startup, always move WALs to the target directory if they are
  found in the rocksdb_datadir yet the rocksdb_wal_dir is different. This
  handles the case of setting rocksdb_wal_dir while instance is shutdown, both
  for clone fixup and regular operation.
- Add PFS instrumentation for all the instrumentable entities in this
  patch (mutexes, rwlocks, file I/O).

MyRocks Clone (8/n): MyRocks clone MTR tests

- Add a new MTR test suite rocksdb_clone
- Support new variable MTR clone/include/clone_command.inc:
  clone_single_threaded, setting which forces single-threaded clone operation.
- Reuse a big part of InnoDB clone tests by including their test files, with the
  exception of tests involving DDL, as concurrent DDL is not supported by
  MyRocks clone at the moment, and debug tests that adjust cloned instance LSN,
  because they desynchronize storage engines. The reused tests are: local_basic,
  local_boundary, local_dml, local_dml_auto_tune, local_dml_no_redo,
  local_insert, local_stage, monitor_progress, remote_basic,
  remote_basic_replace, remote_boundary, remote_dml, remote_dml_auto_tune,
  remote_dml_gtid, remote_dml_no_binlog, remote_dml_no_ssl, remote_dml_recovery,
  remote_dml_replace, remote_dml_ssl, remote_dml_ssl_certificate,
  remote_dml_upgrade, remote_insert, remote_monitor_progress, remote_restart,
  remote_skip_validation, remote_stage, remote_stage_restart.
- As a part of the above, add MyRocks support to local_basic and
  monitor_progress MTR tests.
- Besides the reused InnoDB clone tests, add some new MyRocks-specific tests:
- local_minimal_1_thread, which is the simplest possible local clone test.
- local_minimal_multithreaded, the simplest possible but multithreaded local
  clone test.
- local_minimal_direct_io, which is the simplest possible local clone test in
  all combinations of rocksdb_use_direct_reads and
  rocksdb_use_direct_io_for_flush_and_compaction settings, which are the
  O_DIRECT-enabling options.
- local_dml_o_direct_donor, local_dml_o_direct_client, local_dml_o_direct_both
  that reuse InnoDB local_dml test, but combined with either or both of the
  O_DIRECT-enabling options.
- local_minimal_wal_dir, which clones with various rocksdb_wal_dir settings on
  donor and client.
- local_minimal_wal_dir_copy_delete, which uses a debug injection to test the
  move-across-filesystem-boundary code path in clone startup.
- remote_minimal_direct_io_{1, 2, 3, 4} tests for all the combinations of
  O_DIRECT-enabling options for donor and client instances.
- remote_fail_apply, which uses a debug injection to fail the clone client
  during remote clone.
- remote_inplace_restart, which performs a remote in-place clone with a
  simulated transient error on the client.
- remote_kill_donor, which kill the donor instance at various points of remote
  cloning.
- rolling_checkpoint for testing various combinations of clone rolling
  checkpoint tuning options.
- three_se_basic to test a simple clone across InnoDB, MyRocks, & Blackhole
  engines.

Fix moving RocksDB WALs on startup across file system boundaries

Reviewed By: sunshine-Chun

Differential Revision: D43160162

Pulled By: hermanlee

fbshipit-source-id: 5358a57

--------------------------------------------------------------------------------

Rdb clone fixes (facebook#1271)

Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 2245706

--------------------------------------------------------------------------------

Clone: fix race condition that may skip some SST files (facebook#1274)

Summary:
A donor thread in the main copy loop took a file to send, released the donor state mutex, and then opened the file. If an ENOENT was received at this point, it was assumed that this was a stale SST file from an older checkpoint that was rolled since.

Because the donor state mutex was released between taking of the file and opening it, the following race was possible:
1) Thread 1 takes the file
2) Thread 2 decides to roll the checkpoint, the old checkpoint is deleted 3) Thread 1 tries to open the file, gets ENOENT
4) Thread 2 creates the new checkpoint, the file re-appears, but it's too late.

Rolling the checkpoint in a donor state mutex critical section is a possible fix, but such section would do a lot of I/O, serializing the parallel threads. Instead, fix by taking the file and opening it in the same critical section.

Pull Request resolved: facebook#1274

Reviewed By: sunshine-Chun

Differential Revision: D43629546

Pulled By: hermanlee

fbshipit-source-id: 6e7f315
inikep pushed a commit to inikep/mysql-5.6 that referenced this issue May 17, 2023
Summary: Pull Request resolved: facebook#1250

MyRocks Clone (1/n): make InnoDB clone MTR tests SE-agnostic

The MyRocks clone MTR tests will reuse the InnoDB tests by including them.
Adjust the tests so that they work this way, i.e. make them storage
engine-agnostic by checking the default_storage_engine system variable and
guarding InnoDB-specific tests this way. This includes the tests that
artificially desync the InnoDB redo log position.

MyRocks Clone (2/n): fix bug 109919 (Clone plugin does not support arbitrary
length O_DIRECT files)

The existing clone plugin I/O routines handle O_DIRECT files with the assumption
that their sizes are always a multiple of a block size. This is too restrictive
because in general O_DIRECT files can be of any size and MyRocks files actually are.

Add a flag to Ha_clone_file to support arbitrary file lengths under O_DIRECT:
o_direct_uneven_file_size. Pass it to clone_client.h/cc
Client::get_aligned_buffer to allocate the buffer that bumps the length to the
next CLONE_OS_ALIGN multiple if it's not already one. Introduce a
pad_for_o_direct helper that performs this calculation for the passed length as
needed.

For writes, if O_DIRECT is enabled and the buffer length is not a multiple of
CLONE_OS_ALIGN, pad it with zeros to the next multiple. Then check the actual
bytes written, and if any padding bytes were actually written, call my_chsize to
truncate the file to the needed size.

MyRocks Clone (3/n): fix bug 109920 (SEs cannot add to clone data size estimates)

In the current implementation, once the first SE sets the data size estimate for
a particular stage, there is no way for other SEs to add to it. Address this by
adding Ha_clone_cbk::add_to_data_size_estimate method.

MyRocks Clone (4/n): fix bug 109921 (Cannot clone with synchronization across SEs)

The existing clone implementation synchronizes InnoDB and binlog directly. There
is no way to make other storage engines join this synchronization, thus replace
it with synchronizing by a performance_schema.log_status query.

- Add table_log_status::get_row method to allow doing the log_status query from
  inside the server.
- Handlerton: introduce two new methods: clone_precopy - currently unused, which
  will handle any non-consistent-snapshot copying before the main consistent
  copy, and set_log_stop - which provides the SE the clone stop position
  information.
- Plugin: introduce Ha_clone_common_cbk, which derives from Ha_clone_cbk, and
  from which derive all the concrete classes: Client_Cbk, Local_Cbk &
  Server_Cbk.
- Ha_clone_common_cbk contains two methods: precopy and synchronize_engines. The
  former calls precopy for all the participating clone SE handlertons, if
  implemented. The latter, only to be called from the main clone thread,
  performs the performance_schema.LOG_STATUS query, splits its result per-SE and
  calls handlerton set_log_stop for all SEs.
- The pfs.log_status query still must be called by InnoDB at a certain point in
  clone. That means that InnoDB is still a special SE in that it must always
  come first in the clone locator vectors. This will also handle donor waits on
  clone restarts, as that is implemented in InnoDB-specific code. To ensure
  this, add a helper clone_hton.cc:make_innodb_first which shuffles the vectors
  as needed. Call it in all the clone-starting code paths, add asserts that
  InnoDB comes first.
- Since hton_clone_copy will call into InnoDB clone copy, which at certain point
  will synchronize engines, calling into the plugin again, and the calling into
  SEs again, introduce a backup locator vector Ha_clone_cbk::all_locators, and
  its setters/getters set_all_locators, reset_all_locators, get_all_locators.
  Use them in the re-entrant hton_clone_copy code.
- Introduce a new InnoDB clone state named SST COPY, which calls precopy and has
  zero files and chunks set to copy in the regular InnoDB clone way, i.e. all it
  does it the precopy.
- The redo log archiving keeps running throughout the precopy stage.
- On stage transition from the precopy to the redo log copy. call synchronize.
- Implement innodb_clone_set_log_stop by parsing out the desired clone stop LSN
  from the passed JSON from the log_status query.
- Make InnoDB redo log archiving stop at that LSN instead of on reaching the end
  of redo log.
- Support stopping redo log archiving in a middle of a log block. As the old
  implementation may encounter this case only at the end of the redo log, it
  took that block from the log buffer in memory. This no longer works with the
  new implementation, where the needed block might reside on disk only. Address
  this by forcing the whole block containing the target LSN to disk, reading it
  from there, and patching its memory copy to stop at the required LSN before
  giving it for the clone to copy.
- To implement redo log block reading on a running server, adapt crash recovery
  function recv_read_log_seg to work both for crash recovery and on a running
  server. This is similar to what Percona Server does for its changed page
  tracking feature.
- Make sure not to archive not-yet-written log: pfs.log_status query gets the
  log system LSN, which is the value of highest log reservation. Thus in an
  archiving iteration, archive up to the lower values of log archiving LSN, and
  write_lsn. Set the log archiving LSN to LSN_MAX at clone initialization, so
  that clone archiving may proceed until the stop LSN is set.
- The above implementation results in cloned LSN != log end LSN for the cloned
  instance, thus adjust crash recovery to work with that. The existing crash
  recovery throws away any crash-recovered metadata and reads its snapshot from
  the data files, whereas the snapshot was not written in any consistency with
  the clone LSN. Fix by no longer throwing away the crash-recovered InnoDB
  dynamic metadata and by not re-reading the the last metadata snapshot. This
  forces also not to re-serialize the metadata in the crash recovery as that
  writes redo log which is to be avoided during clone crash recovery.
- Additionally, at the start of the redo log copy the old implementation
  requested all InnoDB updates to be flushed to disk immediately, to avoid clone
  recovery generating new redo log. This is also disabled as it becomes
  redundant with the fix. This issue is similar to Percona XtraBackup
  https://jira.percona.com/browse/PXB-1634
- Disable clone.local_xa test because the new synchronization no longer blocks
  on XA transactions.
- Re-record two clone-related tests in the component_keyring_file MTR suite.
- Temporarily workaround the MyRocks bug where log_status query hangs if RocksDB
  file deletions were not disabled first.

Note that this change breaks some of the clone-related group replication MTR
tests. As they are not tested in fb-mysql and have been already broken, this
patch makes no effort to fix them.

MyRocks Clone (5/n): fix bug 109922 (SEs do not coordinate clone rollbacks on
instance startup)

If InnoDB clone decides to rollback during its startup, other storage engines
are not aware of this and proceed as usual, resulting in inconsistent instance
between storage engines. Ideally this should be fixed through a 2PC-like design
where all the SEs vote to rollback the clone or to continue during the startup,
but for MyRocks clone a simpler solution suffices: make InnoDB write a special
marker file if it decides to rollback on clone startup.

MyRocks Clone (6/n): on remote clone error, always send COM_ACK to InnoDB too

In the case of multithreaded clone, the waits for the donor threads for the
current clone stage to complete are implemented fully in InnoDB, even though
that, and error handling, should be plugin-level features. With two storage
engines participating in the clone this allows the situation where the other
storage engine client reports an error with COM_ACK to one donor thread, while
some other threads are in InnoDB clone state change waits. Those waits can abort
if InnoDB clone gets an error from the client, but they know nothing about other
SE errors, and will be aborted only by timeouts.

As a minimal fix that does not involve pulling the whole error handling and
state change waiting logic to the plugin layer, make the clone client always
send an error-containing COM_ACK to InnoDB too, whenever it sends one to another
storage engine.

MyRocks Clone (7/n): MyRocks clone implementation

- Cleanup any stale RocksDB checkpoint directories on server shutdown too.

- For MyRocks clone network protocol, define four packet types (clone/common.h):
  - LOCATOR_V1: MyRocks-specific clone locator, consisting of a monotonically
    increasing clone session ID for the donor with an optional applier
    state (an array of file IDs and applied offsets) for restarts, or a zero
    in which case it is a version check empty locator used for handshakes.
  - FILE_NAME_MAP_V1: establishes a relation between a file ID, its name and
    target size.
  - FILE_CHUNK_V1: cloned file chunk data
  - ADD_ESTIMATE_V1: how many bytes should be added to the estimated clone data
    size.
- All packet types share the common package header, which stores the version,
  the payload length without the header, and the packet type.
- (clone/common.h): classes locator, file_metadata, chunk_metadata, &
  estimate_metadata correspond to the four packet types. They are subclassed on
  the donor side to add serialization routines and other donor-specific code and
  on the client side to add deserialization routines and other client-specific
  code. Re. serialization/deserialization, an exception is the locator, whose
  routines are implemented in the common code, because it may be passed both
  - ways.
- (clone/common.h): Introduce a session class that handles the task ID set and
  errors, which is common for both donor and client. For errors, allow saving
  one error, with an optional associated path and errno, that can be set by any
  thread and later reported to the clone plugin.
- (clone/common.cc) Setup MyRocks clone capabilities as HA_CLONE_HYBRID (not
  - applicable but matches InnoDB), HA_CLONE_MULTI_TASK (can clone with multiple
  - threads), & HA_CLONE_RESTART (can restart after a network I/O error).
- (clone/common.cc) Enable O_DIRECT for clone file I/O if, on client if
  rocksdb_use_direct_io_for_flush_and_compaction is enabled, and on donor if
  rocksdb_use_direct_reads is enabled.
- On the donor side (clone/donor.cc), encapsulate RocksDB checkpoints in a
  rdb_checkpoint class. Each checkpoint gets its own ID, which is also a donor
  session ID. The ID is constant for rolling checkpoints.
- Subclass the common file_metadata to donor_file_metadata, chunk_metadata to
  donor_chunk_metadata, estimate_metadata to donor_estimate.
- Implement donor going through a sequence of states: NONE, ROLLING, FINAL,
  FINAL_WITH_LOGS. To support multiple clone donor sessions, implement a manager
  class that looks up the donor instance by locator.
- Implement MyRocks clone donor interface:
  - rocksdb_clone_start: handle the possible start modes:
    - HA_CLONE_MODE_START: create and register a new donor instance, register
      its master task ID.
    - HA_CLONE_MODE_ADD_TASK: lookup the existing donor instance, register a new
      task ID.
    - HA_CLONE_MODE_RESTART: lookup the existing donor instance, wait for the
      old master session to drop it by master task ID reference count, then
      process the locator applier state to reset the donor state.
  - rocksdb_clone_precopy: lookup the donor instance, create a RocksDB snapshot
    if state == NONE, otherwise send files by chunks and roll the checkpoint as
    needed.
  - rocksdb_clone_set_log_stop: add the WAL files to the clone file to copy set.
  - rocksdb_clone_copy: copy the clone files in any donor state.
  - rocksdb_clone_ack: acknowledge errors sent by client.
  - rocksdb_clone_end: for master task ID, stop accepting new worker join
    requests, wait until existing workers quit, and, in the case of restartable
    error, wait for any restarts to happen. Then decrement ref count for the
    master task ID (which will handle both the cases of restarted and not
    restarted) clone. For all other task IDs, simply unregister them.
- On the client side (clone/client.cc), have file_in_progress class to track
  each individual file progress during application, and completed_file for
  fully-applied files.
- Subclass the common file_metadata to client_file_metadata, implement its
  reading from the net buffer. Likewise for client_chunk_metadata subclassing
  chunk_metadata, and client_estimate for estimate_metadata.
- TODO client class.
- Implement MyRocks clone client interface:
  - rocksdb_clone_apply_begin: handle the possible start modes:
    - HA_CLONE_MODE_START: initialize the client instance for a new clone
    - HA_CLONE_MODE_ADD_TASK: register a new task ID for an ongoing clone
    - HA_CLONE_MODE_RESTART: restart the interrupted clone session
    - HA_CLONE_MODE_VERSION: prepare new version locator
  - rocksdb_clone_apply: if passed an error code, save it. Otherwise verify the
    passed buffer and dispatch ADD_ESTIMATE_V1, FILE_NAME_MAP_V1, &
    FILE_CHUNK_V1 packets accordingly.
  - rocksdb_clone_apply_end: for master task, wait for all other tasks to
    complete and finish the clone. For both master and non-master tasks
    unregister their IDs.
- Add rocksdb_sys_vars tests for the new clone-related system variables. They
  are included in the current patch and not the MTR one in order not to break
  the existing rocksdb_sys_vars.all_vars test.
- On instance startup, do clone fixup (clone/common.cc). It consists of:
  - Check whether InnoDB clone rollback marker is present. If yes, force
    rollback of MyRocks clone.
  - If the cloned temporary datadir contains clone-in-progress marker, assume
    half-completed clone and abort.
  - Move the cloned datadirs and WAL dirs in-place. For the datadir, do a
    directory-level rename with a save and later delete of the old datadir. For
    the WAL dir do not try to do parent directory-level renames because the
    mysqld process may not have  permissions to delete that directory. Instead,
    rename all the old files individually, move new files in, delete the old
    files.
  - For all the rename operations, handle them returning EXDEV error (a rename
    would cross a filesystem boundary) by falling back to copy and delete. As we
    expect that it is the cloned RocksDB log files that are most affected by
    this, the total copy I/O  wouldn't be large, and so the default MySQL copy
    routine my_copy is used, without extra copy throughput optimizations. Add a
    fault injection point for MTR to test this code paths.
  - Remove any RocksDB clone checkpoints found.
- On instance startup, always move WALs to the target directory if they are
  found in the rocksdb_datadir yet the rocksdb_wal_dir is different. This
  handles the case of setting rocksdb_wal_dir while instance is shutdown, both
  for clone fixup and regular operation.
- Add PFS instrumentation for all the instrumentable entities in this
  patch (mutexes, rwlocks, file I/O).

MyRocks Clone (8/n): MyRocks clone MTR tests

- Add a new MTR test suite rocksdb_clone
- Support new variable MTR clone/include/clone_command.inc:
  clone_single_threaded, setting which forces single-threaded clone operation.
- Reuse a big part of InnoDB clone tests by including their test files, with the
  exception of tests involving DDL, as concurrent DDL is not supported by
  MyRocks clone at the moment, and debug tests that adjust cloned instance LSN,
  because they desynchronize storage engines. The reused tests are: local_basic,
  local_boundary, local_dml, local_dml_auto_tune, local_dml_no_redo,
  local_insert, local_stage, monitor_progress, remote_basic,
  remote_basic_replace, remote_boundary, remote_dml, remote_dml_auto_tune,
  remote_dml_gtid, remote_dml_no_binlog, remote_dml_no_ssl, remote_dml_recovery,
  remote_dml_replace, remote_dml_ssl, remote_dml_ssl_certificate,
  remote_dml_upgrade, remote_insert, remote_monitor_progress, remote_restart,
  remote_skip_validation, remote_stage, remote_stage_restart.
- As a part of the above, add MyRocks support to local_basic and
  monitor_progress MTR tests.
- Besides the reused InnoDB clone tests, add some new MyRocks-specific tests:
- local_minimal_1_thread, which is the simplest possible local clone test.
- local_minimal_multithreaded, the simplest possible but multithreaded local
  clone test.
- local_minimal_direct_io, which is the simplest possible local clone test in
  all combinations of rocksdb_use_direct_reads and
  rocksdb_use_direct_io_for_flush_and_compaction settings, which are the
  O_DIRECT-enabling options.
- local_dml_o_direct_donor, local_dml_o_direct_client, local_dml_o_direct_both
  that reuse InnoDB local_dml test, but combined with either or both of the
  O_DIRECT-enabling options.
- local_minimal_wal_dir, which clones with various rocksdb_wal_dir settings on
  donor and client.
- local_minimal_wal_dir_copy_delete, which uses a debug injection to test the
  move-across-filesystem-boundary code path in clone startup.
- remote_minimal_direct_io_{1, 2, 3, 4} tests for all the combinations of
  O_DIRECT-enabling options for donor and client instances.
- remote_fail_apply, which uses a debug injection to fail the clone client
  during remote clone.
- remote_inplace_restart, which performs a remote in-place clone with a
  simulated transient error on the client.
- remote_kill_donor, which kill the donor instance at various points of remote
  cloning.
- rolling_checkpoint for testing various combinations of clone rolling
  checkpoint tuning options.
- three_se_basic to test a simple clone across InnoDB, MyRocks, & Blackhole
  engines.

Fix moving RocksDB WALs on startup across file system boundaries

Reviewed By: sunshine-Chun

Differential Revision: D43160162

Pulled By: hermanlee

fbshipit-source-id: 5358a57

--------------------------------------------------------------------------------

Rdb clone fixes (facebook#1271)

Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 2245706

--------------------------------------------------------------------------------

Clone: fix race condition that may skip some SST files (facebook#1274)

Summary:
A donor thread in the main copy loop took a file to send, released the donor state mutex, and then opened the file. If an ENOENT was received at this point, it was assumed that this was a stale SST file from an older checkpoint that was rolled since.

Because the donor state mutex was released between taking of the file and opening it, the following race was possible:
1) Thread 1 takes the file
2) Thread 2 decides to roll the checkpoint, the old checkpoint is deleted 3) Thread 1 tries to open the file, gets ENOENT
4) Thread 2 creates the new checkpoint, the file re-appears, but it's too late.

Rolling the checkpoint in a donor state mutex critical section is a possible fix, but such section would do a lot of I/O, serializing the parallel threads. Instead, fix by taking the file and opening it in the same critical section.

Pull Request resolved: facebook#1274

Reviewed By: sunshine-Chun

Differential Revision: D43629546

Pulled By: hermanlee

fbshipit-source-id: 6e7f315
inikep pushed a commit to inikep/mysql-5.6 that referenced this issue May 18, 2023
Summary: Pull Request resolved: facebook#1250

MyRocks Clone (1/n): make InnoDB clone MTR tests SE-agnostic

The MyRocks clone MTR tests will reuse the InnoDB tests by including them.
Adjust the tests so that they work this way, i.e. make them storage
engine-agnostic by checking the default_storage_engine system variable and
guarding InnoDB-specific tests this way. This includes the tests that
artificially desync the InnoDB redo log position.

MyRocks Clone (2/n): fix bug 109919 (Clone plugin does not support arbitrary
length O_DIRECT files)

The existing clone plugin I/O routines handle O_DIRECT files with the assumption
that their sizes are always a multiple of a block size. This is too restrictive
because in general O_DIRECT files can be of any size and MyRocks files actually are.

Add a flag to Ha_clone_file to support arbitrary file lengths under O_DIRECT:
o_direct_uneven_file_size. Pass it to clone_client.h/cc
Client::get_aligned_buffer to allocate the buffer that bumps the length to the
next CLONE_OS_ALIGN multiple if it's not already one. Introduce a
pad_for_o_direct helper that performs this calculation for the passed length as
needed.

For writes, if O_DIRECT is enabled and the buffer length is not a multiple of
CLONE_OS_ALIGN, pad it with zeros to the next multiple. Then check the actual
bytes written, and if any padding bytes were actually written, call my_chsize to
truncate the file to the needed size.

MyRocks Clone (3/n): fix bug 109920 (SEs cannot add to clone data size estimates)

In the current implementation, once the first SE sets the data size estimate for
a particular stage, there is no way for other SEs to add to it. Address this by
adding Ha_clone_cbk::add_to_data_size_estimate method.

MyRocks Clone (4/n): fix bug 109921 (Cannot clone with synchronization across SEs)

The existing clone implementation synchronizes InnoDB and binlog directly. There
is no way to make other storage engines join this synchronization, thus replace
it with synchronizing by a performance_schema.log_status query.

- Add table_log_status::get_row method to allow doing the log_status query from
  inside the server.
- Handlerton: introduce two new methods: clone_precopy - currently unused, which
  will handle any non-consistent-snapshot copying before the main consistent
  copy, and set_log_stop - which provides the SE the clone stop position
  information.
- Plugin: introduce Ha_clone_common_cbk, which derives from Ha_clone_cbk, and
  from which derive all the concrete classes: Client_Cbk, Local_Cbk &
  Server_Cbk.
- Ha_clone_common_cbk contains two methods: precopy and synchronize_engines. The
  former calls precopy for all the participating clone SE handlertons, if
  implemented. The latter, only to be called from the main clone thread,
  performs the performance_schema.LOG_STATUS query, splits its result per-SE and
  calls handlerton set_log_stop for all SEs.
- The pfs.log_status query still must be called by InnoDB at a certain point in
  clone. That means that InnoDB is still a special SE in that it must always
  come first in the clone locator vectors. This will also handle donor waits on
  clone restarts, as that is implemented in InnoDB-specific code. To ensure
  this, add a helper clone_hton.cc:make_innodb_first which shuffles the vectors
  as needed. Call it in all the clone-starting code paths, add asserts that
  InnoDB comes first.
- Since hton_clone_copy will call into InnoDB clone copy, which at certain point
  will synchronize engines, calling into the plugin again, and the calling into
  SEs again, introduce a backup locator vector Ha_clone_cbk::all_locators, and
  its setters/getters set_all_locators, reset_all_locators, get_all_locators.
  Use them in the re-entrant hton_clone_copy code.
- Introduce a new InnoDB clone state named SST COPY, which calls precopy and has
  zero files and chunks set to copy in the regular InnoDB clone way, i.e. all it
  does it the precopy.
- The redo log archiving keeps running throughout the precopy stage.
- On stage transition from the precopy to the redo log copy. call synchronize.
- Implement innodb_clone_set_log_stop by parsing out the desired clone stop LSN
  from the passed JSON from the log_status query.
- Make InnoDB redo log archiving stop at that LSN instead of on reaching the end
  of redo log.
- Support stopping redo log archiving in a middle of a log block. As the old
  implementation may encounter this case only at the end of the redo log, it
  took that block from the log buffer in memory. This no longer works with the
  new implementation, where the needed block might reside on disk only. Address
  this by forcing the whole block containing the target LSN to disk, reading it
  from there, and patching its memory copy to stop at the required LSN before
  giving it for the clone to copy.
- To implement redo log block reading on a running server, adapt crash recovery
  function recv_read_log_seg to work both for crash recovery and on a running
  server. This is similar to what Percona Server does for its changed page
  tracking feature.
- Make sure not to archive not-yet-written log: pfs.log_status query gets the
  log system LSN, which is the value of highest log reservation. Thus in an
  archiving iteration, archive up to the lower values of log archiving LSN, and
  write_lsn. Set the log archiving LSN to LSN_MAX at clone initialization, so
  that clone archiving may proceed until the stop LSN is set.
- The above implementation results in cloned LSN != log end LSN for the cloned
  instance, thus adjust crash recovery to work with that. The existing crash
  recovery throws away any crash-recovered metadata and reads its snapshot from
  the data files, whereas the snapshot was not written in any consistency with
  the clone LSN. Fix by no longer throwing away the crash-recovered InnoDB
  dynamic metadata and by not re-reading the the last metadata snapshot. This
  forces also not to re-serialize the metadata in the crash recovery as that
  writes redo log which is to be avoided during clone crash recovery.
- Additionally, at the start of the redo log copy the old implementation
  requested all InnoDB updates to be flushed to disk immediately, to avoid clone
  recovery generating new redo log. This is also disabled as it becomes
  redundant with the fix. This issue is similar to Percona XtraBackup
  https://jira.percona.com/browse/PXB-1634
- Disable clone.local_xa test because the new synchronization no longer blocks
  on XA transactions.
- Re-record two clone-related tests in the component_keyring_file MTR suite.
- Temporarily workaround the MyRocks bug where log_status query hangs if RocksDB
  file deletions were not disabled first.

Note that this change breaks some of the clone-related group replication MTR
tests. As they are not tested in fb-mysql and have been already broken, this
patch makes no effort to fix them.

MyRocks Clone (5/n): fix bug 109922 (SEs do not coordinate clone rollbacks on
instance startup)

If InnoDB clone decides to rollback during its startup, other storage engines
are not aware of this and proceed as usual, resulting in inconsistent instance
between storage engines. Ideally this should be fixed through a 2PC-like design
where all the SEs vote to rollback the clone or to continue during the startup,
but for MyRocks clone a simpler solution suffices: make InnoDB write a special
marker file if it decides to rollback on clone startup.

MyRocks Clone (6/n): on remote clone error, always send COM_ACK to InnoDB too

In the case of multithreaded clone, the waits for the donor threads for the
current clone stage to complete are implemented fully in InnoDB, even though
that, and error handling, should be plugin-level features. With two storage
engines participating in the clone this allows the situation where the other
storage engine client reports an error with COM_ACK to one donor thread, while
some other threads are in InnoDB clone state change waits. Those waits can abort
if InnoDB clone gets an error from the client, but they know nothing about other
SE errors, and will be aborted only by timeouts.

As a minimal fix that does not involve pulling the whole error handling and
state change waiting logic to the plugin layer, make the clone client always
send an error-containing COM_ACK to InnoDB too, whenever it sends one to another
storage engine.

MyRocks Clone (7/n): MyRocks clone implementation

- Cleanup any stale RocksDB checkpoint directories on server shutdown too.

- For MyRocks clone network protocol, define four packet types (clone/common.h):
  - LOCATOR_V1: MyRocks-specific clone locator, consisting of a monotonically
    increasing clone session ID for the donor with an optional applier
    state (an array of file IDs and applied offsets) for restarts, or a zero
    in which case it is a version check empty locator used for handshakes.
  - FILE_NAME_MAP_V1: establishes a relation between a file ID, its name and
    target size.
  - FILE_CHUNK_V1: cloned file chunk data
  - ADD_ESTIMATE_V1: how many bytes should be added to the estimated clone data
    size.
- All packet types share the common package header, which stores the version,
  the payload length without the header, and the packet type.
- (clone/common.h): classes locator, file_metadata, chunk_metadata, &
  estimate_metadata correspond to the four packet types. They are subclassed on
  the donor side to add serialization routines and other donor-specific code and
  on the client side to add deserialization routines and other client-specific
  code. Re. serialization/deserialization, an exception is the locator, whose
  routines are implemented in the common code, because it may be passed both
  - ways.
- (clone/common.h): Introduce a session class that handles the task ID set and
  errors, which is common for both donor and client. For errors, allow saving
  one error, with an optional associated path and errno, that can be set by any
  thread and later reported to the clone plugin.
- (clone/common.cc) Setup MyRocks clone capabilities as HA_CLONE_HYBRID (not
  - applicable but matches InnoDB), HA_CLONE_MULTI_TASK (can clone with multiple
  - threads), & HA_CLONE_RESTART (can restart after a network I/O error).
- (clone/common.cc) Enable O_DIRECT for clone file I/O if, on client if
  rocksdb_use_direct_io_for_flush_and_compaction is enabled, and on donor if
  rocksdb_use_direct_reads is enabled.
- On the donor side (clone/donor.cc), encapsulate RocksDB checkpoints in a
  rdb_checkpoint class. Each checkpoint gets its own ID, which is also a donor
  session ID. The ID is constant for rolling checkpoints.
- Subclass the common file_metadata to donor_file_metadata, chunk_metadata to
  donor_chunk_metadata, estimate_metadata to donor_estimate.
- Implement donor going through a sequence of states: NONE, ROLLING, FINAL,
  FINAL_WITH_LOGS. To support multiple clone donor sessions, implement a manager
  class that looks up the donor instance by locator.
- Implement MyRocks clone donor interface:
  - rocksdb_clone_start: handle the possible start modes:
    - HA_CLONE_MODE_START: create and register a new donor instance, register
      its master task ID.
    - HA_CLONE_MODE_ADD_TASK: lookup the existing donor instance, register a new
      task ID.
    - HA_CLONE_MODE_RESTART: lookup the existing donor instance, wait for the
      old master session to drop it by master task ID reference count, then
      process the locator applier state to reset the donor state.
  - rocksdb_clone_precopy: lookup the donor instance, create a RocksDB snapshot
    if state == NONE, otherwise send files by chunks and roll the checkpoint as
    needed.
  - rocksdb_clone_set_log_stop: add the WAL files to the clone file to copy set.
  - rocksdb_clone_copy: copy the clone files in any donor state.
  - rocksdb_clone_ack: acknowledge errors sent by client.
  - rocksdb_clone_end: for master task ID, stop accepting new worker join
    requests, wait until existing workers quit, and, in the case of restartable
    error, wait for any restarts to happen. Then decrement ref count for the
    master task ID (which will handle both the cases of restarted and not
    restarted) clone. For all other task IDs, simply unregister them.
- On the client side (clone/client.cc), have file_in_progress class to track
  each individual file progress during application, and completed_file for
  fully-applied files.
- Subclass the common file_metadata to client_file_metadata, implement its
  reading from the net buffer. Likewise for client_chunk_metadata subclassing
  chunk_metadata, and client_estimate for estimate_metadata.
- TODO client class.
- Implement MyRocks clone client interface:
  - rocksdb_clone_apply_begin: handle the possible start modes:
    - HA_CLONE_MODE_START: initialize the client instance for a new clone
    - HA_CLONE_MODE_ADD_TASK: register a new task ID for an ongoing clone
    - HA_CLONE_MODE_RESTART: restart the interrupted clone session
    - HA_CLONE_MODE_VERSION: prepare new version locator
  - rocksdb_clone_apply: if passed an error code, save it. Otherwise verify the
    passed buffer and dispatch ADD_ESTIMATE_V1, FILE_NAME_MAP_V1, &
    FILE_CHUNK_V1 packets accordingly.
  - rocksdb_clone_apply_end: for master task, wait for all other tasks to
    complete and finish the clone. For both master and non-master tasks
    unregister their IDs.
- Add rocksdb_sys_vars tests for the new clone-related system variables. They
  are included in the current patch and not the MTR one in order not to break
  the existing rocksdb_sys_vars.all_vars test.
- On instance startup, do clone fixup (clone/common.cc). It consists of:
  - Check whether InnoDB clone rollback marker is present. If yes, force
    rollback of MyRocks clone.
  - If the cloned temporary datadir contains clone-in-progress marker, assume
    half-completed clone and abort.
  - Move the cloned datadirs and WAL dirs in-place. For the datadir, do a
    directory-level rename with a save and later delete of the old datadir. For
    the WAL dir do not try to do parent directory-level renames because the
    mysqld process may not have  permissions to delete that directory. Instead,
    rename all the old files individually, move new files in, delete the old
    files.
  - For all the rename operations, handle them returning EXDEV error (a rename
    would cross a filesystem boundary) by falling back to copy and delete. As we
    expect that it is the cloned RocksDB log files that are most affected by
    this, the total copy I/O  wouldn't be large, and so the default MySQL copy
    routine my_copy is used, without extra copy throughput optimizations. Add a
    fault injection point for MTR to test this code paths.
  - Remove any RocksDB clone checkpoints found.
- On instance startup, always move WALs to the target directory if they are
  found in the rocksdb_datadir yet the rocksdb_wal_dir is different. This
  handles the case of setting rocksdb_wal_dir while instance is shutdown, both
  for clone fixup and regular operation.
- Add PFS instrumentation for all the instrumentable entities in this
  patch (mutexes, rwlocks, file I/O).

MyRocks Clone (8/n): MyRocks clone MTR tests

- Add a new MTR test suite rocksdb_clone
- Support new variable MTR clone/include/clone_command.inc:
  clone_single_threaded, setting which forces single-threaded clone operation.
- Reuse a big part of InnoDB clone tests by including their test files, with the
  exception of tests involving DDL, as concurrent DDL is not supported by
  MyRocks clone at the moment, and debug tests that adjust cloned instance LSN,
  because they desynchronize storage engines. The reused tests are: local_basic,
  local_boundary, local_dml, local_dml_auto_tune, local_dml_no_redo,
  local_insert, local_stage, monitor_progress, remote_basic,
  remote_basic_replace, remote_boundary, remote_dml, remote_dml_auto_tune,
  remote_dml_gtid, remote_dml_no_binlog, remote_dml_no_ssl, remote_dml_recovery,
  remote_dml_replace, remote_dml_ssl, remote_dml_ssl_certificate,
  remote_dml_upgrade, remote_insert, remote_monitor_progress, remote_restart,
  remote_skip_validation, remote_stage, remote_stage_restart.
- As a part of the above, add MyRocks support to local_basic and
  monitor_progress MTR tests.
- Besides the reused InnoDB clone tests, add some new MyRocks-specific tests:
- local_minimal_1_thread, which is the simplest possible local clone test.
- local_minimal_multithreaded, the simplest possible but multithreaded local
  clone test.
- local_minimal_direct_io, which is the simplest possible local clone test in
  all combinations of rocksdb_use_direct_reads and
  rocksdb_use_direct_io_for_flush_and_compaction settings, which are the
  O_DIRECT-enabling options.
- local_dml_o_direct_donor, local_dml_o_direct_client, local_dml_o_direct_both
  that reuse InnoDB local_dml test, but combined with either or both of the
  O_DIRECT-enabling options.
- local_minimal_wal_dir, which clones with various rocksdb_wal_dir settings on
  donor and client.
- local_minimal_wal_dir_copy_delete, which uses a debug injection to test the
  move-across-filesystem-boundary code path in clone startup.
- remote_minimal_direct_io_{1, 2, 3, 4} tests for all the combinations of
  O_DIRECT-enabling options for donor and client instances.
- remote_fail_apply, which uses a debug injection to fail the clone client
  during remote clone.
- remote_inplace_restart, which performs a remote in-place clone with a
  simulated transient error on the client.
- remote_kill_donor, which kill the donor instance at various points of remote
  cloning.
- rolling_checkpoint for testing various combinations of clone rolling
  checkpoint tuning options.
- three_se_basic to test a simple clone across InnoDB, MyRocks, & Blackhole
  engines.

Fix moving RocksDB WALs on startup across file system boundaries

Reviewed By: sunshine-Chun

Differential Revision: D43160162

Pulled By: hermanlee

fbshipit-source-id: 5358a57

--------------------------------------------------------------------------------

Rdb clone fixes (facebook#1271)

Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 2245706

--------------------------------------------------------------------------------

Clone: fix race condition that may skip some SST files (facebook#1274)

Summary:
A donor thread in the main copy loop took a file to send, released the donor state mutex, and then opened the file. If an ENOENT was received at this point, it was assumed that this was a stale SST file from an older checkpoint that was rolled since.

Because the donor state mutex was released between taking of the file and opening it, the following race was possible:
1) Thread 1 takes the file
2) Thread 2 decides to roll the checkpoint, the old checkpoint is deleted 3) Thread 1 tries to open the file, gets ENOENT
4) Thread 2 creates the new checkpoint, the file re-appears, but it's too late.

Rolling the checkpoint in a donor state mutex critical section is a possible fix, but such section would do a lot of I/O, serializing the parallel threads. Instead, fix by taking the file and opening it in the same critical section.

Pull Request resolved: facebook#1274

Reviewed By: sunshine-Chun

Differential Revision: D43629546

Pulled By: hermanlee

fbshipit-source-id: 6e7f315
inikep pushed a commit to inikep/mysql-5.6 that referenced this issue May 26, 2023
Summary: Pull Request resolved: facebook#1250

MyRocks Clone (1/n): make InnoDB clone MTR tests SE-agnostic

The MyRocks clone MTR tests will reuse the InnoDB tests by including them.
Adjust the tests so that they work this way, i.e. make them storage
engine-agnostic by checking the default_storage_engine system variable and
guarding InnoDB-specific tests this way. This includes the tests that
artificially desync the InnoDB redo log position.

MyRocks Clone (2/n): fix bug 109919 (Clone plugin does not support arbitrary
length O_DIRECT files)

The existing clone plugin I/O routines handle O_DIRECT files with the assumption
that their sizes are always a multiple of a block size. This is too restrictive
because in general O_DIRECT files can be of any size and MyRocks files actually are.

Add a flag to Ha_clone_file to support arbitrary file lengths under O_DIRECT:
o_direct_uneven_file_size. Pass it to clone_client.h/cc
Client::get_aligned_buffer to allocate the buffer that bumps the length to the
next CLONE_OS_ALIGN multiple if it's not already one. Introduce a
pad_for_o_direct helper that performs this calculation for the passed length as
needed.

For writes, if O_DIRECT is enabled and the buffer length is not a multiple of
CLONE_OS_ALIGN, pad it with zeros to the next multiple. Then check the actual
bytes written, and if any padding bytes were actually written, call my_chsize to
truncate the file to the needed size.

MyRocks Clone (3/n): fix bug 109920 (SEs cannot add to clone data size estimates)

In the current implementation, once the first SE sets the data size estimate for
a particular stage, there is no way for other SEs to add to it. Address this by
adding Ha_clone_cbk::add_to_data_size_estimate method.

MyRocks Clone (4/n): fix bug 109921 (Cannot clone with synchronization across SEs)

The existing clone implementation synchronizes InnoDB and binlog directly. There
is no way to make other storage engines join this synchronization, thus replace
it with synchronizing by a performance_schema.log_status query.

- Add table_log_status::get_row method to allow doing the log_status query from
  inside the server.
- Handlerton: introduce two new methods: clone_precopy - currently unused, which
  will handle any non-consistent-snapshot copying before the main consistent
  copy, and set_log_stop - which provides the SE the clone stop position
  information.
- Plugin: introduce Ha_clone_common_cbk, which derives from Ha_clone_cbk, and
  from which derive all the concrete classes: Client_Cbk, Local_Cbk &
  Server_Cbk.
- Ha_clone_common_cbk contains two methods: precopy and synchronize_engines. The
  former calls precopy for all the participating clone SE handlertons, if
  implemented. The latter, only to be called from the main clone thread,
  performs the performance_schema.LOG_STATUS query, splits its result per-SE and
  calls handlerton set_log_stop for all SEs.
- The pfs.log_status query still must be called by InnoDB at a certain point in
  clone. That means that InnoDB is still a special SE in that it must always
  come first in the clone locator vectors. This will also handle donor waits on
  clone restarts, as that is implemented in InnoDB-specific code. To ensure
  this, add a helper clone_hton.cc:make_innodb_first which shuffles the vectors
  as needed. Call it in all the clone-starting code paths, add asserts that
  InnoDB comes first.
- Since hton_clone_copy will call into InnoDB clone copy, which at certain point
  will synchronize engines, calling into the plugin again, and the calling into
  SEs again, introduce a backup locator vector Ha_clone_cbk::all_locators, and
  its setters/getters set_all_locators, reset_all_locators, get_all_locators.
  Use them in the re-entrant hton_clone_copy code.
- Introduce a new InnoDB clone state named SST COPY, which calls precopy and has
  zero files and chunks set to copy in the regular InnoDB clone way, i.e. all it
  does it the precopy.
- The redo log archiving keeps running throughout the precopy stage.
- On stage transition from the precopy to the redo log copy. call synchronize.
- Implement innodb_clone_set_log_stop by parsing out the desired clone stop LSN
  from the passed JSON from the log_status query.
- Make InnoDB redo log archiving stop at that LSN instead of on reaching the end
  of redo log.
- Support stopping redo log archiving in a middle of a log block. As the old
  implementation may encounter this case only at the end of the redo log, it
  took that block from the log buffer in memory. This no longer works with the
  new implementation, where the needed block might reside on disk only. Address
  this by forcing the whole block containing the target LSN to disk, reading it
  from there, and patching its memory copy to stop at the required LSN before
  giving it for the clone to copy.
- To implement redo log block reading on a running server, adapt crash recovery
  function recv_read_log_seg to work both for crash recovery and on a running
  server. This is similar to what Percona Server does for its changed page
  tracking feature.
- Make sure not to archive not-yet-written log: pfs.log_status query gets the
  log system LSN, which is the value of highest log reservation. Thus in an
  archiving iteration, archive up to the lower values of log archiving LSN, and
  write_lsn. Set the log archiving LSN to LSN_MAX at clone initialization, so
  that clone archiving may proceed until the stop LSN is set.
- The above implementation results in cloned LSN != log end LSN for the cloned
  instance, thus adjust crash recovery to work with that. The existing crash
  recovery throws away any crash-recovered metadata and reads its snapshot from
  the data files, whereas the snapshot was not written in any consistency with
  the clone LSN. Fix by no longer throwing away the crash-recovered InnoDB
  dynamic metadata and by not re-reading the the last metadata snapshot. This
  forces also not to re-serialize the metadata in the crash recovery as that
  writes redo log which is to be avoided during clone crash recovery.
- Additionally, at the start of the redo log copy the old implementation
  requested all InnoDB updates to be flushed to disk immediately, to avoid clone
  recovery generating new redo log. This is also disabled as it becomes
  redundant with the fix. This issue is similar to Percona XtraBackup
  https://jira.percona.com/browse/PXB-1634
- Disable clone.local_xa test because the new synchronization no longer blocks
  on XA transactions.
- Re-record two clone-related tests in the component_keyring_file MTR suite.
- Temporarily workaround the MyRocks bug where log_status query hangs if RocksDB
  file deletions were not disabled first.

Note that this change breaks some of the clone-related group replication MTR
tests. As they are not tested in fb-mysql and have been already broken, this
patch makes no effort to fix them.

MyRocks Clone (5/n): fix bug 109922 (SEs do not coordinate clone rollbacks on
instance startup)

If InnoDB clone decides to rollback during its startup, other storage engines
are not aware of this and proceed as usual, resulting in inconsistent instance
between storage engines. Ideally this should be fixed through a 2PC-like design
where all the SEs vote to rollback the clone or to continue during the startup,
but for MyRocks clone a simpler solution suffices: make InnoDB write a special
marker file if it decides to rollback on clone startup.

MyRocks Clone (6/n): on remote clone error, always send COM_ACK to InnoDB too

In the case of multithreaded clone, the waits for the donor threads for the
current clone stage to complete are implemented fully in InnoDB, even though
that, and error handling, should be plugin-level features. With two storage
engines participating in the clone this allows the situation where the other
storage engine client reports an error with COM_ACK to one donor thread, while
some other threads are in InnoDB clone state change waits. Those waits can abort
if InnoDB clone gets an error from the client, but they know nothing about other
SE errors, and will be aborted only by timeouts.

As a minimal fix that does not involve pulling the whole error handling and
state change waiting logic to the plugin layer, make the clone client always
send an error-containing COM_ACK to InnoDB too, whenever it sends one to another
storage engine.

MyRocks Clone (7/n): MyRocks clone implementation

- Cleanup any stale RocksDB checkpoint directories on server shutdown too.

- For MyRocks clone network protocol, define four packet types (clone/common.h):
  - LOCATOR_V1: MyRocks-specific clone locator, consisting of a monotonically
    increasing clone session ID for the donor with an optional applier
    state (an array of file IDs and applied offsets) for restarts, or a zero
    in which case it is a version check empty locator used for handshakes.
  - FILE_NAME_MAP_V1: establishes a relation between a file ID, its name and
    target size.
  - FILE_CHUNK_V1: cloned file chunk data
  - ADD_ESTIMATE_V1: how many bytes should be added to the estimated clone data
    size.
- All packet types share the common package header, which stores the version,
  the payload length without the header, and the packet type.
- (clone/common.h): classes locator, file_metadata, chunk_metadata, &
  estimate_metadata correspond to the four packet types. They are subclassed on
  the donor side to add serialization routines and other donor-specific code and
  on the client side to add deserialization routines and other client-specific
  code. Re. serialization/deserialization, an exception is the locator, whose
  routines are implemented in the common code, because it may be passed both
  - ways.
- (clone/common.h): Introduce a session class that handles the task ID set and
  errors, which is common for both donor and client. For errors, allow saving
  one error, with an optional associated path and errno, that can be set by any
  thread and later reported to the clone plugin.
- (clone/common.cc) Setup MyRocks clone capabilities as HA_CLONE_HYBRID (not
  - applicable but matches InnoDB), HA_CLONE_MULTI_TASK (can clone with multiple
  - threads), & HA_CLONE_RESTART (can restart after a network I/O error).
- (clone/common.cc) Enable O_DIRECT for clone file I/O if, on client if
  rocksdb_use_direct_io_for_flush_and_compaction is enabled, and on donor if
  rocksdb_use_direct_reads is enabled.
- On the donor side (clone/donor.cc), encapsulate RocksDB checkpoints in a
  rdb_checkpoint class. Each checkpoint gets its own ID, which is also a donor
  session ID. The ID is constant for rolling checkpoints.
- Subclass the common file_metadata to donor_file_metadata, chunk_metadata to
  donor_chunk_metadata, estimate_metadata to donor_estimate.
- Implement donor going through a sequence of states: NONE, ROLLING, FINAL,
  FINAL_WITH_LOGS. To support multiple clone donor sessions, implement a manager
  class that looks up the donor instance by locator.
- Implement MyRocks clone donor interface:
  - rocksdb_clone_start: handle the possible start modes:
    - HA_CLONE_MODE_START: create and register a new donor instance, register
      its master task ID.
    - HA_CLONE_MODE_ADD_TASK: lookup the existing donor instance, register a new
      task ID.
    - HA_CLONE_MODE_RESTART: lookup the existing donor instance, wait for the
      old master session to drop it by master task ID reference count, then
      process the locator applier state to reset the donor state.
  - rocksdb_clone_precopy: lookup the donor instance, create a RocksDB snapshot
    if state == NONE, otherwise send files by chunks and roll the checkpoint as
    needed.
  - rocksdb_clone_set_log_stop: add the WAL files to the clone file to copy set.
  - rocksdb_clone_copy: copy the clone files in any donor state.
  - rocksdb_clone_ack: acknowledge errors sent by client.
  - rocksdb_clone_end: for master task ID, stop accepting new worker join
    requests, wait until existing workers quit, and, in the case of restartable
    error, wait for any restarts to happen. Then decrement ref count for the
    master task ID (which will handle both the cases of restarted and not
    restarted) clone. For all other task IDs, simply unregister them.
- On the client side (clone/client.cc), have file_in_progress class to track
  each individual file progress during application, and completed_file for
  fully-applied files.
- Subclass the common file_metadata to client_file_metadata, implement its
  reading from the net buffer. Likewise for client_chunk_metadata subclassing
  chunk_metadata, and client_estimate for estimate_metadata.
- TODO client class.
- Implement MyRocks clone client interface:
  - rocksdb_clone_apply_begin: handle the possible start modes:
    - HA_CLONE_MODE_START: initialize the client instance for a new clone
    - HA_CLONE_MODE_ADD_TASK: register a new task ID for an ongoing clone
    - HA_CLONE_MODE_RESTART: restart the interrupted clone session
    - HA_CLONE_MODE_VERSION: prepare new version locator
  - rocksdb_clone_apply: if passed an error code, save it. Otherwise verify the
    passed buffer and dispatch ADD_ESTIMATE_V1, FILE_NAME_MAP_V1, &
    FILE_CHUNK_V1 packets accordingly.
  - rocksdb_clone_apply_end: for master task, wait for all other tasks to
    complete and finish the clone. For both master and non-master tasks
    unregister their IDs.
- Add rocksdb_sys_vars tests for the new clone-related system variables. They
  are included in the current patch and not the MTR one in order not to break
  the existing rocksdb_sys_vars.all_vars test.
- On instance startup, do clone fixup (clone/common.cc). It consists of:
  - Check whether InnoDB clone rollback marker is present. If yes, force
    rollback of MyRocks clone.
  - If the cloned temporary datadir contains clone-in-progress marker, assume
    half-completed clone and abort.
  - Move the cloned datadirs and WAL dirs in-place. For the datadir, do a
    directory-level rename with a save and later delete of the old datadir. For
    the WAL dir do not try to do parent directory-level renames because the
    mysqld process may not have  permissions to delete that directory. Instead,
    rename all the old files individually, move new files in, delete the old
    files.
  - For all the rename operations, handle them returning EXDEV error (a rename
    would cross a filesystem boundary) by falling back to copy and delete. As we
    expect that it is the cloned RocksDB log files that are most affected by
    this, the total copy I/O  wouldn't be large, and so the default MySQL copy
    routine my_copy is used, without extra copy throughput optimizations. Add a
    fault injection point for MTR to test this code paths.
  - Remove any RocksDB clone checkpoints found.
- On instance startup, always move WALs to the target directory if they are
  found in the rocksdb_datadir yet the rocksdb_wal_dir is different. This
  handles the case of setting rocksdb_wal_dir while instance is shutdown, both
  for clone fixup and regular operation.
- Add PFS instrumentation for all the instrumentable entities in this
  patch (mutexes, rwlocks, file I/O).

MyRocks Clone (8/n): MyRocks clone MTR tests

- Add a new MTR test suite rocksdb_clone
- Support new variable MTR clone/include/clone_command.inc:
  clone_single_threaded, setting which forces single-threaded clone operation.
- Reuse a big part of InnoDB clone tests by including their test files, with the
  exception of tests involving DDL, as concurrent DDL is not supported by
  MyRocks clone at the moment, and debug tests that adjust cloned instance LSN,
  because they desynchronize storage engines. The reused tests are: local_basic,
  local_boundary, local_dml, local_dml_auto_tune, local_dml_no_redo,
  local_insert, local_stage, monitor_progress, remote_basic,
  remote_basic_replace, remote_boundary, remote_dml, remote_dml_auto_tune,
  remote_dml_gtid, remote_dml_no_binlog, remote_dml_no_ssl, remote_dml_recovery,
  remote_dml_replace, remote_dml_ssl, remote_dml_ssl_certificate,
  remote_dml_upgrade, remote_insert, remote_monitor_progress, remote_restart,
  remote_skip_validation, remote_stage, remote_stage_restart.
- As a part of the above, add MyRocks support to local_basic and
  monitor_progress MTR tests.
- Besides the reused InnoDB clone tests, add some new MyRocks-specific tests:
- local_minimal_1_thread, which is the simplest possible local clone test.
- local_minimal_multithreaded, the simplest possible but multithreaded local
  clone test.
- local_minimal_direct_io, which is the simplest possible local clone test in
  all combinations of rocksdb_use_direct_reads and
  rocksdb_use_direct_io_for_flush_and_compaction settings, which are the
  O_DIRECT-enabling options.
- local_dml_o_direct_donor, local_dml_o_direct_client, local_dml_o_direct_both
  that reuse InnoDB local_dml test, but combined with either or both of the
  O_DIRECT-enabling options.
- local_minimal_wal_dir, which clones with various rocksdb_wal_dir settings on
  donor and client.
- local_minimal_wal_dir_copy_delete, which uses a debug injection to test the
  move-across-filesystem-boundary code path in clone startup.
- remote_minimal_direct_io_{1, 2, 3, 4} tests for all the combinations of
  O_DIRECT-enabling options for donor and client instances.
- remote_fail_apply, which uses a debug injection to fail the clone client
  during remote clone.
- remote_inplace_restart, which performs a remote in-place clone with a
  simulated transient error on the client.
- remote_kill_donor, which kill the donor instance at various points of remote
  cloning.
- rolling_checkpoint for testing various combinations of clone rolling
  checkpoint tuning options.
- three_se_basic to test a simple clone across InnoDB, MyRocks, & Blackhole
  engines.

Fix moving RocksDB WALs on startup across file system boundaries

Reviewed By: sunshine-Chun

Differential Revision: D43160162

Pulled By: hermanlee

fbshipit-source-id: 5358a57

--------------------------------------------------------------------------------

Rdb clone fixes (facebook#1271)

Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 2245706

--------------------------------------------------------------------------------

Clone: fix race condition that may skip some SST files (facebook#1274)

Summary:
A donor thread in the main copy loop took a file to send, released the donor state mutex, and then opened the file. If an ENOENT was received at this point, it was assumed that this was a stale SST file from an older checkpoint that was rolled since.

Because the donor state mutex was released between taking of the file and opening it, the following race was possible:
1) Thread 1 takes the file
2) Thread 2 decides to roll the checkpoint, the old checkpoint is deleted 3) Thread 1 tries to open the file, gets ENOENT
4) Thread 2 creates the new checkpoint, the file re-appears, but it's too late.

Rolling the checkpoint in a donor state mutex critical section is a possible fix, but such section would do a lot of I/O, serializing the parallel threads. Instead, fix by taking the file and opening it in the same critical section.

Pull Request resolved: facebook#1274

Reviewed By: sunshine-Chun

Differential Revision: D43629546

Pulled By: hermanlee

fbshipit-source-id: 6e7f315
inikep pushed a commit to inikep/mysql-5.6 that referenced this issue Jun 1, 2023
Summary: Pull Request resolved: facebook#1250

MyRocks Clone (1/n): make InnoDB clone MTR tests SE-agnostic

The MyRocks clone MTR tests will reuse the InnoDB tests by including them.
Adjust the tests so that they work this way, i.e. make them storage
engine-agnostic by checking the default_storage_engine system variable and
guarding InnoDB-specific tests this way. This includes the tests that
artificially desync the InnoDB redo log position.

MyRocks Clone (2/n): fix bug 109919 (Clone plugin does not support arbitrary
length O_DIRECT files)

The existing clone plugin I/O routines handle O_DIRECT files with the assumption
that their sizes are always a multiple of a block size. This is too restrictive
because in general O_DIRECT files can be of any size and MyRocks files actually are.

Add a flag to Ha_clone_file to support arbitrary file lengths under O_DIRECT:
o_direct_uneven_file_size. Pass it to clone_client.h/cc
Client::get_aligned_buffer to allocate the buffer that bumps the length to the
next CLONE_OS_ALIGN multiple if it's not already one. Introduce a
pad_for_o_direct helper that performs this calculation for the passed length as
needed.

For writes, if O_DIRECT is enabled and the buffer length is not a multiple of
CLONE_OS_ALIGN, pad it with zeros to the next multiple. Then check the actual
bytes written, and if any padding bytes were actually written, call my_chsize to
truncate the file to the needed size.

MyRocks Clone (3/n): fix bug 109920 (SEs cannot add to clone data size estimates)

In the current implementation, once the first SE sets the data size estimate for
a particular stage, there is no way for other SEs to add to it. Address this by
adding Ha_clone_cbk::add_to_data_size_estimate method.

MyRocks Clone (4/n): fix bug 109921 (Cannot clone with synchronization across SEs)

The existing clone implementation synchronizes InnoDB and binlog directly. There
is no way to make other storage engines join this synchronization, thus replace
it with synchronizing by a performance_schema.log_status query.

- Add table_log_status::get_row method to allow doing the log_status query from
  inside the server.
- Handlerton: introduce two new methods: clone_precopy - currently unused, which
  will handle any non-consistent-snapshot copying before the main consistent
  copy, and set_log_stop - which provides the SE the clone stop position
  information.
- Plugin: introduce Ha_clone_common_cbk, which derives from Ha_clone_cbk, and
  from which derive all the concrete classes: Client_Cbk, Local_Cbk &
  Server_Cbk.
- Ha_clone_common_cbk contains two methods: precopy and synchronize_engines. The
  former calls precopy for all the participating clone SE handlertons, if
  implemented. The latter, only to be called from the main clone thread,
  performs the performance_schema.LOG_STATUS query, splits its result per-SE and
  calls handlerton set_log_stop for all SEs.
- The pfs.log_status query still must be called by InnoDB at a certain point in
  clone. That means that InnoDB is still a special SE in that it must always
  come first in the clone locator vectors. This will also handle donor waits on
  clone restarts, as that is implemented in InnoDB-specific code. To ensure
  this, add a helper clone_hton.cc:make_innodb_first which shuffles the vectors
  as needed. Call it in all the clone-starting code paths, add asserts that
  InnoDB comes first.
- Since hton_clone_copy will call into InnoDB clone copy, which at certain point
  will synchronize engines, calling into the plugin again, and the calling into
  SEs again, introduce a backup locator vector Ha_clone_cbk::all_locators, and
  its setters/getters set_all_locators, reset_all_locators, get_all_locators.
  Use them in the re-entrant hton_clone_copy code.
- Introduce a new InnoDB clone state named SST COPY, which calls precopy and has
  zero files and chunks set to copy in the regular InnoDB clone way, i.e. all it
  does it the precopy.
- The redo log archiving keeps running throughout the precopy stage.
- On stage transition from the precopy to the redo log copy. call synchronize.
- Implement innodb_clone_set_log_stop by parsing out the desired clone stop LSN
  from the passed JSON from the log_status query.
- Make InnoDB redo log archiving stop at that LSN instead of on reaching the end
  of redo log.
- Support stopping redo log archiving in a middle of a log block. As the old
  implementation may encounter this case only at the end of the redo log, it
  took that block from the log buffer in memory. This no longer works with the
  new implementation, where the needed block might reside on disk only. Address
  this by forcing the whole block containing the target LSN to disk, reading it
  from there, and patching its memory copy to stop at the required LSN before
  giving it for the clone to copy.
- To implement redo log block reading on a running server, adapt crash recovery
  function recv_read_log_seg to work both for crash recovery and on a running
  server. This is similar to what Percona Server does for its changed page
  tracking feature.
- Make sure not to archive not-yet-written log: pfs.log_status query gets the
  log system LSN, which is the value of highest log reservation. Thus in an
  archiving iteration, archive up to the lower values of log archiving LSN, and
  write_lsn. Set the log archiving LSN to LSN_MAX at clone initialization, so
  that clone archiving may proceed until the stop LSN is set.
- The above implementation results in cloned LSN != log end LSN for the cloned
  instance, thus adjust crash recovery to work with that. The existing crash
  recovery throws away any crash-recovered metadata and reads its snapshot from
  the data files, whereas the snapshot was not written in any consistency with
  the clone LSN. Fix by no longer throwing away the crash-recovered InnoDB
  dynamic metadata and by not re-reading the the last metadata snapshot. This
  forces also not to re-serialize the metadata in the crash recovery as that
  writes redo log which is to be avoided during clone crash recovery.
- Additionally, at the start of the redo log copy the old implementation
  requested all InnoDB updates to be flushed to disk immediately, to avoid clone
  recovery generating new redo log. This is also disabled as it becomes
  redundant with the fix. This issue is similar to Percona XtraBackup
  https://jira.percona.com/browse/PXB-1634
- Disable clone.local_xa test because the new synchronization no longer blocks
  on XA transactions.
- Re-record two clone-related tests in the component_keyring_file MTR suite.
- Temporarily workaround the MyRocks bug where log_status query hangs if RocksDB
  file deletions were not disabled first.

Note that this change breaks some of the clone-related group replication MTR
tests. As they are not tested in fb-mysql and have been already broken, this
patch makes no effort to fix them.

MyRocks Clone (5/n): fix bug 109922 (SEs do not coordinate clone rollbacks on
instance startup)

If InnoDB clone decides to rollback during its startup, other storage engines
are not aware of this and proceed as usual, resulting in inconsistent instance
between storage engines. Ideally this should be fixed through a 2PC-like design
where all the SEs vote to rollback the clone or to continue during the startup,
but for MyRocks clone a simpler solution suffices: make InnoDB write a special
marker file if it decides to rollback on clone startup.

MyRocks Clone (6/n): on remote clone error, always send COM_ACK to InnoDB too

In the case of multithreaded clone, the waits for the donor threads for the
current clone stage to complete are implemented fully in InnoDB, even though
that, and error handling, should be plugin-level features. With two storage
engines participating in the clone this allows the situation where the other
storage engine client reports an error with COM_ACK to one donor thread, while
some other threads are in InnoDB clone state change waits. Those waits can abort
if InnoDB clone gets an error from the client, but they know nothing about other
SE errors, and will be aborted only by timeouts.

As a minimal fix that does not involve pulling the whole error handling and
state change waiting logic to the plugin layer, make the clone client always
send an error-containing COM_ACK to InnoDB too, whenever it sends one to another
storage engine.

MyRocks Clone (7/n): MyRocks clone implementation

- Cleanup any stale RocksDB checkpoint directories on server shutdown too.

- For MyRocks clone network protocol, define four packet types (clone/common.h):
  - LOCATOR_V1: MyRocks-specific clone locator, consisting of a monotonically
    increasing clone session ID for the donor with an optional applier
    state (an array of file IDs and applied offsets) for restarts, or a zero
    in which case it is a version check empty locator used for handshakes.
  - FILE_NAME_MAP_V1: establishes a relation between a file ID, its name and
    target size.
  - FILE_CHUNK_V1: cloned file chunk data
  - ADD_ESTIMATE_V1: how many bytes should be added to the estimated clone data
    size.
- All packet types share the common package header, which stores the version,
  the payload length without the header, and the packet type.
- (clone/common.h): classes locator, file_metadata, chunk_metadata, &
  estimate_metadata correspond to the four packet types. They are subclassed on
  the donor side to add serialization routines and other donor-specific code and
  on the client side to add deserialization routines and other client-specific
  code. Re. serialization/deserialization, an exception is the locator, whose
  routines are implemented in the common code, because it may be passed both
  - ways.
- (clone/common.h): Introduce a session class that handles the task ID set and
  errors, which is common for both donor and client. For errors, allow saving
  one error, with an optional associated path and errno, that can be set by any
  thread and later reported to the clone plugin.
- (clone/common.cc) Setup MyRocks clone capabilities as HA_CLONE_HYBRID (not
  - applicable but matches InnoDB), HA_CLONE_MULTI_TASK (can clone with multiple
  - threads), & HA_CLONE_RESTART (can restart after a network I/O error).
- (clone/common.cc) Enable O_DIRECT for clone file I/O if, on client if
  rocksdb_use_direct_io_for_flush_and_compaction is enabled, and on donor if
  rocksdb_use_direct_reads is enabled.
- On the donor side (clone/donor.cc), encapsulate RocksDB checkpoints in a
  rdb_checkpoint class. Each checkpoint gets its own ID, which is also a donor
  session ID. The ID is constant for rolling checkpoints.
- Subclass the common file_metadata to donor_file_metadata, chunk_metadata to
  donor_chunk_metadata, estimate_metadata to donor_estimate.
- Implement donor going through a sequence of states: NONE, ROLLING, FINAL,
  FINAL_WITH_LOGS. To support multiple clone donor sessions, implement a manager
  class that looks up the donor instance by locator.
- Implement MyRocks clone donor interface:
  - rocksdb_clone_start: handle the possible start modes:
    - HA_CLONE_MODE_START: create and register a new donor instance, register
      its master task ID.
    - HA_CLONE_MODE_ADD_TASK: lookup the existing donor instance, register a new
      task ID.
    - HA_CLONE_MODE_RESTART: lookup the existing donor instance, wait for the
      old master session to drop it by master task ID reference count, then
      process the locator applier state to reset the donor state.
  - rocksdb_clone_precopy: lookup the donor instance, create a RocksDB snapshot
    if state == NONE, otherwise send files by chunks and roll the checkpoint as
    needed.
  - rocksdb_clone_set_log_stop: add the WAL files to the clone file to copy set.
  - rocksdb_clone_copy: copy the clone files in any donor state.
  - rocksdb_clone_ack: acknowledge errors sent by client.
  - rocksdb_clone_end: for master task ID, stop accepting new worker join
    requests, wait until existing workers quit, and, in the case of restartable
    error, wait for any restarts to happen. Then decrement ref count for the
    master task ID (which will handle both the cases of restarted and not
    restarted) clone. For all other task IDs, simply unregister them.
- On the client side (clone/client.cc), have file_in_progress class to track
  each individual file progress during application, and completed_file for
  fully-applied files.
- Subclass the common file_metadata to client_file_metadata, implement its
  reading from the net buffer. Likewise for client_chunk_metadata subclassing
  chunk_metadata, and client_estimate for estimate_metadata.
- TODO client class.
- Implement MyRocks clone client interface:
  - rocksdb_clone_apply_begin: handle the possible start modes:
    - HA_CLONE_MODE_START: initialize the client instance for a new clone
    - HA_CLONE_MODE_ADD_TASK: register a new task ID for an ongoing clone
    - HA_CLONE_MODE_RESTART: restart the interrupted clone session
    - HA_CLONE_MODE_VERSION: prepare new version locator
  - rocksdb_clone_apply: if passed an error code, save it. Otherwise verify the
    passed buffer and dispatch ADD_ESTIMATE_V1, FILE_NAME_MAP_V1, &
    FILE_CHUNK_V1 packets accordingly.
  - rocksdb_clone_apply_end: for master task, wait for all other tasks to
    complete and finish the clone. For both master and non-master tasks
    unregister their IDs.
- Add rocksdb_sys_vars tests for the new clone-related system variables. They
  are included in the current patch and not the MTR one in order not to break
  the existing rocksdb_sys_vars.all_vars test.
- On instance startup, do clone fixup (clone/common.cc). It consists of:
  - Check whether InnoDB clone rollback marker is present. If yes, force
    rollback of MyRocks clone.
  - If the cloned temporary datadir contains clone-in-progress marker, assume
    half-completed clone and abort.
  - Move the cloned datadirs and WAL dirs in-place. For the datadir, do a
    directory-level rename with a save and later delete of the old datadir. For
    the WAL dir do not try to do parent directory-level renames because the
    mysqld process may not have  permissions to delete that directory. Instead,
    rename all the old files individually, move new files in, delete the old
    files.
  - For all the rename operations, handle them returning EXDEV error (a rename
    would cross a filesystem boundary) by falling back to copy and delete. As we
    expect that it is the cloned RocksDB log files that are most affected by
    this, the total copy I/O  wouldn't be large, and so the default MySQL copy
    routine my_copy is used, without extra copy throughput optimizations. Add a
    fault injection point for MTR to test this code paths.
  - Remove any RocksDB clone checkpoints found.
- On instance startup, always move WALs to the target directory if they are
  found in the rocksdb_datadir yet the rocksdb_wal_dir is different. This
  handles the case of setting rocksdb_wal_dir while instance is shutdown, both
  for clone fixup and regular operation.
- Add PFS instrumentation for all the instrumentable entities in this
  patch (mutexes, rwlocks, file I/O).

MyRocks Clone (8/n): MyRocks clone MTR tests

- Add a new MTR test suite rocksdb_clone
- Support new variable MTR clone/include/clone_command.inc:
  clone_single_threaded, setting which forces single-threaded clone operation.
- Reuse a big part of InnoDB clone tests by including their test files, with the
  exception of tests involving DDL, as concurrent DDL is not supported by
  MyRocks clone at the moment, and debug tests that adjust cloned instance LSN,
  because they desynchronize storage engines. The reused tests are: local_basic,
  local_boundary, local_dml, local_dml_auto_tune, local_dml_no_redo,
  local_insert, local_stage, monitor_progress, remote_basic,
  remote_basic_replace, remote_boundary, remote_dml, remote_dml_auto_tune,
  remote_dml_gtid, remote_dml_no_binlog, remote_dml_no_ssl, remote_dml_recovery,
  remote_dml_replace, remote_dml_ssl, remote_dml_ssl_certificate,
  remote_dml_upgrade, remote_insert, remote_monitor_progress, remote_restart,
  remote_skip_validation, remote_stage, remote_stage_restart.
- As a part of the above, add MyRocks support to local_basic and
  monitor_progress MTR tests.
- Besides the reused InnoDB clone tests, add some new MyRocks-specific tests:
- local_minimal_1_thread, which is the simplest possible local clone test.
- local_minimal_multithreaded, the simplest possible but multithreaded local
  clone test.
- local_minimal_direct_io, which is the simplest possible local clone test in
  all combinations of rocksdb_use_direct_reads and
  rocksdb_use_direct_io_for_flush_and_compaction settings, which are the
  O_DIRECT-enabling options.
- local_dml_o_direct_donor, local_dml_o_direct_client, local_dml_o_direct_both
  that reuse InnoDB local_dml test, but combined with either or both of the
  O_DIRECT-enabling options.
- local_minimal_wal_dir, which clones with various rocksdb_wal_dir settings on
  donor and client.
- local_minimal_wal_dir_copy_delete, which uses a debug injection to test the
  move-across-filesystem-boundary code path in clone startup.
- remote_minimal_direct_io_{1, 2, 3, 4} tests for all the combinations of
  O_DIRECT-enabling options for donor and client instances.
- remote_fail_apply, which uses a debug injection to fail the clone client
  during remote clone.
- remote_inplace_restart, which performs a remote in-place clone with a
  simulated transient error on the client.
- remote_kill_donor, which kill the donor instance at various points of remote
  cloning.
- rolling_checkpoint for testing various combinations of clone rolling
  checkpoint tuning options.
- three_se_basic to test a simple clone across InnoDB, MyRocks, & Blackhole
  engines.

Fix moving RocksDB WALs on startup across file system boundaries

Reviewed By: sunshine-Chun

Differential Revision: D43160162

Pulled By: hermanlee

fbshipit-source-id: 5358a57

--------------------------------------------------------------------------------

Rdb clone fixes (facebook#1271)

Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 2245706

--------------------------------------------------------------------------------

Clone: fix race condition that may skip some SST files (facebook#1274)

Summary:
A donor thread in the main copy loop took a file to send, released the donor state mutex, and then opened the file. If an ENOENT was received at this point, it was assumed that this was a stale SST file from an older checkpoint that was rolled since.

Because the donor state mutex was released between taking of the file and opening it, the following race was possible:
1) Thread 1 takes the file
2) Thread 2 decides to roll the checkpoint, the old checkpoint is deleted 3) Thread 1 tries to open the file, gets ENOENT
4) Thread 2 creates the new checkpoint, the file re-appears, but it's too late.

Rolling the checkpoint in a donor state mutex critical section is a possible fix, but such section would do a lot of I/O, serializing the parallel threads. Instead, fix by taking the file and opening it in the same critical section.

Pull Request resolved: facebook#1274

Reviewed By: sunshine-Chun

Differential Revision: D43629546

Pulled By: hermanlee

fbshipit-source-id: 6e7f315
inikep pushed a commit to inikep/mysql-5.6 that referenced this issue Jun 14, 2023
Summary: Pull Request resolved: facebook#1250

MyRocks Clone (1/n): make InnoDB clone MTR tests SE-agnostic

The MyRocks clone MTR tests will reuse the InnoDB tests by including them.
Adjust the tests so that they work this way, i.e. make them storage
engine-agnostic by checking the default_storage_engine system variable and
guarding InnoDB-specific tests this way. This includes the tests that
artificially desync the InnoDB redo log position.

MyRocks Clone (2/n): fix bug 109919 (Clone plugin does not support arbitrary
length O_DIRECT files)

The existing clone plugin I/O routines handle O_DIRECT files with the assumption
that their sizes are always a multiple of a block size. This is too restrictive
because in general O_DIRECT files can be of any size and MyRocks files actually are.

Add a flag to Ha_clone_file to support arbitrary file lengths under O_DIRECT:
o_direct_uneven_file_size. Pass it to clone_client.h/cc
Client::get_aligned_buffer to allocate the buffer that bumps the length to the
next CLONE_OS_ALIGN multiple if it's not already one. Introduce a
pad_for_o_direct helper that performs this calculation for the passed length as
needed.

For writes, if O_DIRECT is enabled and the buffer length is not a multiple of
CLONE_OS_ALIGN, pad it with zeros to the next multiple. Then check the actual
bytes written, and if any padding bytes were actually written, call my_chsize to
truncate the file to the needed size.

MyRocks Clone (3/n): fix bug 109920 (SEs cannot add to clone data size estimates)

In the current implementation, once the first SE sets the data size estimate for
a particular stage, there is no way for other SEs to add to it. Address this by
adding Ha_clone_cbk::add_to_data_size_estimate method.

MyRocks Clone (4/n): fix bug 109921 (Cannot clone with synchronization across SEs)

The existing clone implementation synchronizes InnoDB and binlog directly. There
is no way to make other storage engines join this synchronization, thus replace
it with synchronizing by a performance_schema.log_status query.

- Add table_log_status::get_row method to allow doing the log_status query from
  inside the server.
- Handlerton: introduce two new methods: clone_precopy - currently unused, which
  will handle any non-consistent-snapshot copying before the main consistent
  copy, and set_log_stop - which provides the SE the clone stop position
  information.
- Plugin: introduce Ha_clone_common_cbk, which derives from Ha_clone_cbk, and
  from which derive all the concrete classes: Client_Cbk, Local_Cbk &
  Server_Cbk.
- Ha_clone_common_cbk contains two methods: precopy and synchronize_engines. The
  former calls precopy for all the participating clone SE handlertons, if
  implemented. The latter, only to be called from the main clone thread,
  performs the performance_schema.LOG_STATUS query, splits its result per-SE and
  calls handlerton set_log_stop for all SEs.
- The pfs.log_status query still must be called by InnoDB at a certain point in
  clone. That means that InnoDB is still a special SE in that it must always
  come first in the clone locator vectors. This will also handle donor waits on
  clone restarts, as that is implemented in InnoDB-specific code. To ensure
  this, add a helper clone_hton.cc:make_innodb_first which shuffles the vectors
  as needed. Call it in all the clone-starting code paths, add asserts that
  InnoDB comes first.
- Since hton_clone_copy will call into InnoDB clone copy, which at certain point
  will synchronize engines, calling into the plugin again, and the calling into
  SEs again, introduce a backup locator vector Ha_clone_cbk::all_locators, and
  its setters/getters set_all_locators, reset_all_locators, get_all_locators.
  Use them in the re-entrant hton_clone_copy code.
- Introduce a new InnoDB clone state named SST COPY, which calls precopy and has
  zero files and chunks set to copy in the regular InnoDB clone way, i.e. all it
  does it the precopy.
- The redo log archiving keeps running throughout the precopy stage.
- On stage transition from the precopy to the redo log copy. call synchronize.
- Implement innodb_clone_set_log_stop by parsing out the desired clone stop LSN
  from the passed JSON from the log_status query.
- Make InnoDB redo log archiving stop at that LSN instead of on reaching the end
  of redo log.
- Support stopping redo log archiving in a middle of a log block. As the old
  implementation may encounter this case only at the end of the redo log, it
  took that block from the log buffer in memory. This no longer works with the
  new implementation, where the needed block might reside on disk only. Address
  this by forcing the whole block containing the target LSN to disk, reading it
  from there, and patching its memory copy to stop at the required LSN before
  giving it for the clone to copy.
- To implement redo log block reading on a running server, adapt crash recovery
  function recv_read_log_seg to work both for crash recovery and on a running
  server. This is similar to what Percona Server does for its changed page
  tracking feature.
- Make sure not to archive not-yet-written log: pfs.log_status query gets the
  log system LSN, which is the value of highest log reservation. Thus in an
  archiving iteration, archive up to the lower values of log archiving LSN, and
  write_lsn. Set the log archiving LSN to LSN_MAX at clone initialization, so
  that clone archiving may proceed until the stop LSN is set.
- The above implementation results in cloned LSN != log end LSN for the cloned
  instance, thus adjust crash recovery to work with that. The existing crash
  recovery throws away any crash-recovered metadata and reads its snapshot from
  the data files, whereas the snapshot was not written in any consistency with
  the clone LSN. Fix by no longer throwing away the crash-recovered InnoDB
  dynamic metadata and by not re-reading the the last metadata snapshot. This
  forces also not to re-serialize the metadata in the crash recovery as that
  writes redo log which is to be avoided during clone crash recovery.
- Additionally, at the start of the redo log copy the old implementation
  requested all InnoDB updates to be flushed to disk immediately, to avoid clone
  recovery generating new redo log. This is also disabled as it becomes
  redundant with the fix. This issue is similar to Percona XtraBackup
  https://jira.percona.com/browse/PXB-1634
- Disable clone.local_xa test because the new synchronization no longer blocks
  on XA transactions.
- Re-record two clone-related tests in the component_keyring_file MTR suite.
- Temporarily workaround the MyRocks bug where log_status query hangs if RocksDB
  file deletions were not disabled first.

Note that this change breaks some of the clone-related group replication MTR
tests. As they are not tested in fb-mysql and have been already broken, this
patch makes no effort to fix them.

MyRocks Clone (5/n): fix bug 109922 (SEs do not coordinate clone rollbacks on
instance startup)

If InnoDB clone decides to rollback during its startup, other storage engines
are not aware of this and proceed as usual, resulting in inconsistent instance
between storage engines. Ideally this should be fixed through a 2PC-like design
where all the SEs vote to rollback the clone or to continue during the startup,
but for MyRocks clone a simpler solution suffices: make InnoDB write a special
marker file if it decides to rollback on clone startup.

MyRocks Clone (6/n): on remote clone error, always send COM_ACK to InnoDB too

In the case of multithreaded clone, the waits for the donor threads for the
current clone stage to complete are implemented fully in InnoDB, even though
that, and error handling, should be plugin-level features. With two storage
engines participating in the clone this allows the situation where the other
storage engine client reports an error with COM_ACK to one donor thread, while
some other threads are in InnoDB clone state change waits. Those waits can abort
if InnoDB clone gets an error from the client, but they know nothing about other
SE errors, and will be aborted only by timeouts.

As a minimal fix that does not involve pulling the whole error handling and
state change waiting logic to the plugin layer, make the clone client always
send an error-containing COM_ACK to InnoDB too, whenever it sends one to another
storage engine.

MyRocks Clone (7/n): MyRocks clone implementation

- Cleanup any stale RocksDB checkpoint directories on server shutdown too.

- For MyRocks clone network protocol, define four packet types (clone/common.h):
  - LOCATOR_V1: MyRocks-specific clone locator, consisting of a monotonically
    increasing clone session ID for the donor with an optional applier
    state (an array of file IDs and applied offsets) for restarts, or a zero
    in which case it is a version check empty locator used for handshakes.
  - FILE_NAME_MAP_V1: establishes a relation between a file ID, its name and
    target size.
  - FILE_CHUNK_V1: cloned file chunk data
  - ADD_ESTIMATE_V1: how many bytes should be added to the estimated clone data
    size.
- All packet types share the common package header, which stores the version,
  the payload length without the header, and the packet type.
- (clone/common.h): classes locator, file_metadata, chunk_metadata, &
  estimate_metadata correspond to the four packet types. They are subclassed on
  the donor side to add serialization routines and other donor-specific code and
  on the client side to add deserialization routines and other client-specific
  code. Re. serialization/deserialization, an exception is the locator, whose
  routines are implemented in the common code, because it may be passed both
  - ways.
- (clone/common.h): Introduce a session class that handles the task ID set and
  errors, which is common for both donor and client. For errors, allow saving
  one error, with an optional associated path and errno, that can be set by any
  thread and later reported to the clone plugin.
- (clone/common.cc) Setup MyRocks clone capabilities as HA_CLONE_HYBRID (not
  - applicable but matches InnoDB), HA_CLONE_MULTI_TASK (can clone with multiple
  - threads), & HA_CLONE_RESTART (can restart after a network I/O error).
- (clone/common.cc) Enable O_DIRECT for clone file I/O if, on client if
  rocksdb_use_direct_io_for_flush_and_compaction is enabled, and on donor if
  rocksdb_use_direct_reads is enabled.
- On the donor side (clone/donor.cc), encapsulate RocksDB checkpoints in a
  rdb_checkpoint class. Each checkpoint gets its own ID, which is also a donor
  session ID. The ID is constant for rolling checkpoints.
- Subclass the common file_metadata to donor_file_metadata, chunk_metadata to
  donor_chunk_metadata, estimate_metadata to donor_estimate.
- Implement donor going through a sequence of states: NONE, ROLLING, FINAL,
  FINAL_WITH_LOGS. To support multiple clone donor sessions, implement a manager
  class that looks up the donor instance by locator.
- Implement MyRocks clone donor interface:
  - rocksdb_clone_start: handle the possible start modes:
    - HA_CLONE_MODE_START: create and register a new donor instance, register
      its master task ID.
    - HA_CLONE_MODE_ADD_TASK: lookup the existing donor instance, register a new
      task ID.
    - HA_CLONE_MODE_RESTART: lookup the existing donor instance, wait for the
      old master session to drop it by master task ID reference count, then
      process the locator applier state to reset the donor state.
  - rocksdb_clone_precopy: lookup the donor instance, create a RocksDB snapshot
    if state == NONE, otherwise send files by chunks and roll the checkpoint as
    needed.
  - rocksdb_clone_set_log_stop: add the WAL files to the clone file to copy set.
  - rocksdb_clone_copy: copy the clone files in any donor state.
  - rocksdb_clone_ack: acknowledge errors sent by client.
  - rocksdb_clone_end: for master task ID, stop accepting new worker join
    requests, wait until existing workers quit, and, in the case of restartable
    error, wait for any restarts to happen. Then decrement ref count for the
    master task ID (which will handle both the cases of restarted and not
    restarted) clone. For all other task IDs, simply unregister them.
- On the client side (clone/client.cc), have file_in_progress class to track
  each individual file progress during application, and completed_file for
  fully-applied files.
- Subclass the common file_metadata to client_file_metadata, implement its
  reading from the net buffer. Likewise for client_chunk_metadata subclassing
  chunk_metadata, and client_estimate for estimate_metadata.
- TODO client class.
- Implement MyRocks clone client interface:
  - rocksdb_clone_apply_begin: handle the possible start modes:
    - HA_CLONE_MODE_START: initialize the client instance for a new clone
    - HA_CLONE_MODE_ADD_TASK: register a new task ID for an ongoing clone
    - HA_CLONE_MODE_RESTART: restart the interrupted clone session
    - HA_CLONE_MODE_VERSION: prepare new version locator
  - rocksdb_clone_apply: if passed an error code, save it. Otherwise verify the
    passed buffer and dispatch ADD_ESTIMATE_V1, FILE_NAME_MAP_V1, &
    FILE_CHUNK_V1 packets accordingly.
  - rocksdb_clone_apply_end: for master task, wait for all other tasks to
    complete and finish the clone. For both master and non-master tasks
    unregister their IDs.
- Add rocksdb_sys_vars tests for the new clone-related system variables. They
  are included in the current patch and not the MTR one in order not to break
  the existing rocksdb_sys_vars.all_vars test.
- On instance startup, do clone fixup (clone/common.cc). It consists of:
  - Check whether InnoDB clone rollback marker is present. If yes, force
    rollback of MyRocks clone.
  - If the cloned temporary datadir contains clone-in-progress marker, assume
    half-completed clone and abort.
  - Move the cloned datadirs and WAL dirs in-place. For the datadir, do a
    directory-level rename with a save and later delete of the old datadir. For
    the WAL dir do not try to do parent directory-level renames because the
    mysqld process may not have  permissions to delete that directory. Instead,
    rename all the old files individually, move new files in, delete the old
    files.
  - For all the rename operations, handle them returning EXDEV error (a rename
    would cross a filesystem boundary) by falling back to copy and delete. As we
    expect that it is the cloned RocksDB log files that are most affected by
    this, the total copy I/O  wouldn't be large, and so the default MySQL copy
    routine my_copy is used, without extra copy throughput optimizations. Add a
    fault injection point for MTR to test this code paths.
  - Remove any RocksDB clone checkpoints found.
- On instance startup, always move WALs to the target directory if they are
  found in the rocksdb_datadir yet the rocksdb_wal_dir is different. This
  handles the case of setting rocksdb_wal_dir while instance is shutdown, both
  for clone fixup and regular operation.
- Add PFS instrumentation for all the instrumentable entities in this
  patch (mutexes, rwlocks, file I/O).

MyRocks Clone (8/n): MyRocks clone MTR tests

- Add a new MTR test suite rocksdb_clone
- Support new variable MTR clone/include/clone_command.inc:
  clone_single_threaded, setting which forces single-threaded clone operation.
- Reuse a big part of InnoDB clone tests by including their test files, with the
  exception of tests involving DDL, as concurrent DDL is not supported by
  MyRocks clone at the moment, and debug tests that adjust cloned instance LSN,
  because they desynchronize storage engines. The reused tests are: local_basic,
  local_boundary, local_dml, local_dml_auto_tune, local_dml_no_redo,
  local_insert, local_stage, monitor_progress, remote_basic,
  remote_basic_replace, remote_boundary, remote_dml, remote_dml_auto_tune,
  remote_dml_gtid, remote_dml_no_binlog, remote_dml_no_ssl, remote_dml_recovery,
  remote_dml_replace, remote_dml_ssl, remote_dml_ssl_certificate,
  remote_dml_upgrade, remote_insert, remote_monitor_progress, remote_restart,
  remote_skip_validation, remote_stage, remote_stage_restart.
- As a part of the above, add MyRocks support to local_basic and
  monitor_progress MTR tests.
- Besides the reused InnoDB clone tests, add some new MyRocks-specific tests:
- local_minimal_1_thread, which is the simplest possible local clone test.
- local_minimal_multithreaded, the simplest possible but multithreaded local
  clone test.
- local_minimal_direct_io, which is the simplest possible local clone test in
  all combinations of rocksdb_use_direct_reads and
  rocksdb_use_direct_io_for_flush_and_compaction settings, which are the
  O_DIRECT-enabling options.
- local_dml_o_direct_donor, local_dml_o_direct_client, local_dml_o_direct_both
  that reuse InnoDB local_dml test, but combined with either or both of the
  O_DIRECT-enabling options.
- local_minimal_wal_dir, which clones with various rocksdb_wal_dir settings on
  donor and client.
- local_minimal_wal_dir_copy_delete, which uses a debug injection to test the
  move-across-filesystem-boundary code path in clone startup.
- remote_minimal_direct_io_{1, 2, 3, 4} tests for all the combinations of
  O_DIRECT-enabling options for donor and client instances.
- remote_fail_apply, which uses a debug injection to fail the clone client
  during remote clone.
- remote_inplace_restart, which performs a remote in-place clone with a
  simulated transient error on the client.
- remote_kill_donor, which kill the donor instance at various points of remote
  cloning.
- rolling_checkpoint for testing various combinations of clone rolling
  checkpoint tuning options.
- three_se_basic to test a simple clone across InnoDB, MyRocks, & Blackhole
  engines.

Fix moving RocksDB WALs on startup across file system boundaries

Reviewed By: sunshine-Chun

Differential Revision: D43160162

Pulled By: hermanlee

fbshipit-source-id: 5358a57

--------------------------------------------------------------------------------

Rdb clone fixes (facebook#1271)

Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 2245706

--------------------------------------------------------------------------------

Clone: fix race condition that may skip some SST files (facebook#1274)

Summary:
A donor thread in the main copy loop took a file to send, released the donor state mutex, and then opened the file. If an ENOENT was received at this point, it was assumed that this was a stale SST file from an older checkpoint that was rolled since.

Because the donor state mutex was released between taking of the file and opening it, the following race was possible:
1) Thread 1 takes the file
2) Thread 2 decides to roll the checkpoint, the old checkpoint is deleted 3) Thread 1 tries to open the file, gets ENOENT
4) Thread 2 creates the new checkpoint, the file re-appears, but it's too late.

Rolling the checkpoint in a donor state mutex critical section is a possible fix, but such section would do a lot of I/O, serializing the parallel threads. Instead, fix by taking the file and opening it in the same critical section.

Pull Request resolved: facebook#1274

Reviewed By: sunshine-Chun

Differential Revision: D43629546

Pulled By: hermanlee

fbshipit-source-id: 6e7f315

--------------------------------------------------------------------------------

MyRocks clone: add missing donor mutex unlocks on error exits (facebook#1317)

Summary: Pull Request resolved: facebook#1317

Differential Revision: D46564079

fbshipit-source-id: 6fc1a4f
inikep pushed a commit to inikep/mysql-5.6 that referenced this issue Jun 19, 2023
Summary: Pull Request resolved: facebook#1250

MyRocks Clone (1/n): make InnoDB clone MTR tests SE-agnostic

The MyRocks clone MTR tests will reuse the InnoDB tests by including them.
Adjust the tests so that they work this way, i.e. make them storage
engine-agnostic by checking the default_storage_engine system variable and
guarding InnoDB-specific tests this way. This includes the tests that
artificially desync the InnoDB redo log position.

MyRocks Clone (2/n): fix bug 109919 (Clone plugin does not support arbitrary
length O_DIRECT files)

The existing clone plugin I/O routines handle O_DIRECT files with the assumption
that their sizes are always a multiple of a block size. This is too restrictive
because in general O_DIRECT files can be of any size and MyRocks files actually are.

Add a flag to Ha_clone_file to support arbitrary file lengths under O_DIRECT:
o_direct_uneven_file_size. Pass it to clone_client.h/cc
Client::get_aligned_buffer to allocate the buffer that bumps the length to the
next CLONE_OS_ALIGN multiple if it's not already one. Introduce a
pad_for_o_direct helper that performs this calculation for the passed length as
needed.

For writes, if O_DIRECT is enabled and the buffer length is not a multiple of
CLONE_OS_ALIGN, pad it with zeros to the next multiple. Then check the actual
bytes written, and if any padding bytes were actually written, call my_chsize to
truncate the file to the needed size.

MyRocks Clone (3/n): fix bug 109920 (SEs cannot add to clone data size estimates)

In the current implementation, once the first SE sets the data size estimate for
a particular stage, there is no way for other SEs to add to it. Address this by
adding Ha_clone_cbk::add_to_data_size_estimate method.

MyRocks Clone (4/n): fix bug 109921 (Cannot clone with synchronization across SEs)

The existing clone implementation synchronizes InnoDB and binlog directly. There
is no way to make other storage engines join this synchronization, thus replace
it with synchronizing by a performance_schema.log_status query.

- Add table_log_status::get_row method to allow doing the log_status query from
  inside the server.
- Handlerton: introduce two new methods: clone_precopy - currently unused, which
  will handle any non-consistent-snapshot copying before the main consistent
  copy, and set_log_stop - which provides the SE the clone stop position
  information.
- Plugin: introduce Ha_clone_common_cbk, which derives from Ha_clone_cbk, and
  from which derive all the concrete classes: Client_Cbk, Local_Cbk &
  Server_Cbk.
- Ha_clone_common_cbk contains two methods: precopy and synchronize_engines. The
  former calls precopy for all the participating clone SE handlertons, if
  implemented. The latter, only to be called from the main clone thread,
  performs the performance_schema.LOG_STATUS query, splits its result per-SE and
  calls handlerton set_log_stop for all SEs.
- The pfs.log_status query still must be called by InnoDB at a certain point in
  clone. That means that InnoDB is still a special SE in that it must always
  come first in the clone locator vectors. This will also handle donor waits on
  clone restarts, as that is implemented in InnoDB-specific code. To ensure
  this, add a helper clone_hton.cc:make_innodb_first which shuffles the vectors
  as needed. Call it in all the clone-starting code paths, add asserts that
  InnoDB comes first.
- Since hton_clone_copy will call into InnoDB clone copy, which at certain point
  will synchronize engines, calling into the plugin again, and the calling into
  SEs again, introduce a backup locator vector Ha_clone_cbk::all_locators, and
  its setters/getters set_all_locators, reset_all_locators, get_all_locators.
  Use them in the re-entrant hton_clone_copy code.
- Introduce a new InnoDB clone state named SST COPY, which calls precopy and has
  zero files and chunks set to copy in the regular InnoDB clone way, i.e. all it
  does it the precopy.
- The redo log archiving keeps running throughout the precopy stage.
- On stage transition from the precopy to the redo log copy. call synchronize.
- Implement innodb_clone_set_log_stop by parsing out the desired clone stop LSN
  from the passed JSON from the log_status query.
- Make InnoDB redo log archiving stop at that LSN instead of on reaching the end
  of redo log.
- Support stopping redo log archiving in a middle of a log block. As the old
  implementation may encounter this case only at the end of the redo log, it
  took that block from the log buffer in memory. This no longer works with the
  new implementation, where the needed block might reside on disk only. Address
  this by forcing the whole block containing the target LSN to disk, reading it
  from there, and patching its memory copy to stop at the required LSN before
  giving it for the clone to copy.
- To implement redo log block reading on a running server, adapt crash recovery
  function recv_read_log_seg to work both for crash recovery and on a running
  server. This is similar to what Percona Server does for its changed page
  tracking feature.
- Make sure not to archive not-yet-written log: pfs.log_status query gets the
  log system LSN, which is the value of highest log reservation. Thus in an
  archiving iteration, archive up to the lower values of log archiving LSN, and
  write_lsn. Set the log archiving LSN to LSN_MAX at clone initialization, so
  that clone archiving may proceed until the stop LSN is set.
- The above implementation results in cloned LSN != log end LSN for the cloned
  instance, thus adjust crash recovery to work with that. The existing crash
  recovery throws away any crash-recovered metadata and reads its snapshot from
  the data files, whereas the snapshot was not written in any consistency with
  the clone LSN. Fix by no longer throwing away the crash-recovered InnoDB
  dynamic metadata and by not re-reading the the last metadata snapshot. This
  forces also not to re-serialize the metadata in the crash recovery as that
  writes redo log which is to be avoided during clone crash recovery.
- Additionally, at the start of the redo log copy the old implementation
  requested all InnoDB updates to be flushed to disk immediately, to avoid clone
  recovery generating new redo log. This is also disabled as it becomes
  redundant with the fix. This issue is similar to Percona XtraBackup
  https://jira.percona.com/browse/PXB-1634
- Disable clone.local_xa test because the new synchronization no longer blocks
  on XA transactions.
- Re-record two clone-related tests in the component_keyring_file MTR suite.
- Temporarily workaround the MyRocks bug where log_status query hangs if RocksDB
  file deletions were not disabled first.

Note that this change breaks some of the clone-related group replication MTR
tests. As they are not tested in fb-mysql and have been already broken, this
patch makes no effort to fix them.

MyRocks Clone (5/n): fix bug 109922 (SEs do not coordinate clone rollbacks on
instance startup)

If InnoDB clone decides to rollback during its startup, other storage engines
are not aware of this and proceed as usual, resulting in inconsistent instance
between storage engines. Ideally this should be fixed through a 2PC-like design
where all the SEs vote to rollback the clone or to continue during the startup,
but for MyRocks clone a simpler solution suffices: make InnoDB write a special
marker file if it decides to rollback on clone startup.

MyRocks Clone (6/n): on remote clone error, always send COM_ACK to InnoDB too

In the case of multithreaded clone, the waits for the donor threads for the
current clone stage to complete are implemented fully in InnoDB, even though
that, and error handling, should be plugin-level features. With two storage
engines participating in the clone this allows the situation where the other
storage engine client reports an error with COM_ACK to one donor thread, while
some other threads are in InnoDB clone state change waits. Those waits can abort
if InnoDB clone gets an error from the client, but they know nothing about other
SE errors, and will be aborted only by timeouts.

As a minimal fix that does not involve pulling the whole error handling and
state change waiting logic to the plugin layer, make the clone client always
send an error-containing COM_ACK to InnoDB too, whenever it sends one to another
storage engine.

MyRocks Clone (7/n): MyRocks clone implementation

- Cleanup any stale RocksDB checkpoint directories on server shutdown too.

- For MyRocks clone network protocol, define four packet types (clone/common.h):
  - LOCATOR_V1: MyRocks-specific clone locator, consisting of a monotonically
    increasing clone session ID for the donor with an optional applier
    state (an array of file IDs and applied offsets) for restarts, or a zero
    in which case it is a version check empty locator used for handshakes.
  - FILE_NAME_MAP_V1: establishes a relation between a file ID, its name and
    target size.
  - FILE_CHUNK_V1: cloned file chunk data
  - ADD_ESTIMATE_V1: how many bytes should be added to the estimated clone data
    size.
- All packet types share the common package header, which stores the version,
  the payload length without the header, and the packet type.
- (clone/common.h): classes locator, file_metadata, chunk_metadata, &
  estimate_metadata correspond to the four packet types. They are subclassed on
  the donor side to add serialization routines and other donor-specific code and
  on the client side to add deserialization routines and other client-specific
  code. Re. serialization/deserialization, an exception is the locator, whose
  routines are implemented in the common code, because it may be passed both
  - ways.
- (clone/common.h): Introduce a session class that handles the task ID set and
  errors, which is common for both donor and client. For errors, allow saving
  one error, with an optional associated path and errno, that can be set by any
  thread and later reported to the clone plugin.
- (clone/common.cc) Setup MyRocks clone capabilities as HA_CLONE_HYBRID (not
  - applicable but matches InnoDB), HA_CLONE_MULTI_TASK (can clone with multiple
  - threads), & HA_CLONE_RESTART (can restart after a network I/O error).
- (clone/common.cc) Enable O_DIRECT for clone file I/O if, on client if
  rocksdb_use_direct_io_for_flush_and_compaction is enabled, and on donor if
  rocksdb_use_direct_reads is enabled.
- On the donor side (clone/donor.cc), encapsulate RocksDB checkpoints in a
  rdb_checkpoint class. Each checkpoint gets its own ID, which is also a donor
  session ID. The ID is constant for rolling checkpoints.
- Subclass the common file_metadata to donor_file_metadata, chunk_metadata to
  donor_chunk_metadata, estimate_metadata to donor_estimate.
- Implement donor going through a sequence of states: NONE, ROLLING, FINAL,
  FINAL_WITH_LOGS. To support multiple clone donor sessions, implement a manager
  class that looks up the donor instance by locator.
- Implement MyRocks clone donor interface:
  - rocksdb_clone_start: handle the possible start modes:
    - HA_CLONE_MODE_START: create and register a new donor instance, register
      its master task ID.
    - HA_CLONE_MODE_ADD_TASK: lookup the existing donor instance, register a new
      task ID.
    - HA_CLONE_MODE_RESTART: lookup the existing donor instance, wait for the
      old master session to drop it by master task ID reference count, then
      process the locator applier state to reset the donor state.
  - rocksdb_clone_precopy: lookup the donor instance, create a RocksDB snapshot
    if state == NONE, otherwise send files by chunks and roll the checkpoint as
    needed.
  - rocksdb_clone_set_log_stop: add the WAL files to the clone file to copy set.
  - rocksdb_clone_copy: copy the clone files in any donor state.
  - rocksdb_clone_ack: acknowledge errors sent by client.
  - rocksdb_clone_end: for master task ID, stop accepting new worker join
    requests, wait until existing workers quit, and, in the case of restartable
    error, wait for any restarts to happen. Then decrement ref count for the
    master task ID (which will handle both the cases of restarted and not
    restarted) clone. For all other task IDs, simply unregister them.
- On the client side (clone/client.cc), have file_in_progress class to track
  each individual file progress during application, and completed_file for
  fully-applied files.
- Subclass the common file_metadata to client_file_metadata, implement its
  reading from the net buffer. Likewise for client_chunk_metadata subclassing
  chunk_metadata, and client_estimate for estimate_metadata.
- TODO client class.
- Implement MyRocks clone client interface:
  - rocksdb_clone_apply_begin: handle the possible start modes:
    - HA_CLONE_MODE_START: initialize the client instance for a new clone
    - HA_CLONE_MODE_ADD_TASK: register a new task ID for an ongoing clone
    - HA_CLONE_MODE_RESTART: restart the interrupted clone session
    - HA_CLONE_MODE_VERSION: prepare new version locator
  - rocksdb_clone_apply: if passed an error code, save it. Otherwise verify the
    passed buffer and dispatch ADD_ESTIMATE_V1, FILE_NAME_MAP_V1, &
    FILE_CHUNK_V1 packets accordingly.
  - rocksdb_clone_apply_end: for master task, wait for all other tasks to
    complete and finish the clone. For both master and non-master tasks
    unregister their IDs.
- Add rocksdb_sys_vars tests for the new clone-related system variables. They
  are included in the current patch and not the MTR one in order not to break
  the existing rocksdb_sys_vars.all_vars test.
- On instance startup, do clone fixup (clone/common.cc). It consists of:
  - Check whether InnoDB clone rollback marker is present. If yes, force
    rollback of MyRocks clone.
  - If the cloned temporary datadir contains clone-in-progress marker, assume
    half-completed clone and abort.
  - Move the cloned datadirs and WAL dirs in-place. For the datadir, do a
    directory-level rename with a save and later delete of the old datadir. For
    the WAL dir do not try to do parent directory-level renames because the
    mysqld process may not have  permissions to delete that directory. Instead,
    rename all the old files individually, move new files in, delete the old
    files.
  - For all the rename operations, handle them returning EXDEV error (a rename
    would cross a filesystem boundary) by falling back to copy and delete. As we
    expect that it is the cloned RocksDB log files that are most affected by
    this, the total copy I/O  wouldn't be large, and so the default MySQL copy
    routine my_copy is used, without extra copy throughput optimizations. Add a
    fault injection point for MTR to test this code paths.
  - Remove any RocksDB clone checkpoints found.
- On instance startup, always move WALs to the target directory if they are
  found in the rocksdb_datadir yet the rocksdb_wal_dir is different. This
  handles the case of setting rocksdb_wal_dir while instance is shutdown, both
  for clone fixup and regular operation.
- Add PFS instrumentation for all the instrumentable entities in this
  patch (mutexes, rwlocks, file I/O).

MyRocks Clone (8/n): MyRocks clone MTR tests

- Add a new MTR test suite rocksdb_clone
- Support new variable MTR clone/include/clone_command.inc:
  clone_single_threaded, setting which forces single-threaded clone operation.
- Reuse a big part of InnoDB clone tests by including their test files, with the
  exception of tests involving DDL, as concurrent DDL is not supported by
  MyRocks clone at the moment, and debug tests that adjust cloned instance LSN,
  because they desynchronize storage engines. The reused tests are: local_basic,
  local_boundary, local_dml, local_dml_auto_tune, local_dml_no_redo,
  local_insert, local_stage, monitor_progress, remote_basic,
  remote_basic_replace, remote_boundary, remote_dml, remote_dml_auto_tune,
  remote_dml_gtid, remote_dml_no_binlog, remote_dml_no_ssl, remote_dml_recovery,
  remote_dml_replace, remote_dml_ssl, remote_dml_ssl_certificate,
  remote_dml_upgrade, remote_insert, remote_monitor_progress, remote_restart,
  remote_skip_validation, remote_stage, remote_stage_restart.
- As a part of the above, add MyRocks support to local_basic and
  monitor_progress MTR tests.
- Besides the reused InnoDB clone tests, add some new MyRocks-specific tests:
- local_minimal_1_thread, which is the simplest possible local clone test.
- local_minimal_multithreaded, the simplest possible but multithreaded local
  clone test.
- local_minimal_direct_io, which is the simplest possible local clone test in
  all combinations of rocksdb_use_direct_reads and
  rocksdb_use_direct_io_for_flush_and_compaction settings, which are the
  O_DIRECT-enabling options.
- local_dml_o_direct_donor, local_dml_o_direct_client, local_dml_o_direct_both
  that reuse InnoDB local_dml test, but combined with either or both of the
  O_DIRECT-enabling options.
- local_minimal_wal_dir, which clones with various rocksdb_wal_dir settings on
  donor and client.
- local_minimal_wal_dir_copy_delete, which uses a debug injection to test the
  move-across-filesystem-boundary code path in clone startup.
- remote_minimal_direct_io_{1, 2, 3, 4} tests for all the combinations of
  O_DIRECT-enabling options for donor and client instances.
- remote_fail_apply, which uses a debug injection to fail the clone client
  during remote clone.
- remote_inplace_restart, which performs a remote in-place clone with a
  simulated transient error on the client.
- remote_kill_donor, which kill the donor instance at various points of remote
  cloning.
- rolling_checkpoint for testing various combinations of clone rolling
  checkpoint tuning options.
- three_se_basic to test a simple clone across InnoDB, MyRocks, & Blackhole
  engines.

Fix moving RocksDB WALs on startup across file system boundaries

Reviewed By: sunshine-Chun

Differential Revision: D43160162

Pulled By: hermanlee

fbshipit-source-id: 5358a57

--------------------------------------------------------------------------------

Rdb clone fixes (facebook#1271)

Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 2245706

--------------------------------------------------------------------------------

Clone: fix race condition that may skip some SST files (facebook#1274)

Summary:
A donor thread in the main copy loop took a file to send, released the donor state mutex, and then opened the file. If an ENOENT was received at this point, it was assumed that this was a stale SST file from an older checkpoint that was rolled since.

Because the donor state mutex was released between taking of the file and opening it, the following race was possible:
1) Thread 1 takes the file
2) Thread 2 decides to roll the checkpoint, the old checkpoint is deleted 3) Thread 1 tries to open the file, gets ENOENT
4) Thread 2 creates the new checkpoint, the file re-appears, but it's too late.

Rolling the checkpoint in a donor state mutex critical section is a possible fix, but such section would do a lot of I/O, serializing the parallel threads. Instead, fix by taking the file and opening it in the same critical section.

Pull Request resolved: facebook#1274

Reviewed By: sunshine-Chun

Differential Revision: D43629546

Pulled By: hermanlee

fbshipit-source-id: 6e7f315

--------------------------------------------------------------------------------

MyRocks clone: add missing donor mutex unlocks on error exits (facebook#1317)

Summary: Pull Request resolved: facebook#1317

Differential Revision: D46564079

fbshipit-source-id: 6fc1a4f
inikep pushed a commit to inikep/mysql-5.6 that referenced this issue Jun 23, 2023
Summary: Pull Request resolved: facebook#1250

MyRocks Clone (1/n): make InnoDB clone MTR tests SE-agnostic

The MyRocks clone MTR tests will reuse the InnoDB tests by including them.
Adjust the tests so that they work this way, i.e. make them storage
engine-agnostic by checking the default_storage_engine system variable and
guarding InnoDB-specific tests this way. This includes the tests that
artificially desync the InnoDB redo log position.

MyRocks Clone (2/n): fix bug 109919 (Clone plugin does not support arbitrary
length O_DIRECT files)

The existing clone plugin I/O routines handle O_DIRECT files with the assumption
that their sizes are always a multiple of a block size. This is too restrictive
because in general O_DIRECT files can be of any size and MyRocks files actually are.

Add a flag to Ha_clone_file to support arbitrary file lengths under O_DIRECT:
o_direct_uneven_file_size. Pass it to clone_client.h/cc
Client::get_aligned_buffer to allocate the buffer that bumps the length to the
next CLONE_OS_ALIGN multiple if it's not already one. Introduce a
pad_for_o_direct helper that performs this calculation for the passed length as
needed.

For writes, if O_DIRECT is enabled and the buffer length is not a multiple of
CLONE_OS_ALIGN, pad it with zeros to the next multiple. Then check the actual
bytes written, and if any padding bytes were actually written, call my_chsize to
truncate the file to the needed size.

MyRocks Clone (3/n): fix bug 109920 (SEs cannot add to clone data size estimates)

In the current implementation, once the first SE sets the data size estimate for
a particular stage, there is no way for other SEs to add to it. Address this by
adding Ha_clone_cbk::add_to_data_size_estimate method.

MyRocks Clone (4/n): fix bug 109921 (Cannot clone with synchronization across SEs)

The existing clone implementation synchronizes InnoDB and binlog directly. There
is no way to make other storage engines join this synchronization, thus replace
it with synchronizing by a performance_schema.log_status query.

- Add table_log_status::get_row method to allow doing the log_status query from
  inside the server.
- Handlerton: introduce two new methods: clone_precopy - currently unused, which
  will handle any non-consistent-snapshot copying before the main consistent
  copy, and set_log_stop - which provides the SE the clone stop position
  information.
- Plugin: introduce Ha_clone_common_cbk, which derives from Ha_clone_cbk, and
  from which derive all the concrete classes: Client_Cbk, Local_Cbk &
  Server_Cbk.
- Ha_clone_common_cbk contains two methods: precopy and synchronize_engines. The
  former calls precopy for all the participating clone SE handlertons, if
  implemented. The latter, only to be called from the main clone thread,
  performs the performance_schema.LOG_STATUS query, splits its result per-SE and
  calls handlerton set_log_stop for all SEs.
- The pfs.log_status query still must be called by InnoDB at a certain point in
  clone. That means that InnoDB is still a special SE in that it must always
  come first in the clone locator vectors. This will also handle donor waits on
  clone restarts, as that is implemented in InnoDB-specific code. To ensure
  this, add a helper clone_hton.cc:make_innodb_first which shuffles the vectors
  as needed. Call it in all the clone-starting code paths, add asserts that
  InnoDB comes first.
- Since hton_clone_copy will call into InnoDB clone copy, which at certain point
  will synchronize engines, calling into the plugin again, and the calling into
  SEs again, introduce a backup locator vector Ha_clone_cbk::all_locators, and
  its setters/getters set_all_locators, reset_all_locators, get_all_locators.
  Use them in the re-entrant hton_clone_copy code.
- Introduce a new InnoDB clone state named SST COPY, which calls precopy and has
  zero files and chunks set to copy in the regular InnoDB clone way, i.e. all it
  does it the precopy.
- The redo log archiving keeps running throughout the precopy stage.
- On stage transition from the precopy to the redo log copy. call synchronize.
- Implement innodb_clone_set_log_stop by parsing out the desired clone stop LSN
  from the passed JSON from the log_status query.
- Make InnoDB redo log archiving stop at that LSN instead of on reaching the end
  of redo log.
- Support stopping redo log archiving in a middle of a log block. As the old
  implementation may encounter this case only at the end of the redo log, it
  took that block from the log buffer in memory. This no longer works with the
  new implementation, where the needed block might reside on disk only. Address
  this by forcing the whole block containing the target LSN to disk, reading it
  from there, and patching its memory copy to stop at the required LSN before
  giving it for the clone to copy.
- To implement redo log block reading on a running server, adapt crash recovery
  function recv_read_log_seg to work both for crash recovery and on a running
  server. This is similar to what Percona Server does for its changed page
  tracking feature.
- Make sure not to archive not-yet-written log: pfs.log_status query gets the
  log system LSN, which is the value of highest log reservation. Thus in an
  archiving iteration, archive up to the lower values of log archiving LSN, and
  write_lsn. Set the log archiving LSN to LSN_MAX at clone initialization, so
  that clone archiving may proceed until the stop LSN is set.
- The above implementation results in cloned LSN != log end LSN for the cloned
  instance, thus adjust crash recovery to work with that. The existing crash
  recovery throws away any crash-recovered metadata and reads its snapshot from
  the data files, whereas the snapshot was not written in any consistency with
  the clone LSN. Fix by no longer throwing away the crash-recovered InnoDB
  dynamic metadata and by not re-reading the the last metadata snapshot. This
  forces also not to re-serialize the metadata in the crash recovery as that
  writes redo log which is to be avoided during clone crash recovery.
- Additionally, at the start of the redo log copy the old implementation
  requested all InnoDB updates to be flushed to disk immediately, to avoid clone
  recovery generating new redo log. This is also disabled as it becomes
  redundant with the fix. This issue is similar to Percona XtraBackup
  https://jira.percona.com/browse/PXB-1634
- Disable clone.local_xa test because the new synchronization no longer blocks
  on XA transactions.
- Re-record two clone-related tests in the component_keyring_file MTR suite.
- Temporarily workaround the MyRocks bug where log_status query hangs if RocksDB
  file deletions were not disabled first.

Note that this change breaks some of the clone-related group replication MTR
tests. As they are not tested in fb-mysql and have been already broken, this
patch makes no effort to fix them.

MyRocks Clone (5/n): fix bug 109922 (SEs do not coordinate clone rollbacks on
instance startup)

If InnoDB clone decides to rollback during its startup, other storage engines
are not aware of this and proceed as usual, resulting in inconsistent instance
between storage engines. Ideally this should be fixed through a 2PC-like design
where all the SEs vote to rollback the clone or to continue during the startup,
but for MyRocks clone a simpler solution suffices: make InnoDB write a special
marker file if it decides to rollback on clone startup.

MyRocks Clone (6/n): on remote clone error, always send COM_ACK to InnoDB too

In the case of multithreaded clone, the waits for the donor threads for the
current clone stage to complete are implemented fully in InnoDB, even though
that, and error handling, should be plugin-level features. With two storage
engines participating in the clone this allows the situation where the other
storage engine client reports an error with COM_ACK to one donor thread, while
some other threads are in InnoDB clone state change waits. Those waits can abort
if InnoDB clone gets an error from the client, but they know nothing about other
SE errors, and will be aborted only by timeouts.

As a minimal fix that does not involve pulling the whole error handling and
state change waiting logic to the plugin layer, make the clone client always
send an error-containing COM_ACK to InnoDB too, whenever it sends one to another
storage engine.

MyRocks Clone (7/n): MyRocks clone implementation

- Cleanup any stale RocksDB checkpoint directories on server shutdown too.

- For MyRocks clone network protocol, define four packet types (clone/common.h):
  - LOCATOR_V1: MyRocks-specific clone locator, consisting of a monotonically
    increasing clone session ID for the donor with an optional applier
    state (an array of file IDs and applied offsets) for restarts, or a zero
    in which case it is a version check empty locator used for handshakes.
  - FILE_NAME_MAP_V1: establishes a relation between a file ID, its name and
    target size.
  - FILE_CHUNK_V1: cloned file chunk data
  - ADD_ESTIMATE_V1: how many bytes should be added to the estimated clone data
    size.
- All packet types share the common package header, which stores the version,
  the payload length without the header, and the packet type.
- (clone/common.h): classes locator, file_metadata, chunk_metadata, &
  estimate_metadata correspond to the four packet types. They are subclassed on
  the donor side to add serialization routines and other donor-specific code and
  on the client side to add deserialization routines and other client-specific
  code. Re. serialization/deserialization, an exception is the locator, whose
  routines are implemented in the common code, because it may be passed both
  - ways.
- (clone/common.h): Introduce a session class that handles the task ID set and
  errors, which is common for both donor and client. For errors, allow saving
  one error, with an optional associated path and errno, that can be set by any
  thread and later reported to the clone plugin.
- (clone/common.cc) Setup MyRocks clone capabilities as HA_CLONE_HYBRID (not
  - applicable but matches InnoDB), HA_CLONE_MULTI_TASK (can clone with multiple
  - threads), & HA_CLONE_RESTART (can restart after a network I/O error).
- (clone/common.cc) Enable O_DIRECT for clone file I/O if, on client if
  rocksdb_use_direct_io_for_flush_and_compaction is enabled, and on donor if
  rocksdb_use_direct_reads is enabled.
- On the donor side (clone/donor.cc), encapsulate RocksDB checkpoints in a
  rdb_checkpoint class. Each checkpoint gets its own ID, which is also a donor
  session ID. The ID is constant for rolling checkpoints.
- Subclass the common file_metadata to donor_file_metadata, chunk_metadata to
  donor_chunk_metadata, estimate_metadata to donor_estimate.
- Implement donor going through a sequence of states: NONE, ROLLING, FINAL,
  FINAL_WITH_LOGS. To support multiple clone donor sessions, implement a manager
  class that looks up the donor instance by locator.
- Implement MyRocks clone donor interface:
  - rocksdb_clone_start: handle the possible start modes:
    - HA_CLONE_MODE_START: create and register a new donor instance, register
      its master task ID.
    - HA_CLONE_MODE_ADD_TASK: lookup the existing donor instance, register a new
      task ID.
    - HA_CLONE_MODE_RESTART: lookup the existing donor instance, wait for the
      old master session to drop it by master task ID reference count, then
      process the locator applier state to reset the donor state.
  - rocksdb_clone_precopy: lookup the donor instance, create a RocksDB snapshot
    if state == NONE, otherwise send files by chunks and roll the checkpoint as
    needed.
  - rocksdb_clone_set_log_stop: add the WAL files to the clone file to copy set.
  - rocksdb_clone_copy: copy the clone files in any donor state.
  - rocksdb_clone_ack: acknowledge errors sent by client.
  - rocksdb_clone_end: for master task ID, stop accepting new worker join
    requests, wait until existing workers quit, and, in the case of restartable
    error, wait for any restarts to happen. Then decrement ref count for the
    master task ID (which will handle both the cases of restarted and not
    restarted) clone. For all other task IDs, simply unregister them.
- On the client side (clone/client.cc), have file_in_progress class to track
  each individual file progress during application, and completed_file for
  fully-applied files.
- Subclass the common file_metadata to client_file_metadata, implement its
  reading from the net buffer. Likewise for client_chunk_metadata subclassing
  chunk_metadata, and client_estimate for estimate_metadata.
- TODO client class.
- Implement MyRocks clone client interface:
  - rocksdb_clone_apply_begin: handle the possible start modes:
    - HA_CLONE_MODE_START: initialize the client instance for a new clone
    - HA_CLONE_MODE_ADD_TASK: register a new task ID for an ongoing clone
    - HA_CLONE_MODE_RESTART: restart the interrupted clone session
    - HA_CLONE_MODE_VERSION: prepare new version locator
  - rocksdb_clone_apply: if passed an error code, save it. Otherwise verify the
    passed buffer and dispatch ADD_ESTIMATE_V1, FILE_NAME_MAP_V1, &
    FILE_CHUNK_V1 packets accordingly.
  - rocksdb_clone_apply_end: for master task, wait for all other tasks to
    complete and finish the clone. For both master and non-master tasks
    unregister their IDs.
- Add rocksdb_sys_vars tests for the new clone-related system variables. They
  are included in the current patch and not the MTR one in order not to break
  the existing rocksdb_sys_vars.all_vars test.
- On instance startup, do clone fixup (clone/common.cc). It consists of:
  - Check whether InnoDB clone rollback marker is present. If yes, force
    rollback of MyRocks clone.
  - If the cloned temporary datadir contains clone-in-progress marker, assume
    half-completed clone and abort.
  - Move the cloned datadirs and WAL dirs in-place. For the datadir, do a
    directory-level rename with a save and later delete of the old datadir. For
    the WAL dir do not try to do parent directory-level renames because the
    mysqld process may not have  permissions to delete that directory. Instead,
    rename all the old files individually, move new files in, delete the old
    files.
  - For all the rename operations, handle them returning EXDEV error (a rename
    would cross a filesystem boundary) by falling back to copy and delete. As we
    expect that it is the cloned RocksDB log files that are most affected by
    this, the total copy I/O  wouldn't be large, and so the default MySQL copy
    routine my_copy is used, without extra copy throughput optimizations. Add a
    fault injection point for MTR to test this code paths.
  - Remove any RocksDB clone checkpoints found.
- On instance startup, always move WALs to the target directory if they are
  found in the rocksdb_datadir yet the rocksdb_wal_dir is different. This
  handles the case of setting rocksdb_wal_dir while instance is shutdown, both
  for clone fixup and regular operation.
- Add PFS instrumentation for all the instrumentable entities in this
  patch (mutexes, rwlocks, file I/O).

MyRocks Clone (8/n): MyRocks clone MTR tests

- Add a new MTR test suite rocksdb_clone
- Support new variable MTR clone/include/clone_command.inc:
  clone_single_threaded, setting which forces single-threaded clone operation.
- Reuse a big part of InnoDB clone tests by including their test files, with the
  exception of tests involving DDL, as concurrent DDL is not supported by
  MyRocks clone at the moment, and debug tests that adjust cloned instance LSN,
  because they desynchronize storage engines. The reused tests are: local_basic,
  local_boundary, local_dml, local_dml_auto_tune, local_dml_no_redo,
  local_insert, local_stage, monitor_progress, remote_basic,
  remote_basic_replace, remote_boundary, remote_dml, remote_dml_auto_tune,
  remote_dml_gtid, remote_dml_no_binlog, remote_dml_no_ssl, remote_dml_recovery,
  remote_dml_replace, remote_dml_ssl, remote_dml_ssl_certificate,
  remote_dml_upgrade, remote_insert, remote_monitor_progress, remote_restart,
  remote_skip_validation, remote_stage, remote_stage_restart.
- As a part of the above, add MyRocks support to local_basic and
  monitor_progress MTR tests.
- Besides the reused InnoDB clone tests, add some new MyRocks-specific tests:
- local_minimal_1_thread, which is the simplest possible local clone test.
- local_minimal_multithreaded, the simplest possible but multithreaded local
  clone test.
- local_minimal_direct_io, which is the simplest possible local clone test in
  all combinations of rocksdb_use_direct_reads and
  rocksdb_use_direct_io_for_flush_and_compaction settings, which are the
  O_DIRECT-enabling options.
- local_dml_o_direct_donor, local_dml_o_direct_client, local_dml_o_direct_both
  that reuse InnoDB local_dml test, but combined with either or both of the
  O_DIRECT-enabling options.
- local_minimal_wal_dir, which clones with various rocksdb_wal_dir settings on
  donor and client.
- local_minimal_wal_dir_copy_delete, which uses a debug injection to test the
  move-across-filesystem-boundary code path in clone startup.
- remote_minimal_direct_io_{1, 2, 3, 4} tests for all the combinations of
  O_DIRECT-enabling options for donor and client instances.
- remote_fail_apply, which uses a debug injection to fail the clone client
  during remote clone.
- remote_inplace_restart, which performs a remote in-place clone with a
  simulated transient error on the client.
- remote_kill_donor, which kill the donor instance at various points of remote
  cloning.
- rolling_checkpoint for testing various combinations of clone rolling
  checkpoint tuning options.
- three_se_basic to test a simple clone across InnoDB, MyRocks, & Blackhole
  engines.

Fix moving RocksDB WALs on startup across file system boundaries

Reviewed By: sunshine-Chun

Differential Revision: D43160162

Pulled By: hermanlee

fbshipit-source-id: 5358a57

--------------------------------------------------------------------------------

Rdb clone fixes (facebook#1271)

Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 2245706

--------------------------------------------------------------------------------

Clone: fix race condition that may skip some SST files (facebook#1274)

Summary:
A donor thread in the main copy loop took a file to send, released the donor state mutex, and then opened the file. If an ENOENT was received at this point, it was assumed that this was a stale SST file from an older checkpoint that was rolled since.

Because the donor state mutex was released between taking of the file and opening it, the following race was possible:
1) Thread 1 takes the file
2) Thread 2 decides to roll the checkpoint, the old checkpoint is deleted 3) Thread 1 tries to open the file, gets ENOENT
4) Thread 2 creates the new checkpoint, the file re-appears, but it's too late.

Rolling the checkpoint in a donor state mutex critical section is a possible fix, but such section would do a lot of I/O, serializing the parallel threads. Instead, fix by taking the file and opening it in the same critical section.

Pull Request resolved: facebook#1274

Reviewed By: sunshine-Chun

Differential Revision: D43629546

Pulled By: hermanlee

fbshipit-source-id: 6e7f315

--------------------------------------------------------------------------------

MyRocks clone: add missing donor mutex unlocks on error exits (facebook#1317)

Summary: Pull Request resolved: facebook#1317

Differential Revision: D46564079

fbshipit-source-id: 6fc1a4f
inikep pushed a commit to inikep/percona-server that referenced this issue Jun 27, 2023
Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook/mysql-5.6#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
inikep pushed a commit to inikep/percona-server that referenced this issue Jun 29, 2023
Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook/mysql-5.6#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
inikep pushed a commit to inikep/percona-server that referenced this issue Jul 18, 2023
Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook/mysql-5.6#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
inikep pushed a commit to inikep/percona-server that referenced this issue Jul 19, 2023
Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook/mysql-5.6#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
inikep pushed a commit to inikep/percona-server that referenced this issue Jul 24, 2023
Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook/mysql-5.6#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
inikep pushed a commit to inikep/percona-server that referenced this issue Jul 25, 2023
Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook/mysql-5.6#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
inikep pushed a commit to inikep/percona-server that referenced this issue Oct 26, 2023
Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook/mysql-5.6#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
oleksandr-kachan pushed a commit to oleksandr-kachan/percona-server that referenced this issue Nov 1, 2023
Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook/mysql-5.6#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
oleksandr-kachan pushed a commit to oleksandr-kachan/percona-server that referenced this issue Nov 10, 2023
Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook/mysql-5.6#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
inikep pushed a commit to inikep/percona-server that referenced this issue Jan 17, 2024
Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook/mysql-5.6#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
inikep pushed a commit to inikep/percona-server that referenced this issue Jan 17, 2024
Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook/mysql-5.6#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
oleksandr-kachan pushed a commit to oleksandr-kachan/percona-server that referenced this issue Jan 18, 2024
Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook/mysql-5.6#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
oleksandr-kachan pushed a commit to oleksandr-kachan/percona-server that referenced this issue Jan 21, 2024
Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook/mysql-5.6#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
oleksandr-kachan pushed a commit to oleksandr-kachan/percona-server that referenced this issue Jan 26, 2024
Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook/mysql-5.6#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
oleksandr-kachan pushed a commit to oleksandr-kachan/percona-server that referenced this issue Jan 30, 2024
Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook/mysql-5.6#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
oleksandr-kachan pushed a commit to oleksandr-kachan/percona-server that referenced this issue Apr 12, 2024
Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook/mysql-5.6#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
oleksandr-kachan pushed a commit to oleksandr-kachan/percona-server that referenced this issue Apr 12, 2024
Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook/mysql-5.6#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
oleksandr-kachan pushed a commit to oleksandr-kachan/percona-server that referenced this issue Apr 12, 2024
Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook/mysql-5.6#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
oleksandr-kachan pushed a commit to oleksandr-kachan/percona-server that referenced this issue Apr 15, 2024
Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook/mysql-5.6#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
oleksandr-kachan pushed a commit to oleksandr-kachan/percona-server that referenced this issue Apr 15, 2024
Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook/mysql-5.6#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
oleksandr-kachan pushed a commit to oleksandr-kachan/percona-server that referenced this issue Apr 15, 2024
Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook/mysql-5.6#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
oleksandr-kachan pushed a commit to oleksandr-kachan/percona-server that referenced this issue Apr 16, 2024
Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook/mysql-5.6#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
oleksandr-kachan pushed a commit to oleksandr-kachan/percona-server that referenced this issue Apr 16, 2024
Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook/mysql-5.6#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
oleksandr-kachan pushed a commit to oleksandr-kachan/percona-server that referenced this issue Apr 16, 2024
Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook/mysql-5.6#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
oleksandr-kachan pushed a commit to oleksandr-kachan/percona-server that referenced this issue Apr 16, 2024
Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook/mysql-5.6#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
oleksandr-kachan pushed a commit to oleksandr-kachan/percona-server that referenced this issue Apr 16, 2024
Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook/mysql-5.6#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
oleksandr-kachan pushed a commit to oleksandr-kachan/percona-server that referenced this issue Apr 16, 2024
Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook/mysql-5.6#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
oleksandr-kachan pushed a commit to oleksandr-kachan/percona-server that referenced this issue Apr 16, 2024
Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook/mysql-5.6#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
oleksandr-kachan pushed a commit to oleksandr-kachan/percona-server that referenced this issue Apr 18, 2024
Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook/mysql-5.6#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
oleksandr-kachan pushed a commit to oleksandr-kachan/percona-server that referenced this issue Apr 29, 2024
Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook/mysql-5.6#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook/mysql-5.6#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

fbshipit-source-id: 22457061c4588df8c0b10c4f29cf3956d68a6718
inikep pushed a commit to inikep/mysql-5.6 that referenced this issue May 2, 2024
Summary: Pull Request resolved: facebook#1250

MyRocks Clone (1/n): make InnoDB clone MTR tests SE-agnostic

The MyRocks clone MTR tests will reuse the InnoDB tests by including them.
Adjust the tests so that they work this way, i.e. make them storage
engine-agnostic by checking the default_storage_engine system variable and
guarding InnoDB-specific tests this way. This includes the tests that
artificially desync the InnoDB redo log position.

MyRocks Clone (2/n): fix bug 109919 (Clone plugin does not support arbitrary
length O_DIRECT files)

The existing clone plugin I/O routines handle O_DIRECT files with the assumption
that their sizes are always a multiple of a block size. This is too restrictive
because in general O_DIRECT files can be of any size and MyRocks files actually are.

Add a flag to Ha_clone_file to support arbitrary file lengths under O_DIRECT:
o_direct_uneven_file_size. Pass it to clone_client.h/cc
Client::get_aligned_buffer to allocate the buffer that bumps the length to the
next CLONE_OS_ALIGN multiple if it's not already one. Introduce a
pad_for_o_direct helper that performs this calculation for the passed length as
needed.

For writes, if O_DIRECT is enabled and the buffer length is not a multiple of
CLONE_OS_ALIGN, pad it with zeros to the next multiple. Then check the actual
bytes written, and if any padding bytes were actually written, call my_chsize to
truncate the file to the needed size.

MyRocks Clone (3/n): fix bug 109920 (SEs cannot add to clone data size estimates)

In the current implementation, once the first SE sets the data size estimate for
a particular stage, there is no way for other SEs to add to it. Address this by
adding Ha_clone_cbk::add_to_data_size_estimate method.

MyRocks Clone (4/n): fix bug 109921 (Cannot clone with synchronization across SEs)

The existing clone implementation synchronizes InnoDB and binlog directly. There
is no way to make other storage engines join this synchronization, thus replace
it with synchronizing by a performance_schema.log_status query.

- Add table_log_status::get_row method to allow doing the log_status query from
  inside the server.
- Handlerton: introduce two new methods: clone_precopy - currently unused, which
  will handle any non-consistent-snapshot copying before the main consistent
  copy, and set_log_stop - which provides the SE the clone stop position
  information.
- Plugin: introduce Ha_clone_common_cbk, which derives from Ha_clone_cbk, and
  from which derive all the concrete classes: Client_Cbk, Local_Cbk &
  Server_Cbk.
- Ha_clone_common_cbk contains two methods: precopy and synchronize_engines. The
  former calls precopy for all the participating clone SE handlertons, if
  implemented. The latter, only to be called from the main clone thread,
  performs the performance_schema.LOG_STATUS query, splits its result per-SE and
  calls handlerton set_log_stop for all SEs.
- The pfs.log_status query still must be called by InnoDB at a certain point in
  clone. That means that InnoDB is still a special SE in that it must always
  come first in the clone locator vectors. This will also handle donor waits on
  clone restarts, as that is implemented in InnoDB-specific code. To ensure
  this, add a helper clone_hton.cc:make_innodb_first which shuffles the vectors
  as needed. Call it in all the clone-starting code paths, add asserts that
  InnoDB comes first.
- Since hton_clone_copy will call into InnoDB clone copy, which at certain point
  will synchronize engines, calling into the plugin again, and the calling into
  SEs again, introduce a backup locator vector Ha_clone_cbk::all_locators, and
  its setters/getters set_all_locators, reset_all_locators, get_all_locators.
  Use them in the re-entrant hton_clone_copy code.
- Introduce a new InnoDB clone state named SST COPY, which calls precopy and has
  zero files and chunks set to copy in the regular InnoDB clone way, i.e. all it
  does it the precopy.
- The redo log archiving keeps running throughout the precopy stage.
- On stage transition from the precopy to the redo log copy. call synchronize.
- Implement innodb_clone_set_log_stop by parsing out the desired clone stop LSN
  from the passed JSON from the log_status query.
- Make InnoDB redo log archiving stop at that LSN instead of on reaching the end
  of redo log.
- Support stopping redo log archiving in a middle of a log block. As the old
  implementation may encounter this case only at the end of the redo log, it
  took that block from the log buffer in memory. This no longer works with the
  new implementation, where the needed block might reside on disk only. Address
  this by forcing the whole block containing the target LSN to disk, reading it
  from there, and patching its memory copy to stop at the required LSN before
  giving it for the clone to copy.
- To implement redo log block reading on a running server, adapt crash recovery
  function recv_read_log_seg to work both for crash recovery and on a running
  server. This is similar to what Percona Server does for its changed page
  tracking feature.
- Make sure not to archive not-yet-written log: pfs.log_status query gets the
  log system LSN, which is the value of highest log reservation. Thus in an
  archiving iteration, archive up to the lower values of log archiving LSN, and
  write_lsn. Set the log archiving LSN to LSN_MAX at clone initialization, so
  that clone archiving may proceed until the stop LSN is set.
- The above implementation results in cloned LSN != log end LSN for the cloned
  instance, thus adjust crash recovery to work with that. The existing crash
  recovery throws away any crash-recovered metadata and reads its snapshot from
  the data files, whereas the snapshot was not written in any consistency with
  the clone LSN. Fix by no longer throwing away the crash-recovered InnoDB
  dynamic metadata and by not re-reading the the last metadata snapshot. This
  forces also not to re-serialize the metadata in the crash recovery as that
  writes redo log which is to be avoided during clone crash recovery.
- Additionally, at the start of the redo log copy the old implementation
  requested all InnoDB updates to be flushed to disk immediately, to avoid clone
  recovery generating new redo log. This is also disabled as it becomes
  redundant with the fix. This issue is similar to Percona XtraBackup
  https://jira.percona.com/browse/PXB-1634
- Disable clone.local_xa test because the new synchronization no longer blocks
  on XA transactions.
- Re-record two clone-related tests in the component_keyring_file MTR suite.
- Temporarily workaround the MyRocks bug where log_status query hangs if RocksDB
  file deletions were not disabled first.

Note that this change breaks some of the clone-related group replication MTR
tests. As they are not tested in fb-mysql and have been already broken, this
patch makes no effort to fix them.

MyRocks Clone (5/n): fix bug 109922 (SEs do not coordinate clone rollbacks on
instance startup)

If InnoDB clone decides to rollback during its startup, other storage engines
are not aware of this and proceed as usual, resulting in inconsistent instance
between storage engines. Ideally this should be fixed through a 2PC-like design
where all the SEs vote to rollback the clone or to continue during the startup,
but for MyRocks clone a simpler solution suffices: make InnoDB write a special
marker file if it decides to rollback on clone startup.

MyRocks Clone (6/n): on remote clone error, always send COM_ACK to InnoDB too

In the case of multithreaded clone, the waits for the donor threads for the
current clone stage to complete are implemented fully in InnoDB, even though
that, and error handling, should be plugin-level features. With two storage
engines participating in the clone this allows the situation where the other
storage engine client reports an error with COM_ACK to one donor thread, while
some other threads are in InnoDB clone state change waits. Those waits can abort
if InnoDB clone gets an error from the client, but they know nothing about other
SE errors, and will be aborted only by timeouts.

As a minimal fix that does not involve pulling the whole error handling and
state change waiting logic to the plugin layer, make the clone client always
send an error-containing COM_ACK to InnoDB too, whenever it sends one to another
storage engine.

MyRocks Clone (7/n): MyRocks clone implementation

- Cleanup any stale RocksDB checkpoint directories on server shutdown too.

- For MyRocks clone network protocol, define four packet types (clone/common.h):
  - LOCATOR_V1: MyRocks-specific clone locator, consisting of a monotonically
    increasing clone session ID for the donor with an optional applier
    state (an array of file IDs and applied offsets) for restarts, or a zero
    in which case it is a version check empty locator used for handshakes.
  - FILE_NAME_MAP_V1: establishes a relation between a file ID, its name and
    target size.
  - FILE_CHUNK_V1: cloned file chunk data
  - ADD_ESTIMATE_V1: how many bytes should be added to the estimated clone data
    size.
- All packet types share the common package header, which stores the version,
  the payload length without the header, and the packet type.
- (clone/common.h): classes locator, file_metadata, chunk_metadata, &
  estimate_metadata correspond to the four packet types. They are subclassed on
  the donor side to add serialization routines and other donor-specific code and
  on the client side to add deserialization routines and other client-specific
  code. Re. serialization/deserialization, an exception is the locator, whose
  routines are implemented in the common code, because it may be passed both
  - ways.
- (clone/common.h): Introduce a session class that handles the task ID set and
  errors, which is common for both donor and client. For errors, allow saving
  one error, with an optional associated path and errno, that can be set by any
  thread and later reported to the clone plugin.
- (clone/common.cc) Setup MyRocks clone capabilities as HA_CLONE_HYBRID (not
  - applicable but matches InnoDB), HA_CLONE_MULTI_TASK (can clone with multiple
  - threads), & HA_CLONE_RESTART (can restart after a network I/O error).
- (clone/common.cc) Enable O_DIRECT for clone file I/O if, on client if
  rocksdb_use_direct_io_for_flush_and_compaction is enabled, and on donor if
  rocksdb_use_direct_reads is enabled.
- On the donor side (clone/donor.cc), encapsulate RocksDB checkpoints in a
  rdb_checkpoint class. Each checkpoint gets its own ID, which is also a donor
  session ID. The ID is constant for rolling checkpoints.
- Subclass the common file_metadata to donor_file_metadata, chunk_metadata to
  donor_chunk_metadata, estimate_metadata to donor_estimate.
- Implement donor going through a sequence of states: NONE, ROLLING, FINAL,
  FINAL_WITH_LOGS. To support multiple clone donor sessions, implement a manager
  class that looks up the donor instance by locator.
- Implement MyRocks clone donor interface:
  - rocksdb_clone_start: handle the possible start modes:
    - HA_CLONE_MODE_START: create and register a new donor instance, register
      its master task ID.
    - HA_CLONE_MODE_ADD_TASK: lookup the existing donor instance, register a new
      task ID.
    - HA_CLONE_MODE_RESTART: lookup the existing donor instance, wait for the
      old master session to drop it by master task ID reference count, then
      process the locator applier state to reset the donor state.
  - rocksdb_clone_precopy: lookup the donor instance, create a RocksDB snapshot
    if state == NONE, otherwise send files by chunks and roll the checkpoint as
    needed.
  - rocksdb_clone_set_log_stop: add the WAL files to the clone file to copy set.
  - rocksdb_clone_copy: copy the clone files in any donor state.
  - rocksdb_clone_ack: acknowledge errors sent by client.
  - rocksdb_clone_end: for master task ID, stop accepting new worker join
    requests, wait until existing workers quit, and, in the case of restartable
    error, wait for any restarts to happen. Then decrement ref count for the
    master task ID (which will handle both the cases of restarted and not
    restarted) clone. For all other task IDs, simply unregister them.
- On the client side (clone/client.cc), have file_in_progress class to track
  each individual file progress during application, and completed_file for
  fully-applied files.
- Subclass the common file_metadata to client_file_metadata, implement its
  reading from the net buffer. Likewise for client_chunk_metadata subclassing
  chunk_metadata, and client_estimate for estimate_metadata.
- TODO client class.
- Implement MyRocks clone client interface:
  - rocksdb_clone_apply_begin: handle the possible start modes:
    - HA_CLONE_MODE_START: initialize the client instance for a new clone
    - HA_CLONE_MODE_ADD_TASK: register a new task ID for an ongoing clone
    - HA_CLONE_MODE_RESTART: restart the interrupted clone session
    - HA_CLONE_MODE_VERSION: prepare new version locator
  - rocksdb_clone_apply: if passed an error code, save it. Otherwise verify the
    passed buffer and dispatch ADD_ESTIMATE_V1, FILE_NAME_MAP_V1, &
    FILE_CHUNK_V1 packets accordingly.
  - rocksdb_clone_apply_end: for master task, wait for all other tasks to
    complete and finish the clone. For both master and non-master tasks
    unregister their IDs.
- Add rocksdb_sys_vars tests for the new clone-related system variables. They
  are included in the current patch and not the MTR one in order not to break
  the existing rocksdb_sys_vars.all_vars test.
- On instance startup, do clone fixup (clone/common.cc). It consists of:
  - Check whether InnoDB clone rollback marker is present. If yes, force
    rollback of MyRocks clone.
  - If the cloned temporary datadir contains clone-in-progress marker, assume
    half-completed clone and abort.
  - Move the cloned datadirs and WAL dirs in-place. For the datadir, do a
    directory-level rename with a save and later delete of the old datadir. For
    the WAL dir do not try to do parent directory-level renames because the
    mysqld process may not have  permissions to delete that directory. Instead,
    rename all the old files individually, move new files in, delete the old
    files.
  - For all the rename operations, handle them returning EXDEV error (a rename
    would cross a filesystem boundary) by falling back to copy and delete. As we
    expect that it is the cloned RocksDB log files that are most affected by
    this, the total copy I/O  wouldn't be large, and so the default MySQL copy
    routine my_copy is used, without extra copy throughput optimizations. Add a
    fault injection point for MTR to test this code paths.
  - Remove any RocksDB clone checkpoints found.
- On instance startup, always move WALs to the target directory if they are
  found in the rocksdb_datadir yet the rocksdb_wal_dir is different. This
  handles the case of setting rocksdb_wal_dir while instance is shutdown, both
  for clone fixup and regular operation.
- Add PFS instrumentation for all the instrumentable entities in this
  patch (mutexes, rwlocks, file I/O).

MyRocks Clone (8/n): MyRocks clone MTR tests

- Add a new MTR test suite rocksdb_clone
- Support new variable MTR clone/include/clone_command.inc:
  clone_single_threaded, setting which forces single-threaded clone operation.
- Reuse a big part of InnoDB clone tests by including their test files, with the
  exception of tests involving DDL, as concurrent DDL is not supported by
  MyRocks clone at the moment, and debug tests that adjust cloned instance LSN,
  because they desynchronize storage engines. The reused tests are: local_basic,
  local_boundary, local_dml, local_dml_auto_tune, local_dml_no_redo,
  local_insert, local_stage, monitor_progress, remote_basic,
  remote_basic_replace, remote_boundary, remote_dml, remote_dml_auto_tune,
  remote_dml_gtid, remote_dml_no_binlog, remote_dml_no_ssl, remote_dml_recovery,
  remote_dml_replace, remote_dml_ssl, remote_dml_ssl_certificate,
  remote_dml_upgrade, remote_insert, remote_monitor_progress, remote_restart,
  remote_skip_validation, remote_stage, remote_stage_restart.
- As a part of the above, add MyRocks support to local_basic and
  monitor_progress MTR tests.
- Besides the reused InnoDB clone tests, add some new MyRocks-specific tests:
- local_minimal_1_thread, which is the simplest possible local clone test.
- local_minimal_multithreaded, the simplest possible but multithreaded local
  clone test.
- local_minimal_direct_io, which is the simplest possible local clone test in
  all combinations of rocksdb_use_direct_reads and
  rocksdb_use_direct_io_for_flush_and_compaction settings, which are the
  O_DIRECT-enabling options.
- local_dml_o_direct_donor, local_dml_o_direct_client, local_dml_o_direct_both
  that reuse InnoDB local_dml test, but combined with either or both of the
  O_DIRECT-enabling options.
- local_minimal_wal_dir, which clones with various rocksdb_wal_dir settings on
  donor and client.
- local_minimal_wal_dir_copy_delete, which uses a debug injection to test the
  move-across-filesystem-boundary code path in clone startup.
- remote_minimal_direct_io_{1, 2, 3, 4} tests for all the combinations of
  O_DIRECT-enabling options for donor and client instances.
- remote_fail_apply, which uses a debug injection to fail the clone client
  during remote clone.
- remote_inplace_restart, which performs a remote in-place clone with a
  simulated transient error on the client.
- remote_kill_donor, which kill the donor instance at various points of remote
  cloning.
- rolling_checkpoint for testing various combinations of clone rolling
  checkpoint tuning options.
- three_se_basic to test a simple clone across InnoDB, MyRocks, & Blackhole
  engines.

Fix moving RocksDB WALs on startup across file system boundaries

Reviewed By: sunshine-Chun

Differential Revision: D43160162

Pulled By: hermanlee

--------------------------------------------------------------------------------

Rdb clone fixes (facebook#1271)

Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

--------------------------------------------------------------------------------

Clone: fix race condition that may skip some SST files (facebook#1274)

Summary:
A donor thread in the main copy loop took a file to send, released the donor state mutex, and then opened the file. If an ENOENT was received at this point, it was assumed that this was a stale SST file from an older checkpoint that was rolled since.

Because the donor state mutex was released between taking of the file and opening it, the following race was possible:
1) Thread 1 takes the file
2) Thread 2 decides to roll the checkpoint, the old checkpoint is deleted 3) Thread 1 tries to open the file, gets ENOENT
4) Thread 2 creates the new checkpoint, the file re-appears, but it's too late.

Rolling the checkpoint in a donor state mutex critical section is a possible fix, but such section would do a lot of I/O, serializing the parallel threads. Instead, fix by taking the file and opening it in the same critical section.

Pull Request resolved: facebook#1274

Reviewed By: sunshine-Chun

Differential Revision: D43629546

Pulled By: hermanlee

--------------------------------------------------------------------------------

MyRocks clone: add missing donor mutex unlocks on error exits (facebook#1317)

Summary: Pull Request resolved: facebook#1317

Differential Revision: D46564079
inikep pushed a commit to inikep/mysql-5.6 that referenced this issue May 7, 2024
Summary: Pull Request resolved: facebook#1250

MyRocks Clone (1/n): make InnoDB clone MTR tests SE-agnostic

The MyRocks clone MTR tests will reuse the InnoDB tests by including them.
Adjust the tests so that they work this way, i.e. make them storage
engine-agnostic by checking the default_storage_engine system variable and
guarding InnoDB-specific tests this way. This includes the tests that
artificially desync the InnoDB redo log position.

MyRocks Clone (2/n): fix bug 109919 (Clone plugin does not support arbitrary
length O_DIRECT files)

The existing clone plugin I/O routines handle O_DIRECT files with the assumption
that their sizes are always a multiple of a block size. This is too restrictive
because in general O_DIRECT files can be of any size and MyRocks files actually are.

Add a flag to Ha_clone_file to support arbitrary file lengths under O_DIRECT:
o_direct_uneven_file_size. Pass it to clone_client.h/cc
Client::get_aligned_buffer to allocate the buffer that bumps the length to the
next CLONE_OS_ALIGN multiple if it's not already one. Introduce a
pad_for_o_direct helper that performs this calculation for the passed length as
needed.

For writes, if O_DIRECT is enabled and the buffer length is not a multiple of
CLONE_OS_ALIGN, pad it with zeros to the next multiple. Then check the actual
bytes written, and if any padding bytes were actually written, call my_chsize to
truncate the file to the needed size.

MyRocks Clone (3/n): fix bug 109920 (SEs cannot add to clone data size estimates)

In the current implementation, once the first SE sets the data size estimate for
a particular stage, there is no way for other SEs to add to it. Address this by
adding Ha_clone_cbk::add_to_data_size_estimate method.

MyRocks Clone (4/n): fix bug 109921 (Cannot clone with synchronization across SEs)

The existing clone implementation synchronizes InnoDB and binlog directly. There
is no way to make other storage engines join this synchronization, thus replace
it with synchronizing by a performance_schema.log_status query.

- Add table_log_status::get_row method to allow doing the log_status query from
  inside the server.
- Handlerton: introduce two new methods: clone_precopy - currently unused, which
  will handle any non-consistent-snapshot copying before the main consistent
  copy, and set_log_stop - which provides the SE the clone stop position
  information.
- Plugin: introduce Ha_clone_common_cbk, which derives from Ha_clone_cbk, and
  from which derive all the concrete classes: Client_Cbk, Local_Cbk &
  Server_Cbk.
- Ha_clone_common_cbk contains two methods: precopy and synchronize_engines. The
  former calls precopy for all the participating clone SE handlertons, if
  implemented. The latter, only to be called from the main clone thread,
  performs the performance_schema.LOG_STATUS query, splits its result per-SE and
  calls handlerton set_log_stop for all SEs.
- The pfs.log_status query still must be called by InnoDB at a certain point in
  clone. That means that InnoDB is still a special SE in that it must always
  come first in the clone locator vectors. This will also handle donor waits on
  clone restarts, as that is implemented in InnoDB-specific code. To ensure
  this, add a helper clone_hton.cc:make_innodb_first which shuffles the vectors
  as needed. Call it in all the clone-starting code paths, add asserts that
  InnoDB comes first.
- Since hton_clone_copy will call into InnoDB clone copy, which at certain point
  will synchronize engines, calling into the plugin again, and the calling into
  SEs again, introduce a backup locator vector Ha_clone_cbk::all_locators, and
  its setters/getters set_all_locators, reset_all_locators, get_all_locators.
  Use them in the re-entrant hton_clone_copy code.
- Introduce a new InnoDB clone state named SST COPY, which calls precopy and has
  zero files and chunks set to copy in the regular InnoDB clone way, i.e. all it
  does it the precopy.
- The redo log archiving keeps running throughout the precopy stage.
- On stage transition from the precopy to the redo log copy. call synchronize.
- Implement innodb_clone_set_log_stop by parsing out the desired clone stop LSN
  from the passed JSON from the log_status query.
- Make InnoDB redo log archiving stop at that LSN instead of on reaching the end
  of redo log.
- Support stopping redo log archiving in a middle of a log block. As the old
  implementation may encounter this case only at the end of the redo log, it
  took that block from the log buffer in memory. This no longer works with the
  new implementation, where the needed block might reside on disk only. Address
  this by forcing the whole block containing the target LSN to disk, reading it
  from there, and patching its memory copy to stop at the required LSN before
  giving it for the clone to copy.
- To implement redo log block reading on a running server, adapt crash recovery
  function recv_read_log_seg to work both for crash recovery and on a running
  server. This is similar to what Percona Server does for its changed page
  tracking feature.
- Make sure not to archive not-yet-written log: pfs.log_status query gets the
  log system LSN, which is the value of highest log reservation. Thus in an
  archiving iteration, archive up to the lower values of log archiving LSN, and
  write_lsn. Set the log archiving LSN to LSN_MAX at clone initialization, so
  that clone archiving may proceed until the stop LSN is set.
- The above implementation results in cloned LSN != log end LSN for the cloned
  instance, thus adjust crash recovery to work with that. The existing crash
  recovery throws away any crash-recovered metadata and reads its snapshot from
  the data files, whereas the snapshot was not written in any consistency with
  the clone LSN. Fix by no longer throwing away the crash-recovered InnoDB
  dynamic metadata and by not re-reading the the last metadata snapshot. This
  forces also not to re-serialize the metadata in the crash recovery as that
  writes redo log which is to be avoided during clone crash recovery.
- Additionally, at the start of the redo log copy the old implementation
  requested all InnoDB updates to be flushed to disk immediately, to avoid clone
  recovery generating new redo log. This is also disabled as it becomes
  redundant with the fix. This issue is similar to Percona XtraBackup
  https://jira.percona.com/browse/PXB-1634
- Disable clone.local_xa test because the new synchronization no longer blocks
  on XA transactions.
- Re-record two clone-related tests in the component_keyring_file MTR suite.
- Temporarily workaround the MyRocks bug where log_status query hangs if RocksDB
  file deletions were not disabled first.

Note that this change breaks some of the clone-related group replication MTR
tests. As they are not tested in fb-mysql and have been already broken, this
patch makes no effort to fix them.

MyRocks Clone (5/n): fix bug 109922 (SEs do not coordinate clone rollbacks on
instance startup)

If InnoDB clone decides to rollback during its startup, other storage engines
are not aware of this and proceed as usual, resulting in inconsistent instance
between storage engines. Ideally this should be fixed through a 2PC-like design
where all the SEs vote to rollback the clone or to continue during the startup,
but for MyRocks clone a simpler solution suffices: make InnoDB write a special
marker file if it decides to rollback on clone startup.

MyRocks Clone (6/n): on remote clone error, always send COM_ACK to InnoDB too

In the case of multithreaded clone, the waits for the donor threads for the
current clone stage to complete are implemented fully in InnoDB, even though
that, and error handling, should be plugin-level features. With two storage
engines participating in the clone this allows the situation where the other
storage engine client reports an error with COM_ACK to one donor thread, while
some other threads are in InnoDB clone state change waits. Those waits can abort
if InnoDB clone gets an error from the client, but they know nothing about other
SE errors, and will be aborted only by timeouts.

As a minimal fix that does not involve pulling the whole error handling and
state change waiting logic to the plugin layer, make the clone client always
send an error-containing COM_ACK to InnoDB too, whenever it sends one to another
storage engine.

MyRocks Clone (7/n): MyRocks clone implementation

- Cleanup any stale RocksDB checkpoint directories on server shutdown too.

- For MyRocks clone network protocol, define four packet types (clone/common.h):
  - LOCATOR_V1: MyRocks-specific clone locator, consisting of a monotonically
    increasing clone session ID for the donor with an optional applier
    state (an array of file IDs and applied offsets) for restarts, or a zero
    in which case it is a version check empty locator used for handshakes.
  - FILE_NAME_MAP_V1: establishes a relation between a file ID, its name and
    target size.
  - FILE_CHUNK_V1: cloned file chunk data
  - ADD_ESTIMATE_V1: how many bytes should be added to the estimated clone data
    size.
- All packet types share the common package header, which stores the version,
  the payload length without the header, and the packet type.
- (clone/common.h): classes locator, file_metadata, chunk_metadata, &
  estimate_metadata correspond to the four packet types. They are subclassed on
  the donor side to add serialization routines and other donor-specific code and
  on the client side to add deserialization routines and other client-specific
  code. Re. serialization/deserialization, an exception is the locator, whose
  routines are implemented in the common code, because it may be passed both
  - ways.
- (clone/common.h): Introduce a session class that handles the task ID set and
  errors, which is common for both donor and client. For errors, allow saving
  one error, with an optional associated path and errno, that can be set by any
  thread and later reported to the clone plugin.
- (clone/common.cc) Setup MyRocks clone capabilities as HA_CLONE_HYBRID (not
  - applicable but matches InnoDB), HA_CLONE_MULTI_TASK (can clone with multiple
  - threads), & HA_CLONE_RESTART (can restart after a network I/O error).
- (clone/common.cc) Enable O_DIRECT for clone file I/O if, on client if
  rocksdb_use_direct_io_for_flush_and_compaction is enabled, and on donor if
  rocksdb_use_direct_reads is enabled.
- On the donor side (clone/donor.cc), encapsulate RocksDB checkpoints in a
  rdb_checkpoint class. Each checkpoint gets its own ID, which is also a donor
  session ID. The ID is constant for rolling checkpoints.
- Subclass the common file_metadata to donor_file_metadata, chunk_metadata to
  donor_chunk_metadata, estimate_metadata to donor_estimate.
- Implement donor going through a sequence of states: NONE, ROLLING, FINAL,
  FINAL_WITH_LOGS. To support multiple clone donor sessions, implement a manager
  class that looks up the donor instance by locator.
- Implement MyRocks clone donor interface:
  - rocksdb_clone_start: handle the possible start modes:
    - HA_CLONE_MODE_START: create and register a new donor instance, register
      its master task ID.
    - HA_CLONE_MODE_ADD_TASK: lookup the existing donor instance, register a new
      task ID.
    - HA_CLONE_MODE_RESTART: lookup the existing donor instance, wait for the
      old master session to drop it by master task ID reference count, then
      process the locator applier state to reset the donor state.
  - rocksdb_clone_precopy: lookup the donor instance, create a RocksDB snapshot
    if state == NONE, otherwise send files by chunks and roll the checkpoint as
    needed.
  - rocksdb_clone_set_log_stop: add the WAL files to the clone file to copy set.
  - rocksdb_clone_copy: copy the clone files in any donor state.
  - rocksdb_clone_ack: acknowledge errors sent by client.
  - rocksdb_clone_end: for master task ID, stop accepting new worker join
    requests, wait until existing workers quit, and, in the case of restartable
    error, wait for any restarts to happen. Then decrement ref count for the
    master task ID (which will handle both the cases of restarted and not
    restarted) clone. For all other task IDs, simply unregister them.
- On the client side (clone/client.cc), have file_in_progress class to track
  each individual file progress during application, and completed_file for
  fully-applied files.
- Subclass the common file_metadata to client_file_metadata, implement its
  reading from the net buffer. Likewise for client_chunk_metadata subclassing
  chunk_metadata, and client_estimate for estimate_metadata.
- TODO client class.
- Implement MyRocks clone client interface:
  - rocksdb_clone_apply_begin: handle the possible start modes:
    - HA_CLONE_MODE_START: initialize the client instance for a new clone
    - HA_CLONE_MODE_ADD_TASK: register a new task ID for an ongoing clone
    - HA_CLONE_MODE_RESTART: restart the interrupted clone session
    - HA_CLONE_MODE_VERSION: prepare new version locator
  - rocksdb_clone_apply: if passed an error code, save it. Otherwise verify the
    passed buffer and dispatch ADD_ESTIMATE_V1, FILE_NAME_MAP_V1, &
    FILE_CHUNK_V1 packets accordingly.
  - rocksdb_clone_apply_end: for master task, wait for all other tasks to
    complete and finish the clone. For both master and non-master tasks
    unregister their IDs.
- Add rocksdb_sys_vars tests for the new clone-related system variables. They
  are included in the current patch and not the MTR one in order not to break
  the existing rocksdb_sys_vars.all_vars test.
- On instance startup, do clone fixup (clone/common.cc). It consists of:
  - Check whether InnoDB clone rollback marker is present. If yes, force
    rollback of MyRocks clone.
  - If the cloned temporary datadir contains clone-in-progress marker, assume
    half-completed clone and abort.
  - Move the cloned datadirs and WAL dirs in-place. For the datadir, do a
    directory-level rename with a save and later delete of the old datadir. For
    the WAL dir do not try to do parent directory-level renames because the
    mysqld process may not have  permissions to delete that directory. Instead,
    rename all the old files individually, move new files in, delete the old
    files.
  - For all the rename operations, handle them returning EXDEV error (a rename
    would cross a filesystem boundary) by falling back to copy and delete. As we
    expect that it is the cloned RocksDB log files that are most affected by
    this, the total copy I/O  wouldn't be large, and so the default MySQL copy
    routine my_copy is used, without extra copy throughput optimizations. Add a
    fault injection point for MTR to test this code paths.
  - Remove any RocksDB clone checkpoints found.
- On instance startup, always move WALs to the target directory if they are
  found in the rocksdb_datadir yet the rocksdb_wal_dir is different. This
  handles the case of setting rocksdb_wal_dir while instance is shutdown, both
  for clone fixup and regular operation.
- Add PFS instrumentation for all the instrumentable entities in this
  patch (mutexes, rwlocks, file I/O).

MyRocks Clone (8/n): MyRocks clone MTR tests

- Add a new MTR test suite rocksdb_clone
- Support new variable MTR clone/include/clone_command.inc:
  clone_single_threaded, setting which forces single-threaded clone operation.
- Reuse a big part of InnoDB clone tests by including their test files, with the
  exception of tests involving DDL, as concurrent DDL is not supported by
  MyRocks clone at the moment, and debug tests that adjust cloned instance LSN,
  because they desynchronize storage engines. The reused tests are: local_basic,
  local_boundary, local_dml, local_dml_auto_tune, local_dml_no_redo,
  local_insert, local_stage, monitor_progress, remote_basic,
  remote_basic_replace, remote_boundary, remote_dml, remote_dml_auto_tune,
  remote_dml_gtid, remote_dml_no_binlog, remote_dml_no_ssl, remote_dml_recovery,
  remote_dml_replace, remote_dml_ssl, remote_dml_ssl_certificate,
  remote_dml_upgrade, remote_insert, remote_monitor_progress, remote_restart,
  remote_skip_validation, remote_stage, remote_stage_restart.
- As a part of the above, add MyRocks support to local_basic and
  monitor_progress MTR tests.
- Besides the reused InnoDB clone tests, add some new MyRocks-specific tests:
- local_minimal_1_thread, which is the simplest possible local clone test.
- local_minimal_multithreaded, the simplest possible but multithreaded local
  clone test.
- local_minimal_direct_io, which is the simplest possible local clone test in
  all combinations of rocksdb_use_direct_reads and
  rocksdb_use_direct_io_for_flush_and_compaction settings, which are the
  O_DIRECT-enabling options.
- local_dml_o_direct_donor, local_dml_o_direct_client, local_dml_o_direct_both
  that reuse InnoDB local_dml test, but combined with either or both of the
  O_DIRECT-enabling options.
- local_minimal_wal_dir, which clones with various rocksdb_wal_dir settings on
  donor and client.
- local_minimal_wal_dir_copy_delete, which uses a debug injection to test the
  move-across-filesystem-boundary code path in clone startup.
- remote_minimal_direct_io_{1, 2, 3, 4} tests for all the combinations of
  O_DIRECT-enabling options for donor and client instances.
- remote_fail_apply, which uses a debug injection to fail the clone client
  during remote clone.
- remote_inplace_restart, which performs a remote in-place clone with a
  simulated transient error on the client.
- remote_kill_donor, which kill the donor instance at various points of remote
  cloning.
- rolling_checkpoint for testing various combinations of clone rolling
  checkpoint tuning options.
- three_se_basic to test a simple clone across InnoDB, MyRocks, & Blackhole
  engines.

Fix moving RocksDB WALs on startup across file system boundaries

Reviewed By: sunshine-Chun

Differential Revision: D43160162

Pulled By: hermanlee

--------------------------------------------------------------------------------

Rdb clone fixes (facebook#1271)

Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

--------------------------------------------------------------------------------

Clone: fix race condition that may skip some SST files (facebook#1274)

Summary:
A donor thread in the main copy loop took a file to send, released the donor state mutex, and then opened the file. If an ENOENT was received at this point, it was assumed that this was a stale SST file from an older checkpoint that was rolled since.

Because the donor state mutex was released between taking of the file and opening it, the following race was possible:
1) Thread 1 takes the file
2) Thread 2 decides to roll the checkpoint, the old checkpoint is deleted 3) Thread 1 tries to open the file, gets ENOENT
4) Thread 2 creates the new checkpoint, the file re-appears, but it's too late.

Rolling the checkpoint in a donor state mutex critical section is a possible fix, but such section would do a lot of I/O, serializing the parallel threads. Instead, fix by taking the file and opening it in the same critical section.

Pull Request resolved: facebook#1274

Reviewed By: sunshine-Chun

Differential Revision: D43629546

Pulled By: hermanlee

--------------------------------------------------------------------------------

MyRocks clone: add missing donor mutex unlocks on error exits (facebook#1317)

Summary: Pull Request resolved: facebook#1317

Differential Revision: D46564079
inikep pushed a commit to inikep/mysql-5.6 that referenced this issue May 7, 2024
Summary: Pull Request resolved: facebook#1250

MyRocks Clone (1/n): make InnoDB clone MTR tests SE-agnostic

The MyRocks clone MTR tests will reuse the InnoDB tests by including them.
Adjust the tests so that they work this way, i.e. make them storage
engine-agnostic by checking the default_storage_engine system variable and
guarding InnoDB-specific tests this way. This includes the tests that
artificially desync the InnoDB redo log position.

MyRocks Clone (2/n): fix bug 109919 (Clone plugin does not support arbitrary
length O_DIRECT files)

The existing clone plugin I/O routines handle O_DIRECT files with the assumption
that their sizes are always a multiple of a block size. This is too restrictive
because in general O_DIRECT files can be of any size and MyRocks files actually are.

Add a flag to Ha_clone_file to support arbitrary file lengths under O_DIRECT:
o_direct_uneven_file_size. Pass it to clone_client.h/cc
Client::get_aligned_buffer to allocate the buffer that bumps the length to the
next CLONE_OS_ALIGN multiple if it's not already one. Introduce a
pad_for_o_direct helper that performs this calculation for the passed length as
needed.

For writes, if O_DIRECT is enabled and the buffer length is not a multiple of
CLONE_OS_ALIGN, pad it with zeros to the next multiple. Then check the actual
bytes written, and if any padding bytes were actually written, call my_chsize to
truncate the file to the needed size.

MyRocks Clone (3/n): fix bug 109920 (SEs cannot add to clone data size estimates)

In the current implementation, once the first SE sets the data size estimate for
a particular stage, there is no way for other SEs to add to it. Address this by
adding Ha_clone_cbk::add_to_data_size_estimate method.

MyRocks Clone (4/n): fix bug 109921 (Cannot clone with synchronization across SEs)

The existing clone implementation synchronizes InnoDB and binlog directly. There
is no way to make other storage engines join this synchronization, thus replace
it with synchronizing by a performance_schema.log_status query.

- Add table_log_status::get_row method to allow doing the log_status query from
  inside the server.
- Handlerton: introduce two new methods: clone_precopy - currently unused, which
  will handle any non-consistent-snapshot copying before the main consistent
  copy, and set_log_stop - which provides the SE the clone stop position
  information.
- Plugin: introduce Ha_clone_common_cbk, which derives from Ha_clone_cbk, and
  from which derive all the concrete classes: Client_Cbk, Local_Cbk &
  Server_Cbk.
- Ha_clone_common_cbk contains two methods: precopy and synchronize_engines. The
  former calls precopy for all the participating clone SE handlertons, if
  implemented. The latter, only to be called from the main clone thread,
  performs the performance_schema.LOG_STATUS query, splits its result per-SE and
  calls handlerton set_log_stop for all SEs.
- The pfs.log_status query still must be called by InnoDB at a certain point in
  clone. That means that InnoDB is still a special SE in that it must always
  come first in the clone locator vectors. This will also handle donor waits on
  clone restarts, as that is implemented in InnoDB-specific code. To ensure
  this, add a helper clone_hton.cc:make_innodb_first which shuffles the vectors
  as needed. Call it in all the clone-starting code paths, add asserts that
  InnoDB comes first.
- Since hton_clone_copy will call into InnoDB clone copy, which at certain point
  will synchronize engines, calling into the plugin again, and the calling into
  SEs again, introduce a backup locator vector Ha_clone_cbk::all_locators, and
  its setters/getters set_all_locators, reset_all_locators, get_all_locators.
  Use them in the re-entrant hton_clone_copy code.
- Introduce a new InnoDB clone state named SST COPY, which calls precopy and has
  zero files and chunks set to copy in the regular InnoDB clone way, i.e. all it
  does it the precopy.
- The redo log archiving keeps running throughout the precopy stage.
- On stage transition from the precopy to the redo log copy. call synchronize.
- Implement innodb_clone_set_log_stop by parsing out the desired clone stop LSN
  from the passed JSON from the log_status query.
- Make InnoDB redo log archiving stop at that LSN instead of on reaching the end
  of redo log.
- Support stopping redo log archiving in a middle of a log block. As the old
  implementation may encounter this case only at the end of the redo log, it
  took that block from the log buffer in memory. This no longer works with the
  new implementation, where the needed block might reside on disk only. Address
  this by forcing the whole block containing the target LSN to disk, reading it
  from there, and patching its memory copy to stop at the required LSN before
  giving it for the clone to copy.
- To implement redo log block reading on a running server, adapt crash recovery
  function recv_read_log_seg to work both for crash recovery and on a running
  server. This is similar to what Percona Server does for its changed page
  tracking feature.
- Make sure not to archive not-yet-written log: pfs.log_status query gets the
  log system LSN, which is the value of highest log reservation. Thus in an
  archiving iteration, archive up to the lower values of log archiving LSN, and
  write_lsn. Set the log archiving LSN to LSN_MAX at clone initialization, so
  that clone archiving may proceed until the stop LSN is set.
- The above implementation results in cloned LSN != log end LSN for the cloned
  instance, thus adjust crash recovery to work with that. The existing crash
  recovery throws away any crash-recovered metadata and reads its snapshot from
  the data files, whereas the snapshot was not written in any consistency with
  the clone LSN. Fix by no longer throwing away the crash-recovered InnoDB
  dynamic metadata and by not re-reading the the last metadata snapshot. This
  forces also not to re-serialize the metadata in the crash recovery as that
  writes redo log which is to be avoided during clone crash recovery.
- Additionally, at the start of the redo log copy the old implementation
  requested all InnoDB updates to be flushed to disk immediately, to avoid clone
  recovery generating new redo log. This is also disabled as it becomes
  redundant with the fix. This issue is similar to Percona XtraBackup
  https://jira.percona.com/browse/PXB-1634
- Disable clone.local_xa test because the new synchronization no longer blocks
  on XA transactions.
- Re-record two clone-related tests in the component_keyring_file MTR suite.
- Temporarily workaround the MyRocks bug where log_status query hangs if RocksDB
  file deletions were not disabled first.

Note that this change breaks some of the clone-related group replication MTR
tests. As they are not tested in fb-mysql and have been already broken, this
patch makes no effort to fix them.

MyRocks Clone (5/n): fix bug 109922 (SEs do not coordinate clone rollbacks on
instance startup)

If InnoDB clone decides to rollback during its startup, other storage engines
are not aware of this and proceed as usual, resulting in inconsistent instance
between storage engines. Ideally this should be fixed through a 2PC-like design
where all the SEs vote to rollback the clone or to continue during the startup,
but for MyRocks clone a simpler solution suffices: make InnoDB write a special
marker file if it decides to rollback on clone startup.

MyRocks Clone (6/n): on remote clone error, always send COM_ACK to InnoDB too

In the case of multithreaded clone, the waits for the donor threads for the
current clone stage to complete are implemented fully in InnoDB, even though
that, and error handling, should be plugin-level features. With two storage
engines participating in the clone this allows the situation where the other
storage engine client reports an error with COM_ACK to one donor thread, while
some other threads are in InnoDB clone state change waits. Those waits can abort
if InnoDB clone gets an error from the client, but they know nothing about other
SE errors, and will be aborted only by timeouts.

As a minimal fix that does not involve pulling the whole error handling and
state change waiting logic to the plugin layer, make the clone client always
send an error-containing COM_ACK to InnoDB too, whenever it sends one to another
storage engine.

MyRocks Clone (7/n): MyRocks clone implementation

- Cleanup any stale RocksDB checkpoint directories on server shutdown too.

- For MyRocks clone network protocol, define four packet types (clone/common.h):
  - LOCATOR_V1: MyRocks-specific clone locator, consisting of a monotonically
    increasing clone session ID for the donor with an optional applier
    state (an array of file IDs and applied offsets) for restarts, or a zero
    in which case it is a version check empty locator used for handshakes.
  - FILE_NAME_MAP_V1: establishes a relation between a file ID, its name and
    target size.
  - FILE_CHUNK_V1: cloned file chunk data
  - ADD_ESTIMATE_V1: how many bytes should be added to the estimated clone data
    size.
- All packet types share the common package header, which stores the version,
  the payload length without the header, and the packet type.
- (clone/common.h): classes locator, file_metadata, chunk_metadata, &
  estimate_metadata correspond to the four packet types. They are subclassed on
  the donor side to add serialization routines and other donor-specific code and
  on the client side to add deserialization routines and other client-specific
  code. Re. serialization/deserialization, an exception is the locator, whose
  routines are implemented in the common code, because it may be passed both
  - ways.
- (clone/common.h): Introduce a session class that handles the task ID set and
  errors, which is common for both donor and client. For errors, allow saving
  one error, with an optional associated path and errno, that can be set by any
  thread and later reported to the clone plugin.
- (clone/common.cc) Setup MyRocks clone capabilities as HA_CLONE_HYBRID (not
  - applicable but matches InnoDB), HA_CLONE_MULTI_TASK (can clone with multiple
  - threads), & HA_CLONE_RESTART (can restart after a network I/O error).
- (clone/common.cc) Enable O_DIRECT for clone file I/O if, on client if
  rocksdb_use_direct_io_for_flush_and_compaction is enabled, and on donor if
  rocksdb_use_direct_reads is enabled.
- On the donor side (clone/donor.cc), encapsulate RocksDB checkpoints in a
  rdb_checkpoint class. Each checkpoint gets its own ID, which is also a donor
  session ID. The ID is constant for rolling checkpoints.
- Subclass the common file_metadata to donor_file_metadata, chunk_metadata to
  donor_chunk_metadata, estimate_metadata to donor_estimate.
- Implement donor going through a sequence of states: NONE, ROLLING, FINAL,
  FINAL_WITH_LOGS. To support multiple clone donor sessions, implement a manager
  class that looks up the donor instance by locator.
- Implement MyRocks clone donor interface:
  - rocksdb_clone_start: handle the possible start modes:
    - HA_CLONE_MODE_START: create and register a new donor instance, register
      its master task ID.
    - HA_CLONE_MODE_ADD_TASK: lookup the existing donor instance, register a new
      task ID.
    - HA_CLONE_MODE_RESTART: lookup the existing donor instance, wait for the
      old master session to drop it by master task ID reference count, then
      process the locator applier state to reset the donor state.
  - rocksdb_clone_precopy: lookup the donor instance, create a RocksDB snapshot
    if state == NONE, otherwise send files by chunks and roll the checkpoint as
    needed.
  - rocksdb_clone_set_log_stop: add the WAL files to the clone file to copy set.
  - rocksdb_clone_copy: copy the clone files in any donor state.
  - rocksdb_clone_ack: acknowledge errors sent by client.
  - rocksdb_clone_end: for master task ID, stop accepting new worker join
    requests, wait until existing workers quit, and, in the case of restartable
    error, wait for any restarts to happen. Then decrement ref count for the
    master task ID (which will handle both the cases of restarted and not
    restarted) clone. For all other task IDs, simply unregister them.
- On the client side (clone/client.cc), have file_in_progress class to track
  each individual file progress during application, and completed_file for
  fully-applied files.
- Subclass the common file_metadata to client_file_metadata, implement its
  reading from the net buffer. Likewise for client_chunk_metadata subclassing
  chunk_metadata, and client_estimate for estimate_metadata.
- TODO client class.
- Implement MyRocks clone client interface:
  - rocksdb_clone_apply_begin: handle the possible start modes:
    - HA_CLONE_MODE_START: initialize the client instance for a new clone
    - HA_CLONE_MODE_ADD_TASK: register a new task ID for an ongoing clone
    - HA_CLONE_MODE_RESTART: restart the interrupted clone session
    - HA_CLONE_MODE_VERSION: prepare new version locator
  - rocksdb_clone_apply: if passed an error code, save it. Otherwise verify the
    passed buffer and dispatch ADD_ESTIMATE_V1, FILE_NAME_MAP_V1, &
    FILE_CHUNK_V1 packets accordingly.
  - rocksdb_clone_apply_end: for master task, wait for all other tasks to
    complete and finish the clone. For both master and non-master tasks
    unregister their IDs.
- Add rocksdb_sys_vars tests for the new clone-related system variables. They
  are included in the current patch and not the MTR one in order not to break
  the existing rocksdb_sys_vars.all_vars test.
- On instance startup, do clone fixup (clone/common.cc). It consists of:
  - Check whether InnoDB clone rollback marker is present. If yes, force
    rollback of MyRocks clone.
  - If the cloned temporary datadir contains clone-in-progress marker, assume
    half-completed clone and abort.
  - Move the cloned datadirs and WAL dirs in-place. For the datadir, do a
    directory-level rename with a save and later delete of the old datadir. For
    the WAL dir do not try to do parent directory-level renames because the
    mysqld process may not have  permissions to delete that directory. Instead,
    rename all the old files individually, move new files in, delete the old
    files.
  - For all the rename operations, handle them returning EXDEV error (a rename
    would cross a filesystem boundary) by falling back to copy and delete. As we
    expect that it is the cloned RocksDB log files that are most affected by
    this, the total copy I/O  wouldn't be large, and so the default MySQL copy
    routine my_copy is used, without extra copy throughput optimizations. Add a
    fault injection point for MTR to test this code paths.
  - Remove any RocksDB clone checkpoints found.
- On instance startup, always move WALs to the target directory if they are
  found in the rocksdb_datadir yet the rocksdb_wal_dir is different. This
  handles the case of setting rocksdb_wal_dir while instance is shutdown, both
  for clone fixup and regular operation.
- Add PFS instrumentation for all the instrumentable entities in this
  patch (mutexes, rwlocks, file I/O).

MyRocks Clone (8/n): MyRocks clone MTR tests

- Add a new MTR test suite rocksdb_clone
- Support new variable MTR clone/include/clone_command.inc:
  clone_single_threaded, setting which forces single-threaded clone operation.
- Reuse a big part of InnoDB clone tests by including their test files, with the
  exception of tests involving DDL, as concurrent DDL is not supported by
  MyRocks clone at the moment, and debug tests that adjust cloned instance LSN,
  because they desynchronize storage engines. The reused tests are: local_basic,
  local_boundary, local_dml, local_dml_auto_tune, local_dml_no_redo,
  local_insert, local_stage, monitor_progress, remote_basic,
  remote_basic_replace, remote_boundary, remote_dml, remote_dml_auto_tune,
  remote_dml_gtid, remote_dml_no_binlog, remote_dml_no_ssl, remote_dml_recovery,
  remote_dml_replace, remote_dml_ssl, remote_dml_ssl_certificate,
  remote_dml_upgrade, remote_insert, remote_monitor_progress, remote_restart,
  remote_skip_validation, remote_stage, remote_stage_restart.
- As a part of the above, add MyRocks support to local_basic and
  monitor_progress MTR tests.
- Besides the reused InnoDB clone tests, add some new MyRocks-specific tests:
- local_minimal_1_thread, which is the simplest possible local clone test.
- local_minimal_multithreaded, the simplest possible but multithreaded local
  clone test.
- local_minimal_direct_io, which is the simplest possible local clone test in
  all combinations of rocksdb_use_direct_reads and
  rocksdb_use_direct_io_for_flush_and_compaction settings, which are the
  O_DIRECT-enabling options.
- local_dml_o_direct_donor, local_dml_o_direct_client, local_dml_o_direct_both
  that reuse InnoDB local_dml test, but combined with either or both of the
  O_DIRECT-enabling options.
- local_minimal_wal_dir, which clones with various rocksdb_wal_dir settings on
  donor and client.
- local_minimal_wal_dir_copy_delete, which uses a debug injection to test the
  move-across-filesystem-boundary code path in clone startup.
- remote_minimal_direct_io_{1, 2, 3, 4} tests for all the combinations of
  O_DIRECT-enabling options for donor and client instances.
- remote_fail_apply, which uses a debug injection to fail the clone client
  during remote clone.
- remote_inplace_restart, which performs a remote in-place clone with a
  simulated transient error on the client.
- remote_kill_donor, which kill the donor instance at various points of remote
  cloning.
- rolling_checkpoint for testing various combinations of clone rolling
  checkpoint tuning options.
- three_se_basic to test a simple clone across InnoDB, MyRocks, & Blackhole
  engines.

Fix moving RocksDB WALs on startup across file system boundaries

Reviewed By: sunshine-Chun

Differential Revision: D43160162

Pulled By: hermanlee

--------------------------------------------------------------------------------

Rdb clone fixes (facebook#1271)

Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

--------------------------------------------------------------------------------

Clone: fix race condition that may skip some SST files (facebook#1274)

Summary:
A donor thread in the main copy loop took a file to send, released the donor state mutex, and then opened the file. If an ENOENT was received at this point, it was assumed that this was a stale SST file from an older checkpoint that was rolled since.

Because the donor state mutex was released between taking of the file and opening it, the following race was possible:
1) Thread 1 takes the file
2) Thread 2 decides to roll the checkpoint, the old checkpoint is deleted 3) Thread 1 tries to open the file, gets ENOENT
4) Thread 2 creates the new checkpoint, the file re-appears, but it's too late.

Rolling the checkpoint in a donor state mutex critical section is a possible fix, but such section would do a lot of I/O, serializing the parallel threads. Instead, fix by taking the file and opening it in the same critical section.

Pull Request resolved: facebook#1274

Reviewed By: sunshine-Chun

Differential Revision: D43629546

Pulled By: hermanlee

--------------------------------------------------------------------------------

MyRocks clone: add missing donor mutex unlocks on error exits (facebook#1317)

Summary: Pull Request resolved: facebook#1317

Differential Revision: D46564079
inikep pushed a commit to inikep/mysql-5.6 that referenced this issue May 8, 2024
Summary: Pull Request resolved: facebook#1250

MyRocks Clone (1/n): make InnoDB clone MTR tests SE-agnostic

The MyRocks clone MTR tests will reuse the InnoDB tests by including them.
Adjust the tests so that they work this way, i.e. make them storage
engine-agnostic by checking the default_storage_engine system variable and
guarding InnoDB-specific tests this way. This includes the tests that
artificially desync the InnoDB redo log position.

MyRocks Clone (2/n): fix bug 109919 (Clone plugin does not support arbitrary
length O_DIRECT files)

The existing clone plugin I/O routines handle O_DIRECT files with the assumption
that their sizes are always a multiple of a block size. This is too restrictive
because in general O_DIRECT files can be of any size and MyRocks files actually are.

Add a flag to Ha_clone_file to support arbitrary file lengths under O_DIRECT:
o_direct_uneven_file_size. Pass it to clone_client.h/cc
Client::get_aligned_buffer to allocate the buffer that bumps the length to the
next CLONE_OS_ALIGN multiple if it's not already one. Introduce a
pad_for_o_direct helper that performs this calculation for the passed length as
needed.

For writes, if O_DIRECT is enabled and the buffer length is not a multiple of
CLONE_OS_ALIGN, pad it with zeros to the next multiple. Then check the actual
bytes written, and if any padding bytes were actually written, call my_chsize to
truncate the file to the needed size.

MyRocks Clone (3/n): fix bug 109920 (SEs cannot add to clone data size estimates)

In the current implementation, once the first SE sets the data size estimate for
a particular stage, there is no way for other SEs to add to it. Address this by
adding Ha_clone_cbk::add_to_data_size_estimate method.

MyRocks Clone (4/n): fix bug 109921 (Cannot clone with synchronization across SEs)

The existing clone implementation synchronizes InnoDB and binlog directly. There
is no way to make other storage engines join this synchronization, thus replace
it with synchronizing by a performance_schema.log_status query.

- Add table_log_status::get_row method to allow doing the log_status query from
  inside the server.
- Handlerton: introduce two new methods: clone_precopy - currently unused, which
  will handle any non-consistent-snapshot copying before the main consistent
  copy, and set_log_stop - which provides the SE the clone stop position
  information.
- Plugin: introduce Ha_clone_common_cbk, which derives from Ha_clone_cbk, and
  from which derive all the concrete classes: Client_Cbk, Local_Cbk &
  Server_Cbk.
- Ha_clone_common_cbk contains two methods: precopy and synchronize_engines. The
  former calls precopy for all the participating clone SE handlertons, if
  implemented. The latter, only to be called from the main clone thread,
  performs the performance_schema.LOG_STATUS query, splits its result per-SE and
  calls handlerton set_log_stop for all SEs.
- The pfs.log_status query still must be called by InnoDB at a certain point in
  clone. That means that InnoDB is still a special SE in that it must always
  come first in the clone locator vectors. This will also handle donor waits on
  clone restarts, as that is implemented in InnoDB-specific code. To ensure
  this, add a helper clone_hton.cc:make_innodb_first which shuffles the vectors
  as needed. Call it in all the clone-starting code paths, add asserts that
  InnoDB comes first.
- Since hton_clone_copy will call into InnoDB clone copy, which at certain point
  will synchronize engines, calling into the plugin again, and the calling into
  SEs again, introduce a backup locator vector Ha_clone_cbk::all_locators, and
  its setters/getters set_all_locators, reset_all_locators, get_all_locators.
  Use them in the re-entrant hton_clone_copy code.
- Introduce a new InnoDB clone state named SST COPY, which calls precopy and has
  zero files and chunks set to copy in the regular InnoDB clone way, i.e. all it
  does it the precopy.
- The redo log archiving keeps running throughout the precopy stage.
- On stage transition from the precopy to the redo log copy. call synchronize.
- Implement innodb_clone_set_log_stop by parsing out the desired clone stop LSN
  from the passed JSON from the log_status query.
- Make InnoDB redo log archiving stop at that LSN instead of on reaching the end
  of redo log.
- Support stopping redo log archiving in a middle of a log block. As the old
  implementation may encounter this case only at the end of the redo log, it
  took that block from the log buffer in memory. This no longer works with the
  new implementation, where the needed block might reside on disk only. Address
  this by forcing the whole block containing the target LSN to disk, reading it
  from there, and patching its memory copy to stop at the required LSN before
  giving it for the clone to copy.
- To implement redo log block reading on a running server, adapt crash recovery
  function recv_read_log_seg to work both for crash recovery and on a running
  server. This is similar to what Percona Server does for its changed page
  tracking feature.
- Make sure not to archive not-yet-written log: pfs.log_status query gets the
  log system LSN, which is the value of highest log reservation. Thus in an
  archiving iteration, archive up to the lower values of log archiving LSN, and
  write_lsn. Set the log archiving LSN to LSN_MAX at clone initialization, so
  that clone archiving may proceed until the stop LSN is set.
- The above implementation results in cloned LSN != log end LSN for the cloned
  instance, thus adjust crash recovery to work with that. The existing crash
  recovery throws away any crash-recovered metadata and reads its snapshot from
  the data files, whereas the snapshot was not written in any consistency with
  the clone LSN. Fix by no longer throwing away the crash-recovered InnoDB
  dynamic metadata and by not re-reading the the last metadata snapshot. This
  forces also not to re-serialize the metadata in the crash recovery as that
  writes redo log which is to be avoided during clone crash recovery.
- Additionally, at the start of the redo log copy the old implementation
  requested all InnoDB updates to be flushed to disk immediately, to avoid clone
  recovery generating new redo log. This is also disabled as it becomes
  redundant with the fix. This issue is similar to Percona XtraBackup
  https://jira.percona.com/browse/PXB-1634
- Disable clone.local_xa test because the new synchronization no longer blocks
  on XA transactions.
- Re-record two clone-related tests in the component_keyring_file MTR suite.
- Temporarily workaround the MyRocks bug where log_status query hangs if RocksDB
  file deletions were not disabled first.

Note that this change breaks some of the clone-related group replication MTR
tests. As they are not tested in fb-mysql and have been already broken, this
patch makes no effort to fix them.

MyRocks Clone (5/n): fix bug 109922 (SEs do not coordinate clone rollbacks on
instance startup)

If InnoDB clone decides to rollback during its startup, other storage engines
are not aware of this and proceed as usual, resulting in inconsistent instance
between storage engines. Ideally this should be fixed through a 2PC-like design
where all the SEs vote to rollback the clone or to continue during the startup,
but for MyRocks clone a simpler solution suffices: make InnoDB write a special
marker file if it decides to rollback on clone startup.

MyRocks Clone (6/n): on remote clone error, always send COM_ACK to InnoDB too

In the case of multithreaded clone, the waits for the donor threads for the
current clone stage to complete are implemented fully in InnoDB, even though
that, and error handling, should be plugin-level features. With two storage
engines participating in the clone this allows the situation where the other
storage engine client reports an error with COM_ACK to one donor thread, while
some other threads are in InnoDB clone state change waits. Those waits can abort
if InnoDB clone gets an error from the client, but they know nothing about other
SE errors, and will be aborted only by timeouts.

As a minimal fix that does not involve pulling the whole error handling and
state change waiting logic to the plugin layer, make the clone client always
send an error-containing COM_ACK to InnoDB too, whenever it sends one to another
storage engine.

MyRocks Clone (7/n): MyRocks clone implementation

- Cleanup any stale RocksDB checkpoint directories on server shutdown too.

- For MyRocks clone network protocol, define four packet types (clone/common.h):
  - LOCATOR_V1: MyRocks-specific clone locator, consisting of a monotonically
    increasing clone session ID for the donor with an optional applier
    state (an array of file IDs and applied offsets) for restarts, or a zero
    in which case it is a version check empty locator used for handshakes.
  - FILE_NAME_MAP_V1: establishes a relation between a file ID, its name and
    target size.
  - FILE_CHUNK_V1: cloned file chunk data
  - ADD_ESTIMATE_V1: how many bytes should be added to the estimated clone data
    size.
- All packet types share the common package header, which stores the version,
  the payload length without the header, and the packet type.
- (clone/common.h): classes locator, file_metadata, chunk_metadata, &
  estimate_metadata correspond to the four packet types. They are subclassed on
  the donor side to add serialization routines and other donor-specific code and
  on the client side to add deserialization routines and other client-specific
  code. Re. serialization/deserialization, an exception is the locator, whose
  routines are implemented in the common code, because it may be passed both
  - ways.
- (clone/common.h): Introduce a session class that handles the task ID set and
  errors, which is common for both donor and client. For errors, allow saving
  one error, with an optional associated path and errno, that can be set by any
  thread and later reported to the clone plugin.
- (clone/common.cc) Setup MyRocks clone capabilities as HA_CLONE_HYBRID (not
  - applicable but matches InnoDB), HA_CLONE_MULTI_TASK (can clone with multiple
  - threads), & HA_CLONE_RESTART (can restart after a network I/O error).
- (clone/common.cc) Enable O_DIRECT for clone file I/O if, on client if
  rocksdb_use_direct_io_for_flush_and_compaction is enabled, and on donor if
  rocksdb_use_direct_reads is enabled.
- On the donor side (clone/donor.cc), encapsulate RocksDB checkpoints in a
  rdb_checkpoint class. Each checkpoint gets its own ID, which is also a donor
  session ID. The ID is constant for rolling checkpoints.
- Subclass the common file_metadata to donor_file_metadata, chunk_metadata to
  donor_chunk_metadata, estimate_metadata to donor_estimate.
- Implement donor going through a sequence of states: NONE, ROLLING, FINAL,
  FINAL_WITH_LOGS. To support multiple clone donor sessions, implement a manager
  class that looks up the donor instance by locator.
- Implement MyRocks clone donor interface:
  - rocksdb_clone_start: handle the possible start modes:
    - HA_CLONE_MODE_START: create and register a new donor instance, register
      its master task ID.
    - HA_CLONE_MODE_ADD_TASK: lookup the existing donor instance, register a new
      task ID.
    - HA_CLONE_MODE_RESTART: lookup the existing donor instance, wait for the
      old master session to drop it by master task ID reference count, then
      process the locator applier state to reset the donor state.
  - rocksdb_clone_precopy: lookup the donor instance, create a RocksDB snapshot
    if state == NONE, otherwise send files by chunks and roll the checkpoint as
    needed.
  - rocksdb_clone_set_log_stop: add the WAL files to the clone file to copy set.
  - rocksdb_clone_copy: copy the clone files in any donor state.
  - rocksdb_clone_ack: acknowledge errors sent by client.
  - rocksdb_clone_end: for master task ID, stop accepting new worker join
    requests, wait until existing workers quit, and, in the case of restartable
    error, wait for any restarts to happen. Then decrement ref count for the
    master task ID (which will handle both the cases of restarted and not
    restarted) clone. For all other task IDs, simply unregister them.
- On the client side (clone/client.cc), have file_in_progress class to track
  each individual file progress during application, and completed_file for
  fully-applied files.
- Subclass the common file_metadata to client_file_metadata, implement its
  reading from the net buffer. Likewise for client_chunk_metadata subclassing
  chunk_metadata, and client_estimate for estimate_metadata.
- TODO client class.
- Implement MyRocks clone client interface:
  - rocksdb_clone_apply_begin: handle the possible start modes:
    - HA_CLONE_MODE_START: initialize the client instance for a new clone
    - HA_CLONE_MODE_ADD_TASK: register a new task ID for an ongoing clone
    - HA_CLONE_MODE_RESTART: restart the interrupted clone session
    - HA_CLONE_MODE_VERSION: prepare new version locator
  - rocksdb_clone_apply: if passed an error code, save it. Otherwise verify the
    passed buffer and dispatch ADD_ESTIMATE_V1, FILE_NAME_MAP_V1, &
    FILE_CHUNK_V1 packets accordingly.
  - rocksdb_clone_apply_end: for master task, wait for all other tasks to
    complete and finish the clone. For both master and non-master tasks
    unregister their IDs.
- Add rocksdb_sys_vars tests for the new clone-related system variables. They
  are included in the current patch and not the MTR one in order not to break
  the existing rocksdb_sys_vars.all_vars test.
- On instance startup, do clone fixup (clone/common.cc). It consists of:
  - Check whether InnoDB clone rollback marker is present. If yes, force
    rollback of MyRocks clone.
  - If the cloned temporary datadir contains clone-in-progress marker, assume
    half-completed clone and abort.
  - Move the cloned datadirs and WAL dirs in-place. For the datadir, do a
    directory-level rename with a save and later delete of the old datadir. For
    the WAL dir do not try to do parent directory-level renames because the
    mysqld process may not have  permissions to delete that directory. Instead,
    rename all the old files individually, move new files in, delete the old
    files.
  - For all the rename operations, handle them returning EXDEV error (a rename
    would cross a filesystem boundary) by falling back to copy and delete. As we
    expect that it is the cloned RocksDB log files that are most affected by
    this, the total copy I/O  wouldn't be large, and so the default MySQL copy
    routine my_copy is used, without extra copy throughput optimizations. Add a
    fault injection point for MTR to test this code paths.
  - Remove any RocksDB clone checkpoints found.
- On instance startup, always move WALs to the target directory if they are
  found in the rocksdb_datadir yet the rocksdb_wal_dir is different. This
  handles the case of setting rocksdb_wal_dir while instance is shutdown, both
  for clone fixup and regular operation.
- Add PFS instrumentation for all the instrumentable entities in this
  patch (mutexes, rwlocks, file I/O).

MyRocks Clone (8/n): MyRocks clone MTR tests

- Add a new MTR test suite rocksdb_clone
- Support new variable MTR clone/include/clone_command.inc:
  clone_single_threaded, setting which forces single-threaded clone operation.
- Reuse a big part of InnoDB clone tests by including their test files, with the
  exception of tests involving DDL, as concurrent DDL is not supported by
  MyRocks clone at the moment, and debug tests that adjust cloned instance LSN,
  because they desynchronize storage engines. The reused tests are: local_basic,
  local_boundary, local_dml, local_dml_auto_tune, local_dml_no_redo,
  local_insert, local_stage, monitor_progress, remote_basic,
  remote_basic_replace, remote_boundary, remote_dml, remote_dml_auto_tune,
  remote_dml_gtid, remote_dml_no_binlog, remote_dml_no_ssl, remote_dml_recovery,
  remote_dml_replace, remote_dml_ssl, remote_dml_ssl_certificate,
  remote_dml_upgrade, remote_insert, remote_monitor_progress, remote_restart,
  remote_skip_validation, remote_stage, remote_stage_restart.
- As a part of the above, add MyRocks support to local_basic and
  monitor_progress MTR tests.
- Besides the reused InnoDB clone tests, add some new MyRocks-specific tests:
- local_minimal_1_thread, which is the simplest possible local clone test.
- local_minimal_multithreaded, the simplest possible but multithreaded local
  clone test.
- local_minimal_direct_io, which is the simplest possible local clone test in
  all combinations of rocksdb_use_direct_reads and
  rocksdb_use_direct_io_for_flush_and_compaction settings, which are the
  O_DIRECT-enabling options.
- local_dml_o_direct_donor, local_dml_o_direct_client, local_dml_o_direct_both
  that reuse InnoDB local_dml test, but combined with either or both of the
  O_DIRECT-enabling options.
- local_minimal_wal_dir, which clones with various rocksdb_wal_dir settings on
  donor and client.
- local_minimal_wal_dir_copy_delete, which uses a debug injection to test the
  move-across-filesystem-boundary code path in clone startup.
- remote_minimal_direct_io_{1, 2, 3, 4} tests for all the combinations of
  O_DIRECT-enabling options for donor and client instances.
- remote_fail_apply, which uses a debug injection to fail the clone client
  during remote clone.
- remote_inplace_restart, which performs a remote in-place clone with a
  simulated transient error on the client.
- remote_kill_donor, which kill the donor instance at various points of remote
  cloning.
- rolling_checkpoint for testing various combinations of clone rolling
  checkpoint tuning options.
- three_se_basic to test a simple clone across InnoDB, MyRocks, & Blackhole
  engines.

Fix moving RocksDB WALs on startup across file system boundaries

Reviewed By: sunshine-Chun

Differential Revision: D43160162

Pulled By: hermanlee

--------------------------------------------------------------------------------

Rdb clone fixes (facebook#1271)

Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

--------------------------------------------------------------------------------

Clone: fix race condition that may skip some SST files (facebook#1274)

Summary:
A donor thread in the main copy loop took a file to send, released the donor state mutex, and then opened the file. If an ENOENT was received at this point, it was assumed that this was a stale SST file from an older checkpoint that was rolled since.

Because the donor state mutex was released between taking of the file and opening it, the following race was possible:
1) Thread 1 takes the file
2) Thread 2 decides to roll the checkpoint, the old checkpoint is deleted 3) Thread 1 tries to open the file, gets ENOENT
4) Thread 2 creates the new checkpoint, the file re-appears, but it's too late.

Rolling the checkpoint in a donor state mutex critical section is a possible fix, but such section would do a lot of I/O, serializing the parallel threads. Instead, fix by taking the file and opening it in the same critical section.

Pull Request resolved: facebook#1274

Reviewed By: sunshine-Chun

Differential Revision: D43629546

Pulled By: hermanlee

--------------------------------------------------------------------------------

MyRocks clone: add missing donor mutex unlocks on error exits (facebook#1317)

Summary: Pull Request resolved: facebook#1317

Differential Revision: D46564079
inikep pushed a commit to inikep/mysql-5.6 that referenced this issue May 9, 2024
Summary: Pull Request resolved: facebook#1250

MyRocks Clone (1/n): make InnoDB clone MTR tests SE-agnostic

The MyRocks clone MTR tests will reuse the InnoDB tests by including them.
Adjust the tests so that they work this way, i.e. make them storage
engine-agnostic by checking the default_storage_engine system variable and
guarding InnoDB-specific tests this way. This includes the tests that
artificially desync the InnoDB redo log position.

MyRocks Clone (2/n): fix bug 109919 (Clone plugin does not support arbitrary
length O_DIRECT files)

The existing clone plugin I/O routines handle O_DIRECT files with the assumption
that their sizes are always a multiple of a block size. This is too restrictive
because in general O_DIRECT files can be of any size and MyRocks files actually are.

Add a flag to Ha_clone_file to support arbitrary file lengths under O_DIRECT:
o_direct_uneven_file_size. Pass it to clone_client.h/cc
Client::get_aligned_buffer to allocate the buffer that bumps the length to the
next CLONE_OS_ALIGN multiple if it's not already one. Introduce a
pad_for_o_direct helper that performs this calculation for the passed length as
needed.

For writes, if O_DIRECT is enabled and the buffer length is not a multiple of
CLONE_OS_ALIGN, pad it with zeros to the next multiple. Then check the actual
bytes written, and if any padding bytes were actually written, call my_chsize to
truncate the file to the needed size.

MyRocks Clone (3/n): fix bug 109920 (SEs cannot add to clone data size estimates)

In the current implementation, once the first SE sets the data size estimate for
a particular stage, there is no way for other SEs to add to it. Address this by
adding Ha_clone_cbk::add_to_data_size_estimate method.

MyRocks Clone (4/n): fix bug 109921 (Cannot clone with synchronization across SEs)

The existing clone implementation synchronizes InnoDB and binlog directly. There
is no way to make other storage engines join this synchronization, thus replace
it with synchronizing by a performance_schema.log_status query.

- Add table_log_status::get_row method to allow doing the log_status query from
  inside the server.
- Handlerton: introduce two new methods: clone_precopy - currently unused, which
  will handle any non-consistent-snapshot copying before the main consistent
  copy, and set_log_stop - which provides the SE the clone stop position
  information.
- Plugin: introduce Ha_clone_common_cbk, which derives from Ha_clone_cbk, and
  from which derive all the concrete classes: Client_Cbk, Local_Cbk &
  Server_Cbk.
- Ha_clone_common_cbk contains two methods: precopy and synchronize_engines. The
  former calls precopy for all the participating clone SE handlertons, if
  implemented. The latter, only to be called from the main clone thread,
  performs the performance_schema.LOG_STATUS query, splits its result per-SE and
  calls handlerton set_log_stop for all SEs.
- The pfs.log_status query still must be called by InnoDB at a certain point in
  clone. That means that InnoDB is still a special SE in that it must always
  come first in the clone locator vectors. This will also handle donor waits on
  clone restarts, as that is implemented in InnoDB-specific code. To ensure
  this, add a helper clone_hton.cc:make_innodb_first which shuffles the vectors
  as needed. Call it in all the clone-starting code paths, add asserts that
  InnoDB comes first.
- Since hton_clone_copy will call into InnoDB clone copy, which at certain point
  will synchronize engines, calling into the plugin again, and the calling into
  SEs again, introduce a backup locator vector Ha_clone_cbk::all_locators, and
  its setters/getters set_all_locators, reset_all_locators, get_all_locators.
  Use them in the re-entrant hton_clone_copy code.
- Introduce a new InnoDB clone state named SST COPY, which calls precopy and has
  zero files and chunks set to copy in the regular InnoDB clone way, i.e. all it
  does it the precopy.
- The redo log archiving keeps running throughout the precopy stage.
- On stage transition from the precopy to the redo log copy. call synchronize.
- Implement innodb_clone_set_log_stop by parsing out the desired clone stop LSN
  from the passed JSON from the log_status query.
- Make InnoDB redo log archiving stop at that LSN instead of on reaching the end
  of redo log.
- Support stopping redo log archiving in a middle of a log block. As the old
  implementation may encounter this case only at the end of the redo log, it
  took that block from the log buffer in memory. This no longer works with the
  new implementation, where the needed block might reside on disk only. Address
  this by forcing the whole block containing the target LSN to disk, reading it
  from there, and patching its memory copy to stop at the required LSN before
  giving it for the clone to copy.
- To implement redo log block reading on a running server, adapt crash recovery
  function recv_read_log_seg to work both for crash recovery and on a running
  server. This is similar to what Percona Server does for its changed page
  tracking feature.
- Make sure not to archive not-yet-written log: pfs.log_status query gets the
  log system LSN, which is the value of highest log reservation. Thus in an
  archiving iteration, archive up to the lower values of log archiving LSN, and
  write_lsn. Set the log archiving LSN to LSN_MAX at clone initialization, so
  that clone archiving may proceed until the stop LSN is set.
- The above implementation results in cloned LSN != log end LSN for the cloned
  instance, thus adjust crash recovery to work with that. The existing crash
  recovery throws away any crash-recovered metadata and reads its snapshot from
  the data files, whereas the snapshot was not written in any consistency with
  the clone LSN. Fix by no longer throwing away the crash-recovered InnoDB
  dynamic metadata and by not re-reading the the last metadata snapshot. This
  forces also not to re-serialize the metadata in the crash recovery as that
  writes redo log which is to be avoided during clone crash recovery.
- Additionally, at the start of the redo log copy the old implementation
  requested all InnoDB updates to be flushed to disk immediately, to avoid clone
  recovery generating new redo log. This is also disabled as it becomes
  redundant with the fix. This issue is similar to Percona XtraBackup
  https://jira.percona.com/browse/PXB-1634
- Disable clone.local_xa test because the new synchronization no longer blocks
  on XA transactions.
- Re-record two clone-related tests in the component_keyring_file MTR suite.
- Temporarily workaround the MyRocks bug where log_status query hangs if RocksDB
  file deletions were not disabled first.

Note that this change breaks some of the clone-related group replication MTR
tests. As they are not tested in fb-mysql and have been already broken, this
patch makes no effort to fix them.

MyRocks Clone (5/n): fix bug 109922 (SEs do not coordinate clone rollbacks on
instance startup)

If InnoDB clone decides to rollback during its startup, other storage engines
are not aware of this and proceed as usual, resulting in inconsistent instance
between storage engines. Ideally this should be fixed through a 2PC-like design
where all the SEs vote to rollback the clone or to continue during the startup,
but for MyRocks clone a simpler solution suffices: make InnoDB write a special
marker file if it decides to rollback on clone startup.

MyRocks Clone (6/n): on remote clone error, always send COM_ACK to InnoDB too

In the case of multithreaded clone, the waits for the donor threads for the
current clone stage to complete are implemented fully in InnoDB, even though
that, and error handling, should be plugin-level features. With two storage
engines participating in the clone this allows the situation where the other
storage engine client reports an error with COM_ACK to one donor thread, while
some other threads are in InnoDB clone state change waits. Those waits can abort
if InnoDB clone gets an error from the client, but they know nothing about other
SE errors, and will be aborted only by timeouts.

As a minimal fix that does not involve pulling the whole error handling and
state change waiting logic to the plugin layer, make the clone client always
send an error-containing COM_ACK to InnoDB too, whenever it sends one to another
storage engine.

MyRocks Clone (7/n): MyRocks clone implementation

- Cleanup any stale RocksDB checkpoint directories on server shutdown too.

- For MyRocks clone network protocol, define four packet types (clone/common.h):
  - LOCATOR_V1: MyRocks-specific clone locator, consisting of a monotonically
    increasing clone session ID for the donor with an optional applier
    state (an array of file IDs and applied offsets) for restarts, or a zero
    in which case it is a version check empty locator used for handshakes.
  - FILE_NAME_MAP_V1: establishes a relation between a file ID, its name and
    target size.
  - FILE_CHUNK_V1: cloned file chunk data
  - ADD_ESTIMATE_V1: how many bytes should be added to the estimated clone data
    size.
- All packet types share the common package header, which stores the version,
  the payload length without the header, and the packet type.
- (clone/common.h): classes locator, file_metadata, chunk_metadata, &
  estimate_metadata correspond to the four packet types. They are subclassed on
  the donor side to add serialization routines and other donor-specific code and
  on the client side to add deserialization routines and other client-specific
  code. Re. serialization/deserialization, an exception is the locator, whose
  routines are implemented in the common code, because it may be passed both
  - ways.
- (clone/common.h): Introduce a session class that handles the task ID set and
  errors, which is common for both donor and client. For errors, allow saving
  one error, with an optional associated path and errno, that can be set by any
  thread and later reported to the clone plugin.
- (clone/common.cc) Setup MyRocks clone capabilities as HA_CLONE_HYBRID (not
  - applicable but matches InnoDB), HA_CLONE_MULTI_TASK (can clone with multiple
  - threads), & HA_CLONE_RESTART (can restart after a network I/O error).
- (clone/common.cc) Enable O_DIRECT for clone file I/O if, on client if
  rocksdb_use_direct_io_for_flush_and_compaction is enabled, and on donor if
  rocksdb_use_direct_reads is enabled.
- On the donor side (clone/donor.cc), encapsulate RocksDB checkpoints in a
  rdb_checkpoint class. Each checkpoint gets its own ID, which is also a donor
  session ID. The ID is constant for rolling checkpoints.
- Subclass the common file_metadata to donor_file_metadata, chunk_metadata to
  donor_chunk_metadata, estimate_metadata to donor_estimate.
- Implement donor going through a sequence of states: NONE, ROLLING, FINAL,
  FINAL_WITH_LOGS. To support multiple clone donor sessions, implement a manager
  class that looks up the donor instance by locator.
- Implement MyRocks clone donor interface:
  - rocksdb_clone_start: handle the possible start modes:
    - HA_CLONE_MODE_START: create and register a new donor instance, register
      its master task ID.
    - HA_CLONE_MODE_ADD_TASK: lookup the existing donor instance, register a new
      task ID.
    - HA_CLONE_MODE_RESTART: lookup the existing donor instance, wait for the
      old master session to drop it by master task ID reference count, then
      process the locator applier state to reset the donor state.
  - rocksdb_clone_precopy: lookup the donor instance, create a RocksDB snapshot
    if state == NONE, otherwise send files by chunks and roll the checkpoint as
    needed.
  - rocksdb_clone_set_log_stop: add the WAL files to the clone file to copy set.
  - rocksdb_clone_copy: copy the clone files in any donor state.
  - rocksdb_clone_ack: acknowledge errors sent by client.
  - rocksdb_clone_end: for master task ID, stop accepting new worker join
    requests, wait until existing workers quit, and, in the case of restartable
    error, wait for any restarts to happen. Then decrement ref count for the
    master task ID (which will handle both the cases of restarted and not
    restarted) clone. For all other task IDs, simply unregister them.
- On the client side (clone/client.cc), have file_in_progress class to track
  each individual file progress during application, and completed_file for
  fully-applied files.
- Subclass the common file_metadata to client_file_metadata, implement its
  reading from the net buffer. Likewise for client_chunk_metadata subclassing
  chunk_metadata, and client_estimate for estimate_metadata.
- TODO client class.
- Implement MyRocks clone client interface:
  - rocksdb_clone_apply_begin: handle the possible start modes:
    - HA_CLONE_MODE_START: initialize the client instance for a new clone
    - HA_CLONE_MODE_ADD_TASK: register a new task ID for an ongoing clone
    - HA_CLONE_MODE_RESTART: restart the interrupted clone session
    - HA_CLONE_MODE_VERSION: prepare new version locator
  - rocksdb_clone_apply: if passed an error code, save it. Otherwise verify the
    passed buffer and dispatch ADD_ESTIMATE_V1, FILE_NAME_MAP_V1, &
    FILE_CHUNK_V1 packets accordingly.
  - rocksdb_clone_apply_end: for master task, wait for all other tasks to
    complete and finish the clone. For both master and non-master tasks
    unregister their IDs.
- Add rocksdb_sys_vars tests for the new clone-related system variables. They
  are included in the current patch and not the MTR one in order not to break
  the existing rocksdb_sys_vars.all_vars test.
- On instance startup, do clone fixup (clone/common.cc). It consists of:
  - Check whether InnoDB clone rollback marker is present. If yes, force
    rollback of MyRocks clone.
  - If the cloned temporary datadir contains clone-in-progress marker, assume
    half-completed clone and abort.
  - Move the cloned datadirs and WAL dirs in-place. For the datadir, do a
    directory-level rename with a save and later delete of the old datadir. For
    the WAL dir do not try to do parent directory-level renames because the
    mysqld process may not have  permissions to delete that directory. Instead,
    rename all the old files individually, move new files in, delete the old
    files.
  - For all the rename operations, handle them returning EXDEV error (a rename
    would cross a filesystem boundary) by falling back to copy and delete. As we
    expect that it is the cloned RocksDB log files that are most affected by
    this, the total copy I/O  wouldn't be large, and so the default MySQL copy
    routine my_copy is used, without extra copy throughput optimizations. Add a
    fault injection point for MTR to test this code paths.
  - Remove any RocksDB clone checkpoints found.
- On instance startup, always move WALs to the target directory if they are
  found in the rocksdb_datadir yet the rocksdb_wal_dir is different. This
  handles the case of setting rocksdb_wal_dir while instance is shutdown, both
  for clone fixup and regular operation.
- Add PFS instrumentation for all the instrumentable entities in this
  patch (mutexes, rwlocks, file I/O).

MyRocks Clone (8/n): MyRocks clone MTR tests

- Add a new MTR test suite rocksdb_clone
- Support new variable MTR clone/include/clone_command.inc:
  clone_single_threaded, setting which forces single-threaded clone operation.
- Reuse a big part of InnoDB clone tests by including their test files, with the
  exception of tests involving DDL, as concurrent DDL is not supported by
  MyRocks clone at the moment, and debug tests that adjust cloned instance LSN,
  because they desynchronize storage engines. The reused tests are: local_basic,
  local_boundary, local_dml, local_dml_auto_tune, local_dml_no_redo,
  local_insert, local_stage, monitor_progress, remote_basic,
  remote_basic_replace, remote_boundary, remote_dml, remote_dml_auto_tune,
  remote_dml_gtid, remote_dml_no_binlog, remote_dml_no_ssl, remote_dml_recovery,
  remote_dml_replace, remote_dml_ssl, remote_dml_ssl_certificate,
  remote_dml_upgrade, remote_insert, remote_monitor_progress, remote_restart,
  remote_skip_validation, remote_stage, remote_stage_restart.
- As a part of the above, add MyRocks support to local_basic and
  monitor_progress MTR tests.
- Besides the reused InnoDB clone tests, add some new MyRocks-specific tests:
- local_minimal_1_thread, which is the simplest possible local clone test.
- local_minimal_multithreaded, the simplest possible but multithreaded local
  clone test.
- local_minimal_direct_io, which is the simplest possible local clone test in
  all combinations of rocksdb_use_direct_reads and
  rocksdb_use_direct_io_for_flush_and_compaction settings, which are the
  O_DIRECT-enabling options.
- local_dml_o_direct_donor, local_dml_o_direct_client, local_dml_o_direct_both
  that reuse InnoDB local_dml test, but combined with either or both of the
  O_DIRECT-enabling options.
- local_minimal_wal_dir, which clones with various rocksdb_wal_dir settings on
  donor and client.
- local_minimal_wal_dir_copy_delete, which uses a debug injection to test the
  move-across-filesystem-boundary code path in clone startup.
- remote_minimal_direct_io_{1, 2, 3, 4} tests for all the combinations of
  O_DIRECT-enabling options for donor and client instances.
- remote_fail_apply, which uses a debug injection to fail the clone client
  during remote clone.
- remote_inplace_restart, which performs a remote in-place clone with a
  simulated transient error on the client.
- remote_kill_donor, which kill the donor instance at various points of remote
  cloning.
- rolling_checkpoint for testing various combinations of clone rolling
  checkpoint tuning options.
- three_se_basic to test a simple clone across InnoDB, MyRocks, & Blackhole
  engines.

Fix moving RocksDB WALs on startup across file system boundaries

Reviewed By: sunshine-Chun

Differential Revision: D43160162

Pulled By: hermanlee

--------------------------------------------------------------------------------

Rdb clone fixes (facebook#1271)

Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

--------------------------------------------------------------------------------

Clone: fix race condition that may skip some SST files (facebook#1274)

Summary:
A donor thread in the main copy loop took a file to send, released the donor state mutex, and then opened the file. If an ENOENT was received at this point, it was assumed that this was a stale SST file from an older checkpoint that was rolled since.

Because the donor state mutex was released between taking of the file and opening it, the following race was possible:
1) Thread 1 takes the file
2) Thread 2 decides to roll the checkpoint, the old checkpoint is deleted 3) Thread 1 tries to open the file, gets ENOENT
4) Thread 2 creates the new checkpoint, the file re-appears, but it's too late.

Rolling the checkpoint in a donor state mutex critical section is a possible fix, but such section would do a lot of I/O, serializing the parallel threads. Instead, fix by taking the file and opening it in the same critical section.

Pull Request resolved: facebook#1274

Reviewed By: sunshine-Chun

Differential Revision: D43629546

Pulled By: hermanlee

--------------------------------------------------------------------------------

MyRocks clone: add missing donor mutex unlocks on error exits (facebook#1317)

Summary: Pull Request resolved: facebook#1317

Differential Revision: D46564079
inikep pushed a commit to inikep/mysql-5.6 that referenced this issue May 10, 2024
Summary: Pull Request resolved: facebook#1250

MyRocks Clone (1/n): make InnoDB clone MTR tests SE-agnostic

The MyRocks clone MTR tests will reuse the InnoDB tests by including them.
Adjust the tests so that they work this way, i.e. make them storage
engine-agnostic by checking the default_storage_engine system variable and
guarding InnoDB-specific tests this way. This includes the tests that
artificially desync the InnoDB redo log position.

MyRocks Clone (2/n): fix bug 109919 (Clone plugin does not support arbitrary
length O_DIRECT files)

The existing clone plugin I/O routines handle O_DIRECT files with the assumption
that their sizes are always a multiple of a block size. This is too restrictive
because in general O_DIRECT files can be of any size and MyRocks files actually are.

Add a flag to Ha_clone_file to support arbitrary file lengths under O_DIRECT:
o_direct_uneven_file_size. Pass it to clone_client.h/cc
Client::get_aligned_buffer to allocate the buffer that bumps the length to the
next CLONE_OS_ALIGN multiple if it's not already one. Introduce a
pad_for_o_direct helper that performs this calculation for the passed length as
needed.

For writes, if O_DIRECT is enabled and the buffer length is not a multiple of
CLONE_OS_ALIGN, pad it with zeros to the next multiple. Then check the actual
bytes written, and if any padding bytes were actually written, call my_chsize to
truncate the file to the needed size.

MyRocks Clone (3/n): fix bug 109920 (SEs cannot add to clone data size estimates)

In the current implementation, once the first SE sets the data size estimate for
a particular stage, there is no way for other SEs to add to it. Address this by
adding Ha_clone_cbk::add_to_data_size_estimate method.

MyRocks Clone (4/n): fix bug 109921 (Cannot clone with synchronization across SEs)

The existing clone implementation synchronizes InnoDB and binlog directly. There
is no way to make other storage engines join this synchronization, thus replace
it with synchronizing by a performance_schema.log_status query.

- Add table_log_status::get_row method to allow doing the log_status query from
  inside the server.
- Handlerton: introduce two new methods: clone_precopy - currently unused, which
  will handle any non-consistent-snapshot copying before the main consistent
  copy, and set_log_stop - which provides the SE the clone stop position
  information.
- Plugin: introduce Ha_clone_common_cbk, which derives from Ha_clone_cbk, and
  from which derive all the concrete classes: Client_Cbk, Local_Cbk &
  Server_Cbk.
- Ha_clone_common_cbk contains two methods: precopy and synchronize_engines. The
  former calls precopy for all the participating clone SE handlertons, if
  implemented. The latter, only to be called from the main clone thread,
  performs the performance_schema.LOG_STATUS query, splits its result per-SE and
  calls handlerton set_log_stop for all SEs.
- The pfs.log_status query still must be called by InnoDB at a certain point in
  clone. That means that InnoDB is still a special SE in that it must always
  come first in the clone locator vectors. This will also handle donor waits on
  clone restarts, as that is implemented in InnoDB-specific code. To ensure
  this, add a helper clone_hton.cc:make_innodb_first which shuffles the vectors
  as needed. Call it in all the clone-starting code paths, add asserts that
  InnoDB comes first.
- Since hton_clone_copy will call into InnoDB clone copy, which at certain point
  will synchronize engines, calling into the plugin again, and the calling into
  SEs again, introduce a backup locator vector Ha_clone_cbk::all_locators, and
  its setters/getters set_all_locators, reset_all_locators, get_all_locators.
  Use them in the re-entrant hton_clone_copy code.
- Introduce a new InnoDB clone state named SST COPY, which calls precopy and has
  zero files and chunks set to copy in the regular InnoDB clone way, i.e. all it
  does it the precopy.
- The redo log archiving keeps running throughout the precopy stage.
- On stage transition from the precopy to the redo log copy. call synchronize.
- Implement innodb_clone_set_log_stop by parsing out the desired clone stop LSN
  from the passed JSON from the log_status query.
- Make InnoDB redo log archiving stop at that LSN instead of on reaching the end
  of redo log.
- Support stopping redo log archiving in a middle of a log block. As the old
  implementation may encounter this case only at the end of the redo log, it
  took that block from the log buffer in memory. This no longer works with the
  new implementation, where the needed block might reside on disk only. Address
  this by forcing the whole block containing the target LSN to disk, reading it
  from there, and patching its memory copy to stop at the required LSN before
  giving it for the clone to copy.
- To implement redo log block reading on a running server, adapt crash recovery
  function recv_read_log_seg to work both for crash recovery and on a running
  server. This is similar to what Percona Server does for its changed page
  tracking feature.
- Make sure not to archive not-yet-written log: pfs.log_status query gets the
  log system LSN, which is the value of highest log reservation. Thus in an
  archiving iteration, archive up to the lower values of log archiving LSN, and
  write_lsn. Set the log archiving LSN to LSN_MAX at clone initialization, so
  that clone archiving may proceed until the stop LSN is set.
- The above implementation results in cloned LSN != log end LSN for the cloned
  instance, thus adjust crash recovery to work with that. The existing crash
  recovery throws away any crash-recovered metadata and reads its snapshot from
  the data files, whereas the snapshot was not written in any consistency with
  the clone LSN. Fix by no longer throwing away the crash-recovered InnoDB
  dynamic metadata and by not re-reading the the last metadata snapshot. This
  forces also not to re-serialize the metadata in the crash recovery as that
  writes redo log which is to be avoided during clone crash recovery.
- Additionally, at the start of the redo log copy the old implementation
  requested all InnoDB updates to be flushed to disk immediately, to avoid clone
  recovery generating new redo log. This is also disabled as it becomes
  redundant with the fix. This issue is similar to Percona XtraBackup
  https://jira.percona.com/browse/PXB-1634
- Disable clone.local_xa test because the new synchronization no longer blocks
  on XA transactions.
- Re-record two clone-related tests in the component_keyring_file MTR suite.
- Temporarily workaround the MyRocks bug where log_status query hangs if RocksDB
  file deletions were not disabled first.

Note that this change breaks some of the clone-related group replication MTR
tests. As they are not tested in fb-mysql and have been already broken, this
patch makes no effort to fix them.

MyRocks Clone (5/n): fix bug 109922 (SEs do not coordinate clone rollbacks on
instance startup)

If InnoDB clone decides to rollback during its startup, other storage engines
are not aware of this and proceed as usual, resulting in inconsistent instance
between storage engines. Ideally this should be fixed through a 2PC-like design
where all the SEs vote to rollback the clone or to continue during the startup,
but for MyRocks clone a simpler solution suffices: make InnoDB write a special
marker file if it decides to rollback on clone startup.

MyRocks Clone (6/n): on remote clone error, always send COM_ACK to InnoDB too

In the case of multithreaded clone, the waits for the donor threads for the
current clone stage to complete are implemented fully in InnoDB, even though
that, and error handling, should be plugin-level features. With two storage
engines participating in the clone this allows the situation where the other
storage engine client reports an error with COM_ACK to one donor thread, while
some other threads are in InnoDB clone state change waits. Those waits can abort
if InnoDB clone gets an error from the client, but they know nothing about other
SE errors, and will be aborted only by timeouts.

As a minimal fix that does not involve pulling the whole error handling and
state change waiting logic to the plugin layer, make the clone client always
send an error-containing COM_ACK to InnoDB too, whenever it sends one to another
storage engine.

MyRocks Clone (7/n): MyRocks clone implementation

- Cleanup any stale RocksDB checkpoint directories on server shutdown too.

- For MyRocks clone network protocol, define four packet types (clone/common.h):
  - LOCATOR_V1: MyRocks-specific clone locator, consisting of a monotonically
    increasing clone session ID for the donor with an optional applier
    state (an array of file IDs and applied offsets) for restarts, or a zero
    in which case it is a version check empty locator used for handshakes.
  - FILE_NAME_MAP_V1: establishes a relation between a file ID, its name and
    target size.
  - FILE_CHUNK_V1: cloned file chunk data
  - ADD_ESTIMATE_V1: how many bytes should be added to the estimated clone data
    size.
- All packet types share the common package header, which stores the version,
  the payload length without the header, and the packet type.
- (clone/common.h): classes locator, file_metadata, chunk_metadata, &
  estimate_metadata correspond to the four packet types. They are subclassed on
  the donor side to add serialization routines and other donor-specific code and
  on the client side to add deserialization routines and other client-specific
  code. Re. serialization/deserialization, an exception is the locator, whose
  routines are implemented in the common code, because it may be passed both
  - ways.
- (clone/common.h): Introduce a session class that handles the task ID set and
  errors, which is common for both donor and client. For errors, allow saving
  one error, with an optional associated path and errno, that can be set by any
  thread and later reported to the clone plugin.
- (clone/common.cc) Setup MyRocks clone capabilities as HA_CLONE_HYBRID (not
  - applicable but matches InnoDB), HA_CLONE_MULTI_TASK (can clone with multiple
  - threads), & HA_CLONE_RESTART (can restart after a network I/O error).
- (clone/common.cc) Enable O_DIRECT for clone file I/O if, on client if
  rocksdb_use_direct_io_for_flush_and_compaction is enabled, and on donor if
  rocksdb_use_direct_reads is enabled.
- On the donor side (clone/donor.cc), encapsulate RocksDB checkpoints in a
  rdb_checkpoint class. Each checkpoint gets its own ID, which is also a donor
  session ID. The ID is constant for rolling checkpoints.
- Subclass the common file_metadata to donor_file_metadata, chunk_metadata to
  donor_chunk_metadata, estimate_metadata to donor_estimate.
- Implement donor going through a sequence of states: NONE, ROLLING, FINAL,
  FINAL_WITH_LOGS. To support multiple clone donor sessions, implement a manager
  class that looks up the donor instance by locator.
- Implement MyRocks clone donor interface:
  - rocksdb_clone_start: handle the possible start modes:
    - HA_CLONE_MODE_START: create and register a new donor instance, register
      its master task ID.
    - HA_CLONE_MODE_ADD_TASK: lookup the existing donor instance, register a new
      task ID.
    - HA_CLONE_MODE_RESTART: lookup the existing donor instance, wait for the
      old master session to drop it by master task ID reference count, then
      process the locator applier state to reset the donor state.
  - rocksdb_clone_precopy: lookup the donor instance, create a RocksDB snapshot
    if state == NONE, otherwise send files by chunks and roll the checkpoint as
    needed.
  - rocksdb_clone_set_log_stop: add the WAL files to the clone file to copy set.
  - rocksdb_clone_copy: copy the clone files in any donor state.
  - rocksdb_clone_ack: acknowledge errors sent by client.
  - rocksdb_clone_end: for master task ID, stop accepting new worker join
    requests, wait until existing workers quit, and, in the case of restartable
    error, wait for any restarts to happen. Then decrement ref count for the
    master task ID (which will handle both the cases of restarted and not
    restarted) clone. For all other task IDs, simply unregister them.
- On the client side (clone/client.cc), have file_in_progress class to track
  each individual file progress during application, and completed_file for
  fully-applied files.
- Subclass the common file_metadata to client_file_metadata, implement its
  reading from the net buffer. Likewise for client_chunk_metadata subclassing
  chunk_metadata, and client_estimate for estimate_metadata.
- TODO client class.
- Implement MyRocks clone client interface:
  - rocksdb_clone_apply_begin: handle the possible start modes:
    - HA_CLONE_MODE_START: initialize the client instance for a new clone
    - HA_CLONE_MODE_ADD_TASK: register a new task ID for an ongoing clone
    - HA_CLONE_MODE_RESTART: restart the interrupted clone session
    - HA_CLONE_MODE_VERSION: prepare new version locator
  - rocksdb_clone_apply: if passed an error code, save it. Otherwise verify the
    passed buffer and dispatch ADD_ESTIMATE_V1, FILE_NAME_MAP_V1, &
    FILE_CHUNK_V1 packets accordingly.
  - rocksdb_clone_apply_end: for master task, wait for all other tasks to
    complete and finish the clone. For both master and non-master tasks
    unregister their IDs.
- Add rocksdb_sys_vars tests for the new clone-related system variables. They
  are included in the current patch and not the MTR one in order not to break
  the existing rocksdb_sys_vars.all_vars test.
- On instance startup, do clone fixup (clone/common.cc). It consists of:
  - Check whether InnoDB clone rollback marker is present. If yes, force
    rollback of MyRocks clone.
  - If the cloned temporary datadir contains clone-in-progress marker, assume
    half-completed clone and abort.
  - Move the cloned datadirs and WAL dirs in-place. For the datadir, do a
    directory-level rename with a save and later delete of the old datadir. For
    the WAL dir do not try to do parent directory-level renames because the
    mysqld process may not have  permissions to delete that directory. Instead,
    rename all the old files individually, move new files in, delete the old
    files.
  - For all the rename operations, handle them returning EXDEV error (a rename
    would cross a filesystem boundary) by falling back to copy and delete. As we
    expect that it is the cloned RocksDB log files that are most affected by
    this, the total copy I/O  wouldn't be large, and so the default MySQL copy
    routine my_copy is used, without extra copy throughput optimizations. Add a
    fault injection point for MTR to test this code paths.
  - Remove any RocksDB clone checkpoints found.
- On instance startup, always move WALs to the target directory if they are
  found in the rocksdb_datadir yet the rocksdb_wal_dir is different. This
  handles the case of setting rocksdb_wal_dir while instance is shutdown, both
  for clone fixup and regular operation.
- Add PFS instrumentation for all the instrumentable entities in this
  patch (mutexes, rwlocks, file I/O).

MyRocks Clone (8/n): MyRocks clone MTR tests

- Add a new MTR test suite rocksdb_clone
- Support new variable MTR clone/include/clone_command.inc:
  clone_single_threaded, setting which forces single-threaded clone operation.
- Reuse a big part of InnoDB clone tests by including their test files, with the
  exception of tests involving DDL, as concurrent DDL is not supported by
  MyRocks clone at the moment, and debug tests that adjust cloned instance LSN,
  because they desynchronize storage engines. The reused tests are: local_basic,
  local_boundary, local_dml, local_dml_auto_tune, local_dml_no_redo,
  local_insert, local_stage, monitor_progress, remote_basic,
  remote_basic_replace, remote_boundary, remote_dml, remote_dml_auto_tune,
  remote_dml_gtid, remote_dml_no_binlog, remote_dml_no_ssl, remote_dml_recovery,
  remote_dml_replace, remote_dml_ssl, remote_dml_ssl_certificate,
  remote_dml_upgrade, remote_insert, remote_monitor_progress, remote_restart,
  remote_skip_validation, remote_stage, remote_stage_restart.
- As a part of the above, add MyRocks support to local_basic and
  monitor_progress MTR tests.
- Besides the reused InnoDB clone tests, add some new MyRocks-specific tests:
- local_minimal_1_thread, which is the simplest possible local clone test.
- local_minimal_multithreaded, the simplest possible but multithreaded local
  clone test.
- local_minimal_direct_io, which is the simplest possible local clone test in
  all combinations of rocksdb_use_direct_reads and
  rocksdb_use_direct_io_for_flush_and_compaction settings, which are the
  O_DIRECT-enabling options.
- local_dml_o_direct_donor, local_dml_o_direct_client, local_dml_o_direct_both
  that reuse InnoDB local_dml test, but combined with either or both of the
  O_DIRECT-enabling options.
- local_minimal_wal_dir, which clones with various rocksdb_wal_dir settings on
  donor and client.
- local_minimal_wal_dir_copy_delete, which uses a debug injection to test the
  move-across-filesystem-boundary code path in clone startup.
- remote_minimal_direct_io_{1, 2, 3, 4} tests for all the combinations of
  O_DIRECT-enabling options for donor and client instances.
- remote_fail_apply, which uses a debug injection to fail the clone client
  during remote clone.
- remote_inplace_restart, which performs a remote in-place clone with a
  simulated transient error on the client.
- remote_kill_donor, which kill the donor instance at various points of remote
  cloning.
- rolling_checkpoint for testing various combinations of clone rolling
  checkpoint tuning options.
- three_se_basic to test a simple clone across InnoDB, MyRocks, & Blackhole
  engines.

Fix moving RocksDB WALs on startup across file system boundaries

Reviewed By: sunshine-Chun

Differential Revision: D43160162

Pulled By: hermanlee

--------------------------------------------------------------------------------

Rdb clone fixes (facebook#1271)

Summary:
Do not leave an empty WAL temp dir behind after clone if rocksdb_wal_dir != rocksdb_datadir

Add asserts and __builtin_unreachable after exhaustive switch statements

To help the compiler with the case where it cannot tell that the switch is
actually exhaustive. This fixes, with GCC:

/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc: In function ‘bool myrocks::clone::should_use_direct_io(const string&, myrocks::clone::mode_for_direct_io)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.cc:295:1: error: control reaches end of non-void function [-Werror=return-type]
  295 | }
      | ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In function ‘int myrocks::rocksdb_clone_apply(handlerton*, THD*, const uchar*, uint, uint, int, Ha_clone_cbk*)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:821:1: error: control reaches end of non-void function [-Werror=return-type]
  821 | }
      | ^

Remove redundant constexpr

This fixes, under GCC:

In file included from /usr/include/c++/11/cassert:44,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_checksum.h:11,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/options.h:26,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/file_system.h:34,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/rocksdb/include/rocksdb/sst_file_manager.h:13,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/../ha_rocksdb.h:45,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:30:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc: In member function ‘constexpr void {anonymous}::client::assert_inactive() const’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:288:37: error: call to non-‘constexpr’ function ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’
  288 |     assert(m_files_in_progress.empty());
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/c++/11/unordered_map:47,
                 from /usr/include/c++/11/functional:61,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/common.h:25,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/clone/client.cc:17:
/usr/include/c++/11/bits/unordered_map.h:305:7: note: ‘bool std::unordered_map<_Key, _Tp, _Hash, _Pred, _Alloc>::empty() const [with _Key = long unsigned int; _Tp = {anonymous}::file_in_progress; _Hash = std::hash<long unsigned int>; _Pred = std::equal_to<long unsigned int>; _Alloc = std::allocator<std::pair<const long unsigned int, {anonymous}::file_in_progress> >]’ declared here
  305 |       empty() const noexcept
      |       ^~~~~

Factor out clone metadata deserialization buffer length checking

Add an overflow check and a cast, extract resulting code to a new helper
function.

This fixes GCC warnings:

Fix a cast

"const" was redundant because it indicated that the pointer itself, not its
contents, were const. At the same time replace the C style cast with a C++ one,
and drop a redundant trailing semicolon.

This fixes a GCC warning:

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/rdb_cf_options.cc:36:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h: In function ‘std::string myrocks::get_index_key(myrocks::Index_id)’:
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./rdb_sst_partitioner_factory.h:38:26: error: type qualifiers ignored on cast result type [-Werror=ignored-qualifiers]
   38 |   rdb_netbuf_store_index((uchar *const)buf.data(), index_id);
      |                          ^~~~~~~~~~~~~~~~~~~~~~~~

Use GCC-compatible attribute((nonnull)) syntax

Clang allows adding this attribute to individual function parameters, while GCC
only allows annotating the whole declaration with relevant parameter indexes.
Use the GCC-compatible syntax.

This fixes

In file included from /home/laurynas/vilniusdb/rdb-clone-fixes/include/my_checksum.h:39,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:32,
                 from /home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/ha_rocksdb.cc:24:
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:495:43: note: in expansion of macro ‘MY_ATTRIBUTE’
  495 |   int rename_table(const char *const from MY_ATTRIBUTE((__nonnull__)),
      |                                           ^~~~~~~~~~~~
/home/laurynas/vilniusdb/rdb-clone-fixes/include/my_compiler.h:100:40: error: ‘nonnull’ attribute only applies to function types [-Werror=attributes]
  100 | #define MY_ATTRIBUTE(A) __attribute__(A)
      |                                        ^
/home/laurynas/vilniusdb/rdb-clone-fixes/storage/rocksdb/./ha_rocksdb.h:496:41: note: in expansion of macro ‘MY_ATTRIBUTE’
  496 |                    const char *const to MY_ATTRIBUTE((__nonnull__)),
      |                                         ^~~~~~~~~~~~

Handle my_tell return value correctly. Fixes facebook#1270

The incorrect my_tell value handling was pointed out by GCC:

[ 22%] Building CXX object plugin/clone/CMakeFiles/clone.dir/src/clone_os.cc.o
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc: In function ‘int clone_os_copy_buf_to_file(uchar*, Ha_clone_file, uint, const char*)’:
/home/laurynas/vilniusdb/fb-mysql/plugin/clone/src/clone_os.cc:312:21: error: comparison of unsigned expression in ‘< 0’ is always false [-Werror=type-limits]
  312 |       if (file_size < 0) {
      |           ~~~~~~~~~~^~~
cc1plus: all warnings being treated as errors

Pull Request resolved: facebook#1271

Reviewed By: sunshine-Chun

Differential Revision: D43259662

Pulled By: hermanlee

--------------------------------------------------------------------------------

Clone: fix race condition that may skip some SST files (facebook#1274)

Summary:
A donor thread in the main copy loop took a file to send, released the donor state mutex, and then opened the file. If an ENOENT was received at this point, it was assumed that this was a stale SST file from an older checkpoint that was rolled since.

Because the donor state mutex was released between taking of the file and opening it, the following race was possible:
1) Thread 1 takes the file
2) Thread 2 decides to roll the checkpoint, the old checkpoint is deleted 3) Thread 1 tries to open the file, gets ENOENT
4) Thread 2 creates the new checkpoint, the file re-appears, but it's too late.

Rolling the checkpoint in a donor state mutex critical section is a possible fix, but such section would do a lot of I/O, serializing the parallel threads. Instead, fix by taking the file and opening it in the same critical section.

Pull Request resolved: facebook#1274

Reviewed By: sunshine-Chun

Differential Revision: D43629546

Pulled By: hermanlee

--------------------------------------------------------------------------------

MyRocks clone: add missing donor mutex unlocks on error exits (facebook#1317)

Summary: Pull Request resolved: facebook#1317

Differential Revision: D46564079
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants