PS 7921 [8.0]: Merge percona-202104 and percona-202105 #4496
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
inikep
reviewed
Oct 18, 2021
inikep
reviewed
Oct 18, 2021
inikep
reviewed
Oct 18, 2021
inikep
reviewed
Oct 18, 2021
inikep
reviewed
Oct 18, 2021
ldonoso
force-pushed
the
PS-7921
branch
3 times, most recently
from
290d629
to
745d809
Compare
ldonoso
force-pushed
the
PS-7921
branch
4 times, most recently
from
0eae182
to
29c2db5
Compare
inikep
requested changes
Oct 28, 2021
| @@ -0,0 +1 @@ | |||
| --rocksdb_rate_limiter_bytes_per_sec=256k --rocksdb_default_cf_options=write_buffer_size=64k;target_file_size_base=64k;max_bytes_for_level_base=1m;compression_per_level=kNoCompression;max_compaction_bytes=256k | |||
There was a problem hiding this comment.
In any of the myrocks .opt files we need to prefix all myrocks options with --loose for the cases/platforms where myrocks isn't built but the default suites are run
(from https://confluence.percona.com/display/PS/MyRocks+Merges)
| @@ -0,0 +1,2 @@ | |||
| --rocksdb_default_cf_options=write_buffer_size=128m;target_file_size_base=32m;max_bytes_for_level_base=512m;level0_file_num_compaction_trigger=4;level0_slowdown_writes_trigger=20;level0_stop_writes_trigger=30;max_write_buffer_number=4;compression_per_level=kLZ4Compression;bottommost_compression=kZSTD;compression_opts=-14:6:0;block_based_table_factory={cache_index_and_filter_blocks=1;filter_policy=bloomfilter:10:false;whole_key_filtering=0};prefix_extractor=capped:12;level_compaction_dynamic_level_bytes=true;optimize_filters_for_hits=true;memtable_prefix_bloom_size_ratio=0.039;max_compaction_bytes=402653184;report_bg_io_stats=true;compaction_pri=kMinOverlappingRatio;soft_pending_compaction_bytes_limit=20480000000 | |||
There was a problem hiding this comment.
In any of the myrocks .opt files we need to prefix all myrocks options with --loose for the cases/platforms where myrocks isn't built but the default suites are run
| @@ -0,0 +1,3 @@ | |||
| --initialize --default_authentication_plugin=mysql_native_password | |||
| --rocksdb_default_cf_options=write_buffer_size=128m;target_file_size_base=32m;max_bytes_for_level_base=512m;level0_file_num_compaction_trigger=4;level0_slowdown_writes_trigger=20;level0_stop_writes_trigger=30;max_write_buffer_number=4;compression_per_level=kLZ4Compression;bottommost_compression=kZSTD;compression_opts=-14:6:0;block_based_table_factory={cache_index_and_filter_blocks=1;filter_policy=bloomfilter:10:false;whole_key_filtering=0};prefix_extractor=capped:12;level_compaction_dynamic_level_bytes=true;optimize_filters_for_hits=true;memtable_prefix_bloom_size_ratio=0.039;max_compaction_bytes=402653184;report_bg_io_stats=true;compaction_pri=kMinOverlappingRatio;soft_pending_compaction_bytes_limit=20480000000 | |||
There was a problem hiding this comment.
In any of the myrocks .opt files we need to prefix all myrocks options with --loose for the cases/platforms where myrocks isn't built but the default suites are run
Comment on lines
19
to
22
| --disable_warnings | ||
| DROP TABLE IF EXISTS t1; | ||
| --enable_warnings |
There was a problem hiding this comment.
Can be changed to
drop table t1;
in 8.0.
There was a problem hiding this comment.
@inikep , please, mind if I do that I get the following error:
CURRENT_TEST: rocksdb.cancel_mc
mysqltest: At line 21: Query 'DROP TABLE t1' failed.
ERROR 1051 (42S02): Unknown table 'test.t1'
There was a problem hiding this comment.
Sorry, drop table t1; can be removed because mysql-test/include/check-testcase.test makes sure that there is no leftovers from previous tests. You may leave it here but please remove it next time.
Comment on lines
1
to
19
| eval | ||
| CREATE TABLE t (i varchar(64), j varchar(64), k varchar(64), l varchar(64), | ||
| PRIMARY KEY (i, j), | ||
| KEY ik1 (i, k) COMMENT 'cfname=$cfname;partial_group_keyparts=1;partial_group_threshold=5', | ||
| KEY ik2 (i, k) COMMENT 'cfname=$cfname' | ||
| ) ENGINE=ROCKSDB; | ||
|
|
||
| INSERT INTO t values ("1", "1", "2", "1"); | ||
| INSERT INTO t values ("1", "2", "1", "1"); | ||
|
|
||
| INSERT INTO t values ("11111111", "1", "9", "1"); | ||
| INSERT INTO t values ("11111111", "2", "8", "1"); | ||
| INSERT INTO t values ("11111111", "3", "7", "1"); | ||
| INSERT INTO t values ("11111111", "4", "5", "1"); | ||
| INSERT INTO t values ("11111111", "5", "4", "1"); | ||
| INSERT INTO t values ("11111111", "6", "2", "1"); | ||
|
|
||
| INSERT INTO t values ("111111111", "1", "9", "1"); | ||
| INSERT INTO t values ("111111111", "2", "2", "1"); |
There was a problem hiding this comment.
The .inc files should be introduced into "rocksdb/include" and not "rocksdb/t"
| @@ -0,0 +1,79 @@ | |||
| SET rocksdb_bulk_load_allow_sk=1; | |||
There was a problem hiding this comment.
The .inc files should be introduced into "rocksdb/include" and not "rocksdb/t"
| @@ -0,0 +1,95 @@ | |||
| --let $binary_id1=1 | |||
There was a problem hiding this comment.
The .inc files should be introduced into "rocksdb/include" and not "rocksdb/t"
Comment on lines
+42
to
+41
| --connect (con1,localhost,root,,) | ||
| --connect (con2,localhost,root,,) |
There was a problem hiding this comment.
Add --source include/count_sessions.inc and --source include/wait_until_count_sessions.inc to check if all connections were closed.
Comment on lines
+3
to
+5
| --connect (con1,localhost,root,,) | ||
| --connect (con2,localhost,root,,) |
There was a problem hiding this comment.
Add --source include/count_sessions.inc and --source include/wait_until_count_sessions.inc to check if all connections were closed
ldonoso
force-pushed
the
PS-7921
branch
2 times, most recently
from
c2812a2
to
8594759
Compare
inikep
reviewed
Oct 29, 2021
storage/rocksdb/ha_rocksdb.cc
Outdated
| return tx->set_status_error(tx->get_thd(), s, kd, tbl_def); | ||
| } | ||
|
|
||
| // TODOLUIS >>>>>>> 045ead26e81... Implement iterator class |
There was a problem hiding this comment.
I can see it in commit history, no big deal.
ldonoso
force-pushed
the
PS-7921
branch
4 times, most recently
from
4e4773f
to
c5a7be9
Compare
ldonoso
changed the title
added 4 commits
November 2, 2021 17:56
Upstream commit ID : fb-mysql-5.6.35/5b302cbdd16553f9f163c98fe44fa63e41dff341 PS-7921 : Merge percona-202104 Summary: In MyRocksOnWS scenarios, RocksDB will place the data corruption marker locally using rdb_persist_corruption_marker and check locally with rdb_check_rocksdb_corruption. While this would work in theory, ideally we want the data corruption markers to be in RocksDB data folder in WS and not locally, so that the data is self-contained. This involves using RocksDB Env API to create / access the file instead of using MySQL API. Reviewed By: yizhang82 Differential Revision: D28605924 fbshipit-source-id: 4e930de43e5
Upstream commit ID : fb-mysql-5.6.35/a736771d4a7d1c90448c7bbfb352a10c684c16e4 PS-7921 : Merge percona-202104 Summary: In MyRocksOnWS scenarios, MyRocks writes bulk load tmp file into WS (through SSTFileWriter) but the cleanup code in Rdb_sst_info::init still assumes local directory and will enumerate from local directory to delete garbage temp files that weren't cleaned up during a crash. This would leave files in WS around and end up not cleaning anything. This involves using RocksDB Env API to create / access the file instead of using MySQL API, and should work regardless whether it is WS or local disk. Reviewed By: yizhang82 Differential Revision: D28680122 fbshipit-source-id: 1c95319f69b
Upstream commit ID : fb-mysql-5.6.35/15d69687142bdf5efa7b999504ef7a390fdf59e3 PS-7921 : Merge percona-202104 Summary: SKIP LOCKED is very similar with TTL / ICP that we need to skip locked rows and proceed to the next one. 5.6 didn't have correct implementation even for TTL in many cases (index_last / index_first / index_read_map_impl) but in 8.0 Manuel's refactoring made things much easier. Note that when skipping locked rows, we also skip rows that can cause deadlock and snapshot conflict in the spirit of SKIP LOCKED in order to lock the rows that we can update later. NOWAIT is quite straight-forward - just set timeout to 0 and restore it after we are done. We still return ER_LOCK_WAIT_TIMEOUT instead of ER_LOCK_NOWAIT (in InnoDB) if we couldn't take the lock and personally I don't think it makes much difference. Reviewed By: lth Differential Revision: D28922476 fbshipit-source-id: 1bb698a2bda Percona changes: As RocksDB in Percona Server does not implement gap locks, the isolation level `READ_COMMITTED` was used in the tests. This change affected the result of the tests wrt the seconday key.
Upstream commit ID : fb-mysql-5.6.35/6a279bc3c6a05424e7a99660908849b3a0bbe8aa PS-7921 : Merge percona-202104 Summary: This adds a test case for RENAME TABLE across different database and then dropping the database - this is the scenario that triggers bug in db_number change. This makes sure our metadata works correctly in these cases regardless of format. Reviewed By: hermanlee Differential Revision: D28922167 fbshipit-source-id: afa3304b204
added 15 commits
November 2, 2021 17:56
Upstream commit ID : fb-mysql-5.6.35/25fa8acf4c0e4076af16b500c6199876dbb87812 PS-7921 : Merge percona-202104 Summary: As titled to update the rocksdb submodule. update-submodule: rocksdb Reviewed By: pradeep1288 Differential Revision: D29040681 fbshipit-source-id: 02cdc2cf1d2
Upstream commit ID : fb-mysql-5.6.35/b659d0f2524d22385ad36fda61bbe05a9a80b182 PS-7921 : Merge percona-202104 Summary: This test often failed on asan debug because the third reduction was often an increase in size. Add some sleep calls to give rocksdb time to complete the sst compactions so that the table reductions are realized. Reviewed By: luqun Differential Revision: D29044209 fbshipit-source-id: cb8f4565a60
Upstream commit ID : fb-mysql-5.6.35/8dfbf7987848f70c0c48b3e97e01c535f70b299c PS-7921 : Merge percona-202105 Summary: Rdb_transactions are added and removed from the global list by the Rdb_transaction constructor/destructors. However, information_schema.rocksdb_trx queries scan this global list and calls the virtual function is_writebatch_trx() to determine how to cast the object. It is possible for the subclasses destructors to have completed, but before Rdb_transaction's destructor can remove the object from the global list that an information_schema.rocksdb_trx query is initiated. The query walks the list of transactions and comes across a partially deconstructed Rdb_transaction_impl object. When it calls is_writebatch_trx() on this object, it triggers a pure function access crash because the function has already been removed/cleared. Fix this by adding the objects to the global list after the object is fully constructed and remove it from the global list when the object enters the destructor. Reviewed By: yizhang82 Differential Revision: D29164519 fbshipit-source-id: aaf39f330c8
Upstream commit ID : fb-mysql-5.6.35/f35c50e28dc68d831930ef2d49e5d98fc603d05a PS-7921 : Merge percona-202105 Summary: Today for secondary keys where we have the full key, we are still using an iterator to read the key when a point query would suffice. This is more efficient, but also makes the bloom filter code cleaner. Reviewed By: hermanlee Differential Revision: D25906968 fbshipit-source-id: cff7d9022e5
Upstream commit ID : fb-mysql-5.6.35/045ead26e815218e8e135eb06ebecfe01a3c9ffa PS-7921 : Merge percona-202105 Summary: This abstracts basic iteration over keys (with TTL filtering) into the `Rdb_iterator_base` class. Logic that does ICP or primary key locking is not included. Reviewed By: hermanlee Differential Revision: D25933169 fbshipit-source-id: 0815fc948fe
Upstream commit ID : fb-mysql-5.6.35/e6bb5f208bd7d9383105c0c63517838c35f3fdb4 PS-7921 : Merge percona-202105 Summary: This adds support for parsing the comment section on secondary keys for partial index keywords. This is then populated onto the `Rdb_key_def` structure. This isn't persisted to the data dictionary because we don't really need this information for compaction (unlike some of our TTL related fields). Reviewed By: hermanlee Differential Revision: D25933175 fbshipit-source-id: b1347d759c1
Upstream commit ID : fb-mysql-5.6.35/d054548154184f51ce297fc033e68b61aed71f3e PS-7921 : Merge percona-202105 Summary: Instead of leaving an index empty after bulk loading a table, we can prematerialize groups that are over the materialization threshold. Reviewed By: hermanlee Differential Revision: D27344590 fbshipit-source-id: 3cd5f7dcc70
Upstream commit ID : fb-mysql-5.6.35/3a8fcccc599b306ed907db6bda107ac3488cf50c PS-7921 : Merge percona-202105 Percona: The original commit does not belong to tag percona-202105. It is a fix for f35c50e, which belongs to percona-202105. Summary: This fixes facebook/mysql-5.6#1174 For secondary keys, it is not sufficient to look at keypart_map to determine whether the full is passed in or not, since the sql layer sometimes passes in `HA_WHOLE_KEY` even when it does not include extended keys. We already have handling for it, and we only do point lookups when the secondary key includes extended keys as well. However, we were always setting `m_full_key_lookup` even if we end up not doing a point lookup. Reviewed By: yoshinorim Differential Revision: D31862865 fbshipit-source-id: be77d1199f6
Upstream commit ID : fb-mysql-5.6.35/7deb48465e0471073514f0490837170f2b731803 PS-7921 : Merge percona-202105 Summary: 6.22.fb is the latest RocksDB release, aligned to fb-MySQL 5.6. update-submodule: rocksdb Reviewed By: yizhang82 Differential Revision: D29476464 fbshipit-source-id: 85ea846d8b0
Upstream commit ID : fb-mysql-5.6.35/b6af72667a11e7c184ee97a154e7d323006fe098 PS-7921 : Merge percona-202105 Summary: If the iterator returns any error, we should break out immediately, as we would get invalid results if we called m_iterator->key/value afterwards. Reviewed By: hermanlee Differential Revision: D29423205 fbshipit-source-id: 390798520f2
Upstream commit ID : fb-mysql-5.6.35/63ab55f35f9780ecef55dcba8b36475f5c4cb073 PS-7921 : Merge percona-202105 Summary: Fix a race condition during deletes in partial indexes. Currently we're not locking the prefix during deletes, which means that we could end up materializing deleted rows. For example, if we have: 1. T1 deletes a row A without committing 2. T2 reads row A during materialization, and generates SK for row A 3. T1 commits 4. T2 commits Then the partial index will contain an extra row that shouldn't exist. The solution is to also add locking there (similar to insert path). Fix the stress test to catch this case, since previously, it was not surfacing errors correctly. Also, create a new iterator object during materialization, since it is possible that the old iterator may be on an old superversion, and may not see all keys. Reviewed By: hermanlee Differential Revision: D29683427 fbshipit-source-id: ad8c1b7d4e9
Upstream commit ID : fb-mysql-5.6.35/3518e50dacfe641c7a672a61b5c61cd717cdc87e PS-7921 : Merge percona-202105 Summary: This diff introduces two features to cancel ongoing manual compactions. This diff has a dependency on RocksDB 6.22. 1. Client sessions running manual compactions by "SET GLOBAL rocksdb_compact_cf.." can be cancelled, like by Control+C (from the session) or KILL (from other sessions). 2. Introduced a global variable rocksdb_cancel_manual_compactions to cancel all ongoing manual compactions. Prior to this diff, only way to cancelling ongoing manual compaction was shutting down the MySQL instance. Now you can cancel specific, potentially expensive, compactions online. 1 is a safer way to cancel specific manual compaction. 2 is a global scope and cancels all internally triggered manual compactions as well, like OPTIMIZE TABLE, dropping indexes. OPTIMIZE TABLE fails by cancelling global manual compaction, but it does not make the database inconsistent. Dropping indexes will resume shortly after, so it will not keep stale data eventually. This diff was ported from 5.6 6b4c3016076312d64303abb904a0b4c278c8b0f5 and edited for 8.0 (e.g. using scoped guard). Reviewed By: hermanlee Differential Revision: D29559908 fbshipit-source-id: 4470388f917
Upstream commit ID : fb-mysql-5.6.35/4ff643f5e47bb873cd8f7e19bfbf01c9f06e2bc5 PS-7921 : Merge percona-202105 Summary: Updating the submodule to 6.23.fb update-submodule: rocksdb Reviewed By: yoshinorim Differential Revision: D29866079 fbshipit-source-id: 804b1186c10
Upstream commit ID : None PS-7921 : Merge percona-202105 Summary: - The column families are configured properly. - Test finishes orderly.
Upstream commit ID : fb-mysql-5.6.35/7d1be2321e2 PS-7921 : Merge percona-202105 Summary: The testcase sometime fail due to table t2 data length value is much smaller than table t1 data length value. since table t2 data length value is calculated in L2 sst file while table t1 data length value is calculated in L1 sst file. BTW, I think the testcase doesn't make sense. since table.data_length(and row number) value isn't sampled. only distinct key(cardinality) is sampled. Thus we should test cardinality value instead of table.data_length value. Reviewed By: lth Differential Revision: D30506323 fbshipit-source-id: 3016234435f
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
https://jira.percona.com/browse/PS-7921
https://ps80.cd.percona.com/job/percona-server-8.0-param/1081/ (baseline)
https://ps80.cd.percona.com/job/percona-server-8.0-param/1092/
https://ps80.cd.percona.com/job/percona-server-8.0-pipeline/34157/ (ASAN)
https://github.com/facebook/mysql-5.6/tree/percona-202104
https://github.com/facebook/mysql-5.6/tree/percona-202105
percona-202104
Update rocksdb submodule to bfb9496af09 Modify script which generates TARGETS (#8366) (v6.21.1)
Support SELECT FOR UPDATE SKIP LOCKED / NOWAIT
Cherry-pick the following commits from
percona-202104:5b302cb Using RocksDB Env API to create / access the corruption marker file.
a736771 Delete garbage temp sst files using RocksDB Env API
15d6968 Support SELECT FOR UPDATE SKIP LOCKED / NOWAIT
6a279bc Add RENAME TABLE test for x-database RENAME and DROP DATABASE
25fa8ac Update rocksdb submodule to 6.21.1
b659d0f Reduce flakiness of rocksdb.optimize_table
Null cherry-picked RocksDB upstream commits:
45a8f6b add mrr partition support
90fd80d Use upstream method of forcing group by plan
Null cherry-picked upstream commits:
24d89cf Fix assertion during SHOW ENGINE
16b2705 fix flaky raft testcase
b0df78d Fix heap overflow in group_relay_log_name handling
a9f3a2f This is the 8.0 port of D28735624
4c6fbb7 Use DBUG_TRACE instead of DBUG_ENTER
a1ad6e1 Add bounds checks to response attributes
337fcca Backport BUG#32113015 - WHEN THE INPUT OF NET_LENGTH_SIZE IS 251, THE RETURN SHOULD BE 3
125380f check for sql_bin_log = 0 with write throttling
82efb8e fix memory during MYSQL_BIN_LOG::open_existing_binlog
241994a add more timeout for asandebug shutdown
cb7ea1a Port COMMIT wait event and admission_control_multiquery_filter to 8.0
4f7e9fd Add valgrind suppression of 'scope array' in dlclose()
b6242ec Using locks in Dump_log methods only when raft is enabled
abb9f51 Port D27349016 Add low HLC boundary support for transactions to 8.0.20
87f9846 Porting D28003281 "Upper HLC bound support" to 8.0.20.
1a7a8ce leader election to be a sync point in the new leader
c9c2dbe Track out of order hlc value during slave apply
d456ee6 Add integer_digits.h to installed extra headers
8701973 Add optimizer hint to force group by plans
ae7527c add new test fault macros reset_hlc_for_tests
6904d7e Error out SQL thread on out of order opids in raft logs
9fbe1c1 Consider limit when costing group-by plans
d11fc6b Support commit order deadlocks resolution in dependency replication
f7192b6 Fallback to TBL mode instead of syncing trxs when errors are encountered while parsing keys
ed02e3f Fixing rpl_raft_out_of_order_raft_log test
5473299 Limit memory allocated for FTS table sync messages
1deffd7 SHOW SLAVE STATUS shows real error
0d46cc9 Fixing rpl_raft_out_of_order_raft_log test again
ab824f9 Add variable to disable events_wait_summary_by_thread_by_event_name to reduce 40M per instance
46e095c Backport Bug #31549724: MYSQL --BINARY-AS-HEX PRINTS NULL AS EMPTY STRING
percona-202105
Update rocksdb submodule to c04a86a0e9a Update HISTORY.md and version.h 6.23 release (v6.23.fb)
New functionality:
New MyRocks variables:
rocksdb-partial-index-sort-max-memrocksdb-bulk-load-partial-indexrocksdb_cancel_manual_compactionsSupporting cancelling manual compactions
Cherry-pick the following commits from
percona-202105:8dfbf79 MyRocks crashes on pure virtual function when querying rocksdb_trx list
f35c50e Use Get for secondary key point lookups
045ead2 Implement iterator class
e6bb5f2 Support parsing index comments for partial indexes
0249a96 Add partial index iterator
d054548 Prematerialize partial indexes during bulk loading
7deb484 update rocksdb submodule to 6.22.fb
b6af726 Fix error handling during iteration
63ab55f Fix locking in partial indexes
3518e50 Supporting cancelling manual compactions
4ff643f Updating RocksDB submodule to 6.23.fb
New commits:
3a8fccc Fix incorrect setting of m_full_key_lookup
[Fix rocksdb.non_blocking_manual_compaction]
7d1be23 Fix incorrect setting of m_full_key_lookup
Null cherry-picked RocksDB upstream commits:
47b45a2 Fix two bugs in bypass
d7fe2e2 Move myrocks settings to my.cnf
Null cherry-picked upstream commits:
d719f01 Add bounds checks to all resp attr code
354b67c Add MTR tests for privacy plugin
582f9c9 fix show engine trx stats
09220b3 Port MTR tests for COMMIT wait event and admission_control_multiquery_filter
1721ff6 Add privacy test suite into default suite list
21fcf7b port dscp_on_socket to 8.0
5f43db5 remove rpl_dscp_on_socket
8ee0867 Suppress valgrind issue with rocksdb CacheEntryStats
0931a17 always use lz4_pic.a for libmysqlclient
d1f8102 Fix sending server_cpu as part of query response attributes
e21b92b Generate code coverage using suite option
54365ee Remove privacy test suite from default suites
bcd67a2 add GET_COMMITTED_GTIDS for raft
bf8b761 raft: skip mts_recovery_groups during start slave
184e729 Revert SHARED_LIB_MAJOR_VERSION back
cd90913 Port mysql_change_user_nonblocking from 8.0.17 to 8.0.20
9daef99 Fix table names in COLUMN_STATISTICS
7e48040 Fix valgrind suppression for thread pool
95431ae Fix valgrind error on rpl.rpl_io_thd_wait_for_disk_space
e2fdc59 Fix invalid MDL wait timeout in replication threads
d394360 Backport BUG#29169193 MAIN.LOG_BUFFERED-BIG FAILS SPORADICALLY ON PB2