This repository has been archived by the owner on Feb 10, 2023. It is now read-only.
forked from percona/percona-server
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Supporting START TRANSACTION WITH CONSISTENT [ROCKSDB] SNAPSHOT
Summary: This adds two features in RocksDB. 1. Supporting START TRANSACTION WITH CONSISTENT SNAPSHOT 2. Getting current binlog position in addition to #1. With these features, mysqldump can take consistent logical backup. The second feature is done by START TRANSACTION WITH CONSISTENT ROCKSDB SNAPSHOT. This is Facebook's extension, and it works like existing START TRANSACTION WITH CONSISTENT INNODB SNAPSHOT. This diff changed some existing codebase/behaviors. - Original Facebook-MySQL always started InnoDB transaction regardless of engine clause. For example, START TRANSACTION WITH CONSISTENT MYISAM SNAPSHOT was accepted but it actually started InnoDB transaction, not MyISAM. This patch does not allow setting engine that does not support consistent snapshot. mysql> start transaction with consistent myisam snapshot; ERROR 1105 (HY000): Consistent Snapshot is not supported for this engine Currently only InnoDB and RocksDB support consistent snapshot. To check engines, I modified sql/sql_yacc.yy, trans_begin() and ha_start_consistent_snapshot() to pass handlerton. - Changed constant name from MYSQL_START_TRANS_OPT_WITH_CONS_INNODB_SNAPSHOT to MYSQL_START_TRANS_OPT_WITH_CONS_ENGINE_SNAPSHOT, because it's no longer InnoDB dependent. - When not setting engine, START TRANSACTION WITH CONSISTENT SNAPSHOT takes both InnoDB and RocksDB snapshots, and both InnoDB and RocksDB participate in transaction. When executing COMMIT, both InnoDB and RocksDB modifications are committed. Remember that XA is not supported yet, so mixing engines is not recommended anyway. - When setting engine, START TRANSACTION WITH CONSISTENT.. takes snapshot for the specified engine only. But it starts both InnoDB and RocksDB transactions. Test Plan: mtr --suite=rocksdb,rocksdb_rpl, --repeat=3 Reviewers: hermanlee4, jonahcohen, jtolmer, tian.xia, maykov, spetrunia Reviewed By: spetrunia Subscribers: steaphan Differential Revision: https://reviews.facebook.net/D32355
- Loading branch information
1 parent
061f05e
commit 37996cc
Showing
9 changed files
with
777 additions
and
0 deletions.
There are no files selected for viewing
68 changes: 68 additions & 0 deletions
68
mysql-test/suite/rocksdb.rpl/r/consistent_snapshot_mixed_engines.result
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| DROP TABLE IF EXISTS t1; | ||
| connect con1,localhost,root,,; | ||
| connect con2,localhost,root,,; | ||
| connection con1; | ||
| create table i1 (id int primary key , value int) engine=innodb; | ||
| create table r1 (id int primary key , value int) engine=rocksdb; | ||
| SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ; | ||
| START TRANSACTION WITH CONSISTENT SNAPSHOT; | ||
| connection con2; | ||
| insert into i1 values (1,1); | ||
| insert into r1 values (1,1); | ||
| connection con1; | ||
| select * from i1; | ||
| id value | ||
| select * from r1; | ||
| id value | ||
| START TRANSACTION WITH CONSISTENT ROCKSDB SNAPSHOT; | ||
| File Position Gtid_executed | ||
| master-bin.000001 1109 uuid:1-5 | ||
| connection con2; | ||
| insert into i1 values (2,2); | ||
| insert into r1 values (2,2); | ||
| connection con1; | ||
| select * from i1; | ||
| id value | ||
| 1 1 | ||
| 2 2 | ||
| select * from r1; | ||
| id value | ||
| 1 1 | ||
| connection con2; | ||
| insert into i1 values (3,2); | ||
| insert into r1 values (3,2); | ||
| connection con1; | ||
| select * from i1; | ||
| id value | ||
| 1 1 | ||
| 2 2 | ||
| select * from r1; | ||
| id value | ||
| 1 1 | ||
| START TRANSACTION WITH CONSISTENT INNODB SNAPSHOT; | ||
| File Position Gtid_executed | ||
| master-bin.000001 1997 uuid:1-9 | ||
| connection con2; | ||
| insert into r1 values (4,4); | ||
| connection con1; | ||
| select * from r1; | ||
| id value | ||
| 1 1 | ||
| 2 2 | ||
| 3 2 | ||
| 4 4 | ||
| connection con2; | ||
| insert into r1 values (5,5); | ||
| connection con1; | ||
| select * from r1; | ||
| id value | ||
| 1 1 | ||
| 2 2 | ||
| 3 2 | ||
| 4 4 | ||
| drop table i1; | ||
| drop table r1; | ||
| connection default; | ||
| disconnect con1; | ||
| disconnect con2; | ||
| reset master; |
222 changes: 222 additions & 0 deletions
222
mysql-test/suite/rocksdb.rpl/r/rpl_rocksdb_snapshot.result
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,222 @@ | ||
| include/master-slave.inc | ||
| Warnings: | ||
| Note #### Sending passwords in plain text without SSL/TLS is extremely insecure. | ||
| Note #### Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information. | ||
| [connection master] | ||
| DROP TABLE IF EXISTS t1; | ||
| # Establish connection con1 (user=root) | ||
| # Establish connection con2 (user=root) | ||
| # Establish connection con3 (user=root) | ||
| # Establish connection con4 (user=root) | ||
| # reset replication to guarantee that master-bin.000001 is used | ||
| include/stop_slave.inc | ||
| RESET SLAVE; | ||
| RESET MASTER; | ||
| RESET MASTER; | ||
| CHANGE MASTER TO master_host="127.0.0.1",master_port=MASTER_PORT,master_user="root"; | ||
| Warnings: | ||
| Note 1759 Sending passwords in plain text without SSL/TLS is extremely insecure. | ||
| Note 1760 Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information. | ||
| include/start_slave.inc | ||
| # Switch to connection con1 | ||
| CREATE TABLE t1 (a INT PRIMARY KEY) ENGINE=rocksdb; | ||
| INSERT INTO t1 VALUES(1); | ||
| SET TRANSACTION ISOLATION LEVEL READ COMMITTED; | ||
| START TRANSACTION WITH CONSISTENT SNAPSHOT; | ||
| ERROR HY000: Only REPEATABLE READ isolation level is supported for START TRANSACTION WITH CONSISTENT SNAPSHOT in RocksDB Storage Engine. | ||
| START TRANSACTION WITH CONSISTENT ROCKSDB SNAPSHOT; | ||
| ERROR HY000: Only REPEATABLE READ isolation level is supported for START TRANSACTION WITH CONSISTENT SNAPSHOT in RocksDB Storage Engine. | ||
| ROLLBACK; | ||
| SET TRANSACTION ISOLATION LEVEL REPEATABLE READ; | ||
| START TRANSACTION WITH CONSISTENT ROCKSDB SNAPSHOT; | ||
| File Position Gtid_executed | ||
| master-bin.000001 528 UUID:1-2 | ||
| # Switch to connection con2 | ||
| INSERT INTO t1 VALUES(2); | ||
| INSERT INTO t1 VALUES(3); | ||
| # Switch to connection con1 | ||
| SELECT * FROM t1; | ||
| a | ||
| 1 | ||
| COMMIT; | ||
| SELECT * FROM t1; | ||
| a | ||
| 1 | ||
| 2 | ||
| 3 | ||
| DROP TABLE t1; | ||
| # Switch to connection con1 | ||
| CREATE TABLE t1 (a INT PRIMARY KEY) ENGINE=rocksdb; | ||
| INSERT INTO t1 VALUES(1); | ||
| START TRANSACTION WITH CONSISTENT ROCKSDB SNAPSHOT; | ||
| File Position Gtid_executed | ||
| master-bin.000001 1498 UUID:1-7 | ||
| START TRANSACTION WITH CONSISTENT ROCKSDB SNAPSHOT; | ||
| File Position Gtid_executed | ||
| master-bin.000001 1498 UUID:1-7 | ||
| START TRANSACTION WITH CONSISTENT ROCKSDB SNAPSHOT; | ||
| File Position Gtid_executed | ||
| master-bin.000001 1498 UUID:1-7 | ||
| START TRANSACTION WITH CONSISTENT ROCKSDB SNAPSHOT; | ||
| File Position Gtid_executed | ||
| master-bin.000001 1498 UUID:1-7 | ||
| # Switch to connection con2 | ||
| INSERT INTO t1 VALUES(2); | ||
| INSERT INTO t1 VALUES(3); | ||
| # Switch to connection con1 | ||
| SELECT * FROM t1; | ||
| a | ||
| 1 | ||
| SELECT * INTO OUTFILE '<MYSQLTEST_VARDIR>/tmp/rpl_rocksdb_snapshot.out.file' FROM t1; | ||
| COMMIT; | ||
| # Switch to slave | ||
| CREATE TABLE t1_backup LIKE t1; | ||
| INSERT INTO t1_backup SELECT * FROM t1; | ||
| include/stop_slave.inc | ||
| RESET SLAVE; | ||
| RESET MASTER; | ||
| DELETE FROM t1; | ||
| LOAD DATA INFILE '<MYSQLTEST_VARDIR>/tmp/rpl_rocksdb_snapshot.out.file' INTO TABLE t1; | ||
| SELECT * FROM t1; | ||
| a | ||
| 1 | ||
| CHANGE MASTER TO master_host="127.0.0.1",master_port=MASTER_PORT,master_user="root",master_log_file="master-bin.000001",master_log_pos=binlog_pos; | ||
| Warnings: | ||
| Note 1759 Sending passwords in plain text without SSL/TLS is extremely insecure. | ||
| Note 1760 Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information. | ||
| include/start_slave.inc | ||
| SELECT * FROM t1; | ||
| a | ||
| 1 | ||
| 2 | ||
| 3 | ||
| SELECT * FROM t1_backup; | ||
| a | ||
| 1 | ||
| 2 | ||
| 3 | ||
| DROP TABLE t1_backup; | ||
| DROP TABLE t1; | ||
| # Switch to connection con1 | ||
| CREATE TABLE t1 (a INT PRIMARY KEY) ENGINE=rocksdb; | ||
| INSERT INTO t1 VALUES(1); | ||
| # async queries from con2 | ||
| INSERT INTO t1 VALUES(2); | ||
| # async queries from con3 | ||
| INSERT INTO t1 VALUES(21); | ||
| # Switch to connection con1 | ||
| # Switch to connection con4 | ||
| INSERT INTO t1 VALUES(9); | ||
| # Switch to connection con1 | ||
| SELECT * INTO OUTFILE '<MYSQLTEST_VARDIR>/tmp/rpl_rocksdb_snapshot.out.file' FROM t1; | ||
| COMMIT; | ||
| # reap async statements | ||
| # Switch to slave | ||
| CREATE TABLE t1_backup LIKE t1; | ||
| INSERT INTO t1_backup SELECT * FROM t1; | ||
| include/stop_slave.inc | ||
| RESET SLAVE; | ||
| RESET MASTER; | ||
| DELETE FROM t1; | ||
| LOAD DATA INFILE '<MYSQLTEST_VARDIR>/tmp/rpl_rocksdb_snapshot.out.file' INTO TABLE t1; | ||
| CHANGE MASTER TO master_host="127.0.0.1",master_port=MASTER_PORT,master_user="root",master_log_file="master-bin.000001",master_log_pos=binlog_pos; | ||
| Warnings: | ||
| Note 1759 Sending passwords in plain text without SSL/TLS is extremely insecure. | ||
| Note 1760 Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information. | ||
| include/start_slave.inc | ||
| # sync and then query slave | ||
| ShouldBeZero | ||
| 0 | ||
| DROP TABLE t1_backup; | ||
| DROP TABLE t1; | ||
| # Switch to connection con1 | ||
| CREATE TABLE t1 (a INT PRIMARY KEY) ENGINE=rocksdb; | ||
| INSERT INTO t1 VALUES(1); | ||
| START TRANSACTION WITH CONSISTENT ROCKSDB SNAPSHOT; | ||
| File Position Gtid_executed | ||
| master-bin.000001 3655 UUID:1-18 | ||
| START TRANSACTION WITH CONSISTENT ROCKSDB SNAPSHOT; | ||
| File Position Gtid_executed | ||
| master-bin.000001 3655 UUID:1-18 | ||
| START TRANSACTION WITH CONSISTENT ROCKSDB SNAPSHOT; | ||
| File Position Gtid_executed | ||
| master-bin.000001 3655 UUID:1-18 | ||
| START TRANSACTION WITH CONSISTENT ROCKSDB SNAPSHOT; | ||
| File Position Gtid_executed | ||
| master-bin.000001 3655 UUID:1-18 | ||
| # Switch to connection con2 | ||
| INSERT INTO t1 VALUES(2); | ||
| INSERT INTO t1 VALUES(3); | ||
| # Switch to connection con1 | ||
| SELECT * FROM t1; | ||
| a | ||
| 1 | ||
| SELECT * INTO OUTFILE '<MYSQLTEST_VARDIR>/tmp/rpl_rocksdb_snapshot.out.file' FROM t1; | ||
| COMMIT; | ||
| # Switch to slave | ||
| CREATE TABLE t1_backup LIKE t1; | ||
| INSERT INTO t1_backup SELECT * FROM t1; | ||
| include/stop_slave.inc | ||
| RESET SLAVE; | ||
| RESET MASTER; | ||
| SET @@global.gtid_purged='gtid_executed_from_snapshot'; | ||
| DELETE FROM t1; | ||
| LOAD DATA INFILE '<MYSQLTEST_VARDIR>/tmp/rpl_rocksdb_snapshot.out.file' INTO TABLE t1; | ||
| SELECT * FROM t1; | ||
| a | ||
| 1 | ||
| CHANGE MASTER TO master_host="127.0.0.1",master_port=MASTER_PORT,master_user="root", master_auto_position=1; | ||
| Warnings: | ||
| Note 1759 Sending passwords in plain text without SSL/TLS is extremely insecure. | ||
| Note 1760 Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information. | ||
| include/start_slave.inc | ||
| SELECT * FROM t1; | ||
| a | ||
| 1 | ||
| 2 | ||
| 3 | ||
| SELECT * FROM t1_backup; | ||
| a | ||
| 1 | ||
| 2 | ||
| 3 | ||
| DROP TABLE t1_backup; | ||
| DROP TABLE t1; | ||
| # Switch to connection con1 | ||
| CREATE TABLE t1 (a INT PRIMARY KEY) ENGINE=rocksdb; | ||
| INSERT INTO t1 VALUES(1); | ||
| # async queries from con2 | ||
| INSERT INTO t1 VALUES(2); | ||
| # async queries from con3 | ||
| INSERT INTO t1 VALUES(21); | ||
| # Switch to connection con1 | ||
| # Switch to connection con4 | ||
| INSERT INTO t1 VALUES(9); | ||
| # Switch to connection con1 | ||
| SELECT * INTO OUTFILE '<MYSQLTEST_VARDIR>/tmp/rpl_rocksdb_snapshot.out.file' FROM t1; | ||
| COMMIT; | ||
| # reap async statements | ||
| # Switch to slave | ||
| CREATE TABLE t1_backup LIKE t1; | ||
| INSERT INTO t1_backup SELECT * FROM t1; | ||
| include/stop_slave.inc | ||
| RESET SLAVE; | ||
| RESET MASTER; | ||
| SET @@global.gtid_purged='gtid_executed_from_snapshot'; | ||
| DELETE FROM t1; | ||
| LOAD DATA INFILE '<MYSQLTEST_VARDIR>/tmp/rpl_rocksdb_snapshot.out.file' INTO TABLE t1; | ||
| CHANGE MASTER TO master_host="127.0.0.1",master_port=MASTER_PORT,master_user="root", master_auto_position=1; | ||
| Warnings: | ||
| Note 1759 Sending passwords in plain text without SSL/TLS is extremely insecure. | ||
| Note 1760 Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information. | ||
| include/start_slave.inc | ||
| # sync and then query slave | ||
| ShouldBeZero | ||
| 0 | ||
| DROP TABLE t1_backup; | ||
| DROP TABLE t1; | ||
| # Switch to connection default + close connections con1 and con2 | ||
| include/stop_slave.inc | ||
| CHANGE MASTER to master_auto_position=0; | ||
| include/start_slave.inc | ||
| include/rpl_end.inc |
15 changes: 15 additions & 0 deletions
15
mysql-test/suite/rocksdb.rpl/r/rpl_rocksdb_snapshot_without_gtid.result
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| include/master-slave.inc | ||
| Warnings: | ||
| Note #### Sending passwords in plain text without SSL/TLS is extremely insecure. | ||
| Note #### Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information. | ||
| [connection master] | ||
| create table t1(a int); | ||
| FLUSH LOGS; | ||
| insert into t1 values(1); | ||
| insert into t1 values(2); | ||
| FLUSH LOGS; | ||
| START TRANSACTION WITH CONSISTENT ROCKSDB SNAPSHOT; | ||
| File Position Gtid_executed | ||
| master-bin.000003 120 | ||
| drop table t1; | ||
| include/rpl_end.inc |
1 change: 1 addition & 0 deletions
1
mysql-test/suite/rocksdb.rpl/t/consistent_snapshot_mixed_engines-master.opt
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| --gtid_mode=ON --enforce_gtid_consistency --log_bin --log_slave_updates |
79 changes: 79 additions & 0 deletions
79
mysql-test/suite/rocksdb.rpl/t/consistent_snapshot_mixed_engines.test
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,79 @@ | ||
| --source include/have_log_bin.inc | ||
| --enable_connect_log | ||
| -- let $uuid = `select @@server_uuid;` | ||
|
|
||
| # Save the initial number of concurrent sessions | ||
| --source include/count_sessions.inc | ||
|
|
||
| --disable_warnings | ||
| DROP TABLE IF EXISTS t1; | ||
| --enable_warnings | ||
|
|
||
| connect (con1,localhost,root,,); | ||
| connect (con2,localhost,root,,); | ||
|
|
||
| connection con1; | ||
| create table i1 (id int primary key , value int) engine=innodb; | ||
| create table r1 (id int primary key , value int) engine=rocksdb; | ||
|
|
||
|
|
||
| SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ; | ||
|
|
||
| # Without setting engine, this takes both InnoDB and RocksDB snapshots | ||
| -- replace_result $uuid uuid | ||
| START TRANSACTION WITH CONSISTENT SNAPSHOT; | ||
|
|
||
| connection con2; | ||
| insert into i1 values (1,1); | ||
| insert into r1 values (1,1); | ||
|
|
||
| connection con1; | ||
| select * from i1; | ||
| select * from r1; | ||
|
|
||
| # This takes RocksDB snapshot only but both InnoDB participates in transaction. | ||
| -- replace_result $uuid uuid | ||
| START TRANSACTION WITH CONSISTENT ROCKSDB SNAPSHOT; | ||
|
|
||
| connection con2; | ||
| insert into i1 values (2,2); | ||
| insert into r1 values (2,2); | ||
|
|
||
| connection con1; | ||
| # takes InnoDB snapshot here so changes after that not visible | ||
| select * from i1; | ||
| select * from r1; | ||
|
|
||
| connection con2; | ||
| insert into i1 values (3,2); | ||
| insert into r1 values (3,2); | ||
|
|
||
| connection con1; | ||
| select * from i1; | ||
| select * from r1; | ||
|
|
||
| # RocksDB also partipates in transaction | ||
| -- replace_result $uuid uuid | ||
| START TRANSACTION WITH CONSISTENT INNODB SNAPSHOT; | ||
|
|
||
| connection con2; | ||
| insert into r1 values (4,4); | ||
|
|
||
| connection con1; | ||
| # takes RocksDB snapshot here so changes after that are not visible | ||
| select * from r1; | ||
|
|
||
| connection con2; | ||
| insert into r1 values (5,5); | ||
|
|
||
| connection con1; | ||
| select * from r1; | ||
|
|
||
| drop table i1; | ||
| drop table r1; | ||
|
|
||
| connection default; | ||
| disconnect con1; | ||
| disconnect con2; | ||
| reset master; | ||
| --source include/wait_until_count_sessions.inc |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| --gtid_mode=ON --enforce_gtid_consistency --log_bin --log_slave_updates |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| --gtid_mode=ON --enforce_gtid_consistency --log_bin --log_slave_updates |
Oops, something went wrong.