Crash safe slave and master in RocksDB

Summary:
This diff has the following changes.

- Changed system tables managing replication state from InnoDB to
  RocksDB. I also converted CHAR types to VARCHAR types since it's more
  space efficient in RocksDB.

- Fixed an issue that Xid binlog event was not written at RocksDB
  commit. Without Xid event, mysql.slave_*info tables are not updated on
slaves, so crash safe slave didn't work. This was fixed by defining
rocksdb_prepare() empty function as a
handler::prepare method. Since there were two storage engines supporting
prepare() (InnoDB and RocksDB, though it's fake in RocksDB),
it was also necessary to adjust "total_ha_2pc" variable in MySQL
which checks conditions in order to avoid errors/crashes
(total_ha_2pc should be binlog + 2 -- InnoDB and RocksDB). With this
fix, slave_*info tables were updated on slaves, which was required for
slave safe slave to work (but not enough when GTID is enabled).

- Wrote binlog state (binlog name, binlog pos and optionally GTID) in
  RocksDB at transaction commit. This is treated as single Put (part of
WriteBatch), and
written together at commit. Since binlog is enabled on both master and
slaves, This is written on both master and slave.
This means on transaction commit, one additional Put (row) is
written on master, two (one for this, the other is for slave_*info)
Puts are written on slave. To write and read binlog state, I added a
utility class Binlog_info_manager and refactored some existing
functions.
 To store/read binlog state in RocksDB, I allocated a system index id
BINLOG_INFO_INDEX_NUMBER (0xfffffff0) which is not likely to be
used (unless creating 0xfffffff0 indexes). Currently there is no
maximum number of index checking but it will be definitely needed.
 Note that binlog state is directly written to RocksDB, not through
MySQL data dictionary like slave_relay_log_info. This is for
performance reason. Opening/closing additional MySQL table
is expensive. Writing directly to RocksDB avoids the overhead.

- Added rocksdb_recover() function as a handler::recover function.
It reads binlog state stored in RocksDB, and prints last committed
binlog filename, position and GTID. This makes it possible to recover
both crashed slave and master. Printed format is aligned to Facebook's
enhancement for InnoDB
(22e4a82).

- Changed variable names innodb_binlog_file and innodb_binlog_pos
in binlog.h to engine_binlog_file and engine_binlog_pos because
they are now set from both RocksDB and InnoDB.

- Added binlog state check log in rocksdb_recover() and
  innobase_xa_recover(). Both set binlog info from RocksDB/InnoDB (in
RocksDB->InnoDB ordering). Without checking anything, binlog state
fetched at rocksdb_recover() is overwritten at innobase_xa_recover().
When using RocksDB storage engine only, this means binlog state is
overwritten to very old (wrong) position.
To fix this issue, I added a utility function is_binlog_advanced()
in handlerr.h and handler.cc. rocksdb_recover() and innobase_xa_recover()
check binlog state, and if state is already set and overwrite
only new state is advanced. This prevents to overwrite with older binlog state.

- Added test cases to cover WAL crash, and modified existing crash
  safe master/slave test cases to cover RocksDB. They are now stored at rocksdb_rpl
test suite. Since RocksDB storage engine now
does not do anything at prepare(), roll forward recovery (committing
transactions that are written as binlog Xid events) doesn't happen. This
is a different behaior from InnoDB so I adjusted result files for
RocksDB.

@update-submodule: rocksdb

Test Plan: mtr --repeat=3 --suite=rocksdb and --suite=rocksdb_rpl

Reviewers: hermanlee4, jonahcohen, santoshb, maykov

Subscribers: jtolmer, MarkCallaghan

Differential Revision: https://reviews.facebook.net/D31365
Differential Revision: https://reviews.facebook.net/D48045
Differential Revision: https://reviews.facebook.net/D49221
Differential Revision: https://reviews.facebook.net/D49557
Differential Revision: https://reviews.facebook.net/D51837

Author: yoshinorim

mysql-test/extra/rpl_tests/rpl_gtid_crash_safe.inc

@@ -2,7 +2,7 @@
 
 connection master;
 -- let $uuid = `select @@server_uuid;`
-create table t1(a int, PRIMARY KEY(a)) ENGINE=INNODB;
+-- eval create table t1(a int, PRIMARY KEY(a)) ENGINE=$engine
 insert into t1 values(1);
 insert into t1 values(2);
 sync_slave_with_master;

mysql-test/extra/rpl_tests/rpl_parallel_load_innodb.test

@@ -6,7 +6,12 @@
 # load volume parameter
 #
 
-if ($let == '')
+if ($storage_engine == '')
+{
+  let $storage_engine='innodb';
+}
+
+if ($iter == '')
 {
   let $iter=20;
 }
@@ -68,8 +73,8 @@ while($i)
   eval drop database if exists test$i1;
   eval create database test$i1;
   eval use test$i1;
-  create table ti_nk (a int, b int, c text) engine=innodb;
-  create table ti_wk (a int auto_increment primary key, b int, c text) engine=innodb;
+  eval create table ti_nk (a int, b int, c varchar(36) primary key) engine=$storage_engine default charset=latin1 collate=latin1_bin;
+  eval create table ti_wk (a int auto_increment primary key, b int, c text) engine=$storage_engine;
   let $l1= $init_rows;
   while($l1)
   {
@@ -79,7 +84,7 @@ while($i)
 
   # this table is special - just for timing. It's more special on test0 db
   # where it contains master timing of the load as well.
-  create table benchmark (state text) engine=innodb; # timestamp keep on the slave side
+  eval create table benchmark (state varchar(100) primary key) engine=$storage_engine default charset=latin1 collate=latin1_bin; # timestamp keep on the slave side
 
   dec $i;
 }
@@ -278,11 +283,33 @@ let $slave_pid_file=`SELECT @@pid_file;`;
 
 
 let $wait_condition= SELECT count(*)+sleep(1) = 5 FROM test0.benchmark;
+--let slave_data_dir= query_get_value(SELECT @@DATADIR, @@DATADIR, 1)
+
 --disable_query_log
 while ($num_crashes)
 {
   SELECT sleep(2 * rand());
   exec kill -9 `head -1 $slave_pid_file`;
+
+  if ($storage_engine == 'rocksdb')
+  {
+--write_file $MYSQL_TMP_DIR/truncate_tail_wal.sh
+#!/bin/bash
+F=`ls -t $slave_data_dir/.rocksdb/*.log | head -n 1`
+SIZE=`stat -c %s $F`
+NEW_SIZE=`expr $SIZE - 100`
+truncate -s $NEW_SIZE $F
+rc=$?
+if [[ $rc != 0 ]]; then
+  exit 1
+fi
+exit 0
+EOF
+    --chmod 0755 $MYSQL_TMP_DIR/truncate_tail_wal.sh
+    --exec $MYSQL_TMP_DIR/truncate_tail_wal.sh
+    --remove_file $MYSQL_TMP_DIR/truncate_tail_wal.sh
+  }
+
   -- let $rpl_server_number=2
   -- source include/rpl_start_server.inc
   -- source include/start_slave.inc

mysql-test/r/rocksdb.result

@@ -1,6 +1,6 @@
 select * from information_schema.engines where engine = 'rocksdb';
 ENGINE	SUPPORT	COMMENT	TRANSACTIONS	XA	SAVEPOINTS
-ROCKSDB	YES	RocksDB storage engine	YES	NO	NO
+ROCKSDB	YES	RocksDB storage engine	YES	YES	NO
 drop table if exists t0,t1,t2,t3,t4,t5,t6,t7,t8,t9,t10;
 drop table if exists t11,t12,t13,t14,t15,t16,t17,t18,t19,t20;
 drop table if exists t21,t22,t23,t24,t25,t26,t27,t28,t29;

mysql-test/suite/rocksdb_rpl/combinations

@@ -0,0 +1,2 @@
+[row]
+binlog-format=row

mysql-test/suite/rocksdb_rpl/r/rpl_crash_safe_wal_corrupt.result

@@ -0,0 +1,78 @@
+include/master-slave.inc
+Warnings:
+Note	####	Sending passwords in plain text without SSL/TLS is extremely insecure.
+Note	####	Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
+[connection master]
+drop table if exists x;
+select @@binlog_format;
+@@binlog_format
+ROW
+create table x (id int primary key, value int, value2 int, index(value)) engine=rocksdb;
+insert into x values (1,1,1);
+insert into x values (2,1,1);
+insert into x values (3,1,1);
+insert into x values (4,1,1);
+insert into x values (5,1,1);
+select @@global.gtid_executed;
+@@global.gtid_executed
+
+
+--- slave state before crash ---
+select * from x;
+id	value	value2
+1	1	1
+2	1	1
+3	1	1
+4	1	1
+5	1	1
+select @@global.gtid_executed;
+@@global.gtid_executed
+
+select * from mysql.slave_gtid_info;
+Id	Database_name	Last_gtid
+include/rpl_start_server.inc [server_number=2]
+
+--- slave state after crash recovery, slave stop, one WAL entry missing ---
+select * from x;
+id	value	value2
+1	1	1
+2	1	1
+3	1	1
+4	1	1
+select @@global.gtid_executed;
+@@global.gtid_executed
+
+select * from mysql.slave_gtid_info;
+Id	Database_name	Last_gtid
+
+--- slave state after restart, slave start ---
+include/start_slave.inc
+select * from x;
+id	value	value2
+1	1	1
+2	1	1
+3	1	1
+4	1	1
+5	1	1
+select @@global.gtid_executed;
+@@global.gtid_executed
+
+select * from mysql.slave_gtid_info;
+Id	Database_name	Last_gtid
+insert into x values (6,1,1);
+select * from x;
+id	value	value2
+1	1	1
+2	1	1
+3	1	1
+4	1	1
+5	1	1
+6	1	1
+select @@global.gtid_executed;
+@@global.gtid_executed
+
+select * from mysql.slave_gtid_info;
+Id	Database_name	Last_gtid
+drop table x;
+include/rpl_end.inc
+Binlog Info Found