MDEV-11782: Redefine the innodb_encrypt_log format

Write only one encryption key to the checkpoint page.
Use 4 bytes of nonce. Encrypt more of each redo log block,
only skipping the 4-byte field LOG_BLOCK_HDR_NO which the
initialization vector is derived from.

Issue notes, not warning messages for rewriting the redo log files.

recv_recovery_from_checkpoint_finish(): Do not generate any redo log,
because we must avoid that before rewriting the redo log files, or
otherwise a crash during a redo log rewrite (removing or adding
encryption) may end up making the database unrecoverable.
Instead, do these tasks in innobase_start_or_create_for_mysql().

Issue a firm "Missing MLOG_CHECKPOINT" error message. Remove some
unreachable code and duplicated error messages for log corruption.

LOG_HEADER_FORMAT_ENCRYPTED: A flag for identifying an encrypted redo
log format.

log_group_t::is_encrypted(), log_t::is_encrypted(): Determine
if the redo log is in encrypted format.

recv_find_max_checkpoint(): Interpret LOG_HEADER_FORMAT_ENCRYPTED.

srv_prepare_to_delete_redo_log_files(): Display NOTE messages about
adding or removing encryption. Do not issue warnings for redo log
resizing any more.

innobase_start_or_create_for_mysql(): Rebuild the redo logs also when
the encryption changes.

innodb_log_checksums_func_update(): Always use the CRC-32C checksum
if innodb_encrypt_log. If needed, issue a warning
that innodb_encrypt_log implies innodb_log_checksums.

log_group_write_buf(): Compute the checksum on the encrypted
block contents, so that transmission errors or incomplete blocks can be
detected without decrypting.

Rewrite most of the redo log encryption code. Only remember one
encryption key at a time (but remember up to 5 when upgrading from the
MariaDB 10.1 format.)

Author: dr-m

mysql-test/mysql-test-run.pl

@@ -4345,10 +4345,7 @@ ($$)
      qr/error .*connecting to master/,
      qr/InnoDB: Error: in ALTER TABLE `test`.`t[12]`/,
      qr/InnoDB: Error: table `test`.`t[12]` .*does not exist in the InnoDB internal/,
-     qr/InnoDB: Warning: Setting innodb_use_sys_malloc/,
      qr/InnoDB: Warning: a long semaphore wait:/,
-     qr/InnoDB: Disabling redo log encryption/,
-     qr/InnoDB: Redo log crypto: Can't initialize to key version -1u/,
      qr/InnoDB: Dumping buffer pool.*/,
      qr/InnoDB: Buffer pool.*/,
      qr/InnoDB: Warning: Writer thread is waiting this semaphore/,
@@ -4422,9 +4419,6 @@ ($$)
      qr|InnoDB: TABLE to scan your table for corruption|,
      qr/InnoDB: See also */,
      qr/InnoDB: Cannot open .*ib_buffer_pool.* for reading: No such file or directory*/,
-     qr/InnoDB: Upgrading redo log:*/,
-     qr|InnoDB: Starting to delete and rewrite log files.|,
-     qr/InnoDB: New log files created, LSN=*/,
      qr|InnoDB: Creating foreign key constraint system tables.|,
      qr/InnoDB: Table .*mysql.*innodb_table_stats.* not found./,
      qr/InnoDB: User stopword table .* does not exist./

mysql-test/suite/encryption/r/innodb-log-encrypt-crash.result

@@ -0,0 +1,65 @@
+#
+# MDEV-9011: Redo log encryption does not work
+#
+#
+# MDEV-9422 Encrypted redo log checksum errors
+# on restart after killing busy server instance
+#
+SET GLOBAL innodb_log_checksums=0;
+Warnings:
+Warning	138	innodb_encrypt_log implies innodb_log_checksums
+SELECT @@global.innodb_log_checksums;
+@@global.innodb_log_checksums
+1
+CREATE TABLE t0 (
+pk bigint auto_increment,
+col_int int,
+col_int_key int,
+col_char char(12),
+col_char_key char(12),
+primary key (pk),
+key (col_int_key),
+key (col_char_key)
+) ENGINE=InnoDB ENCRYPTED=YES ENCRYPTION_KEY_ID=1;
+CREATE TEMPORARY TABLE t LIKE t0;
+INSERT INTO t VALUES
+(NULL,1,1,'private','secret'),(NULL,2,2,'sacred','success'),
+(NULL,3,3,'story','secure'),(NULL,4,4,'security','sacrament');
+SET GLOBAL innodb_flush_log_at_trx_commit=1;
+INSERT INTO t0
+SELECT NULL, t1.col_int, t1.col_int_key, t1.col_char, t1.col_char_key
+FROM t t1, t t2, t t3, t t4, t t5;
+# Kill the server
+# ibdata1 expecting NOT FOUND
+NOT FOUND /private|secret|sacr(ed|ament)|success|story|secur(e|ity)/ in ibdata1
+# t0.ibd expecting NOT FOUND
+NOT FOUND /private|secret|sacr(ed|ament)|success|story|secur(e|ity)/ in t0.ibd
+# ib_logfile0 expecting NOT FOUND
+NOT FOUND /private|secret|sacr(ed|ament)|success|story|secur(e|ity)/ in ib_logfile0
+# ib_logfile1 expecting NOT FOUND
+NOT FOUND /private|secret|sacr(ed|ament)|success|story|secur(e|ity)/ in ib_logfile1
+# Restart without redo log encryption
+SELECT COUNT(*) FROM t0;
+COUNT(*)
+1024
+CHECK TABLE t0;
+Table	Op	Msg_type	Msg_text
+test.t0	check	status	OK
+SET GLOBAL innodb_flush_log_at_trx_commit=1;
+INSERT INTO t0 VALUES(NULL, 5, 5, 'public', 'gossip');
+# Kill the server
+# ib_logfile0 expecting NOT FOUND
+NOT FOUND /private|secret|sacr(ed|ament)|success|story|secur(e|ity)/ in ib_logfile0
+# ib_logfile0 expecting FOUND
+FOUND /public|gossip/ in ib_logfile0
+# ibdata1 expecting NOT FOUND
+NOT FOUND /private|secret|sacr(ed|ament)|success|story|secur(e|ity)|public|gossip/ in ibdata1
+# t0.ibd expecting NOT FOUND
+NOT FOUND /private|secret|sacr(ed|ament)|success|story|secur(e|ity)|public|gossip/ in t0.ibd
+SELECT COUNT(*) FROM t0;
+COUNT(*)
+1025
+CHECK TABLE t0;
+Table	Op	Msg_type	Msg_text
+test.t0	check	status	OK
+DROP TABLE t0;

mysql-test/suite/encryption/r/innodb-log-encrypt.result

@@ -41,7 +41,7 @@ WHERE engine = 'innodb'
 AND support IN ('YES', 'DEFAULT', 'ENABLED');
 ENGINE	SUPPORT	COMMENT	TRANSACTIONS	XA	SAVEPOINTS
 FOUND /InnoDB: Invalid log block checksum. block: 2372 checkpoint no: 1 expected: 3362026715 found: 144444122/ in mysqld.1.err
-FOUND /InnoDB: Ignoring the redo log due to missing MLOG_CHECKPOINT between the checkpoint 1213964 and the end 1213952\./ in mysqld.1.err
+FOUND /InnoDB: Missing MLOG_CHECKPOINT between the checkpoint 1213964 and the end 1213952\./ in mysqld.1.err
 # --innodb-force-recovery=6 (skip the entire redo log)
 SELECT * FROM INFORMATION_SCHEMA.ENGINES
 WHERE engine = 'innodb'
@@ -54,7 +54,6 @@ SELECT * FROM INFORMATION_SCHEMA.ENGINES
 WHERE engine = 'innodb'
 AND support IN ('YES', 'DEFAULT', 'ENABLED');
 ENGINE	SUPPORT	COMMENT	TRANSACTIONS	XA	SAVEPOINTS
-FOUND /InnoDB: Ignoring the redo log due to missing MLOG_CHECKPOINT between the checkpoint 1213964 and the end 1213952\./ in mysqld.1.err
 # --innodb-force-recovery=6 (skip the entire redo log)
 SELECT * FROM INFORMATION_SCHEMA.ENGINES
 WHERE engine = 'innodb'
@@ -90,6 +89,7 @@ SELECT COUNT(*) `1` FROM INFORMATION_SCHEMA.ENGINES WHERE engine='innodb'
 AND support IN ('YES', 'DEFAULT', 'ENABLED');
 1
 1
+FOUND /InnoDB: Encrypting redo log/ in mysqld.1.err
 ib_buffer_pool
 ib_logfile0
 ib_logfile1