MDEV-12873 InnoDB SYS_TABLES.TYPE incompatibility for PAGE_COMPRESSED…

…=YES in MariaDB 10.2.2 to 10.2.6

Remove the SHARED_SPACE flag that was erroneously introduced in
MariaDB 10.2.2, and shift the SYS_TABLES.TYPE flags back to where
they were before MariaDB 10.2.2. While doing this, ensure that
tables created with affected MariaDB versions can be loaded,
and also ensure that tables created with MySQL 5.7 using the
TABLESPACE attribute cannot be loaded.

MariaDB 10.2.2 picked the SHARED_SPACE flag from MySQL 5.7,
shifting the MariaDB 10.1 flags PAGE_COMPRESSION, PAGE_COMPRESSION_LEVEL,
ATOMIC_WRITES by one bit. The SHARED_SPACE flag would always
be written as 0 by MariaDB, because MariaDB does not support
CREATE TABLESPACE or CREATE TABLE...TABLESPACE for InnoDB.

So, instead of the bits AALLLLCxxxxxxx we would have
AALLLLC0xxxxxxx if the table was created with MariaDB 10.2.2
to 10.2.6. (AA=ATOMIC_WRITES, LLLL=PAGE_COMPRESSION_LEVEL,
C=PAGE_COMPRESSED, xxxxxxx=7 bits that were not moved.)

PAGE_COMPRESSED=NO implies LLLLC=00000. That is not a problem.

If someone created a table in MariaDB 10.2.2 or 10.2.3 with
the attribute ATOMIC_WRITES=OFF (value 2; AA=10) and without
PAGE_COMPRESSED=YES or PAGE_COMPRESSION_LEVEL, the table should be
rejected. We ignore this problem, because it should be unlikely
for anyone to specify ATOMIC_WRITES=OFF, and because 10.2.2 and
10.2.2 were not mature releases. The value ATOMIC_WRITES=ON (1)
would be interpreted as ATOMIC_WRITES=OFF, but starting with
MariaDB 10.2.4 the ATOMIC_WRITES attribute is ignored.

PAGE_COMPRESSED=YES implies that PAGE_COMPRESSION_LEVEL be between
1 and 9 and that ROW_FORMAT be COMPACT or DYNAMIC. Thus, the affected
wrong bit pattern in SYS_TABLES.TYPE is of the form AALLLL10DB00001
where D signals the presence of a DATA DIRECTORY attribute and B is 1
for ROW_FORMAT=DYNAMIC and 0 for ROW_FORMAT=COMPACT. We must interpret
this bit pattern as AALLLL1DB00001 (discarding the extraneous 0 bit).

dict_sys_tables_rec_read(): Adjust the affected bit pattern when
reading the SYS_TABLES.TYPE column. In case of invalid flags,
report both SYS_TABLES.TYPE (after possible adjustment) and
SYS_TABLES.MIX_LEN.

dict_load_table_one(): Replace an unreachable condition on
!dict_tf2_is_valid() with a debug assertion. The flags will already
have been validated by dict_sys_tables_rec_read(); if that validation
fails, dict_load_table_low() will have failed.

fil_ibd_create(): Shorten an error message about a file pre-existing.

Datafile::validate_to_dd(): Clarify an error message about tablespace
flags mismatch.

ha_innobase::open(): Remove an unnecessary warning message.

dict_tf_is_valid(): Simplify and stricten the logic. Validate the
values of PAGE_COMPRESSION. Remove error log output; let the callers
handle that.

DICT_TF_BITS: Remove ATOMIC_WRITES, PAGE_ENCRYPTION, PAGE_ENCRYPTION_KEY.
The ATOMIC_WRITES is ignored once the SYS_TABLES.TYPE has been validated;
there is no need to store it in dict_table_t::flags. The PAGE_ENCRYPTION
and PAGE_ENCRYPTION_KEY are unused since MariaDB 10.1.4 (the GA release
was 10.1.8).

DICT_TF_BIT_MASK: Remove (unused).

FSP_FLAGS_MEM_ATOMIC_WRITES: Remove (the flags are never read).

row_import_read_v1(): Display an error if dict_tf_is_valid() fails.

mysql-test/suite/innodb/r/row_format_redundant.result

@@ -72,7 +72,7 @@ DROP TABLE t1;
 Warnings:
 Warning	1932	Table 'test.t1' doesn't exist in engine
 DROP TABLE t2,t3;
-FOUND 49 /\[ERROR\] InnoDB: Table `test`\.`t1` in InnoDB data dictionary contains invalid flags\. SYS_TABLES.MIX_LEN=255\b/ in mysqld.1.err
+FOUND 49 /\[ERROR\] InnoDB: Table `test`\.`t1` in InnoDB data dictionary contains invalid flags\. SYS_TABLES\.TYPE=1 SYS_TABLES\.MIX_LEN=255\b/ in mysqld.1.err
 ib_buffer_pool
 ib_logfile0
 ib_logfile1

mysql-test/suite/innodb/t/row_format_redundant.test

@@ -4,7 +4,7 @@
 
 --disable_query_log
 call mtr.add_suppression("InnoDB: Table `mysql`\\.`innodb_table_stats` not found");
-call mtr.add_suppression("InnoDB: Table `test`.`t1` in InnoDB data dictionary contains invalid flags. SYS_TABLES.MIX_LEN=255");
+call mtr.add_suppression("InnoDB: Table `test`.`t1` in InnoDB data dictionary contains invalid flags. SYS_TABLES\\.TYPE=1 SYS_TABLES\\.MIX_LEN=255\\r?$");
 call mtr.add_suppression("InnoDB: Parent table of FTS auxiliary table test/FTS_.* not found");
 call mtr.add_suppression("InnoDB: Cannot open table test/t1 from the internal data dictionary");
 call mtr.add_suppression("InnoDB: Table `test`.`t1` does not exist in the InnoDB internal data dictionary though MariaDB is trying to (rename|drop)");
@@ -141,7 +141,7 @@ RENAME TABLE t1 TO tee_one;
 DROP TABLE t1;
 DROP TABLE t2,t3;
 
---let SEARCH_PATTERN= \[ERROR\] InnoDB: Table `test`\.`t1` in InnoDB data dictionary contains invalid flags\. SYS_TABLES.MIX_LEN=255\b
+--let SEARCH_PATTERN= \[ERROR\] InnoDB: Table `test`\.`t1` in InnoDB data dictionary contains invalid flags\. SYS_TABLES\.TYPE=1 SYS_TABLES\.MIX_LEN=255\b
 --source include/search_pattern_in_file.inc
 
 --let $restart_parameters=

storage/innobase/dict/dict0load.cc

@@ -1170,6 +1170,80 @@ dict_sys_tables_rec_read(
 	ut_a(len == 4);
 	type = mach_read_from_4(field);
 
+	/* Handle MDEV-12873 InnoDB SYS_TABLES.TYPE incompatibility
+	for PAGE_COMPRESSED=YES in MariaDB 10.2.2 to 10.2.6.
+
+	MariaDB 10.2.2 introduced the SHARED_SPACE flag from MySQL 5.7,
+	shifting the flags PAGE_COMPRESSION, PAGE_COMPRESSION_LEVEL,
+	ATOMIC_WRITES by one bit. The SHARED_SPACE flag would always
+	be written as 0 by MariaDB, because MariaDB does not support
+	CREATE TABLESPACE or CREATE TABLE...TABLESPACE for InnoDB.
+
+	So, instead of the bits AALLLLCxxxxxxx we would have
+	AALLLLC0xxxxxxx if the table was created with MariaDB 10.2.2
+	to 10.2.6. (AA=ATOMIC_WRITES, LLLL=PAGE_COMPRESSION_LEVEL,
+	C=PAGE_COMPRESSED, xxxxxxx=7 bits that were not moved.)
+
+	The case LLLLC=00000 is not a problem. The problem is the case
+	AALLLL10DB00001 where D is the (mostly ignored) DATA_DIRECTORY
+	flag and B is the ATOMIC_BLOBS flag (1 for ROW_FORMAT=DYNAMIC
+	and 0 for ROW_FORMAT=COMPACT in this case). Other low-order
+	bits must be so, because PAGE_COMPRESSED=YES is only allowed
+	for ROW_FORMAT=DYNAMIC and ROW_FORMAT=COMPACT, not for
+	ROW_FORMAT=REDUNDANT or ROW_FORMAT=COMPRESSED.
+
+	Starting with MariaDB 10.2.4, the flags would be
+	00LLLL10DB00001, because ATOMIC_WRITES is always written as 0.
+
+	We will concentrate on the PAGE_COMPRESSION_LEVEL and
+	PAGE_COMPRESSED=YES. PAGE_COMPRESSED=NO implies
+	PAGE_COMPRESSION_LEVEL=0, and in that case all the affected
+	bits will be 0. For PAGE_COMPRESSED=YES, the values 1..9 are
+	allowed for PAGE_COMPRESSION_LEVEL. That is, we must interpret
+	the bits AALLLL10DB00001 as AALLLL1DB00001.
+
+	If someone created a table in MariaDB 10.2.2 or 10.2.3 with
+	the attribute ATOMIC_WRITES=OFF (value 2) and without
+	PAGE_COMPRESSED=YES or PAGE_COMPRESSION_LEVEL, that should be
+	rejected. The value ATOMIC_WRITES=ON (1) would look like
+	ATOMIC_WRITES=OFF, but it would be ignored starting with
+	MariaDB 10.2.4. */
+	compile_time_assert(DICT_TF_POS_PAGE_COMPRESSION == 7);
+	compile_time_assert(DICT_TF_POS_UNUSED == 14);
+
+	if ((type & 0x19f) != 0x101) {
+		/* The table cannot have been created with MariaDB
+		10.2.2 to 10.2.6, because they would write the
+		low-order bits of SYS_TABLES.TYPE as 0b10xx00001 for
+		PAGE_COMPRESSED=YES. No adjustment is applicable. */
+	} else if (type >= 3 << 13) {
+		/* 10.2.2 and 10.2.3 write ATOMIC_WRITES less than 3,
+		and no other flags above that can be set for the
+		SYS_TABLES.TYPE to be in the 10.2.2..10.2.6 format.
+		This would in any case be invalid format for 10.2 and
+		earlier releases. */
+		ut_ad(ULINT_UNDEFINED == dict_sys_tables_type_validate(
+			      type, DICT_N_COLS_COMPACT));
+	} else {
+		/* SYS_TABLES.TYPE is of the form AALLLL10DB00001.  We
+		must still validate that the LLLL bits are between 0
+		and 9 before we can discard the extraneous 0 bit. */
+		ut_ad(!DICT_TF_GET_PAGE_COMPRESSION(type));
+
+		if ((((type >> 9) & 0xf) - 1) < 9) {
+			ut_ad(DICT_TF_GET_PAGE_COMPRESSION_LEVEL(type) & 1);
+
+			type = (type & 0x7fU) | (type >> 1 & ~0x7fU);
+
+			ut_ad(DICT_TF_GET_PAGE_COMPRESSION(type));
+			ut_ad(DICT_TF_GET_PAGE_COMPRESSION_LEVEL(type) >= 1);
+			ut_ad(DICT_TF_GET_PAGE_COMPRESSION_LEVEL(type) <= 9);
+		} else {
+			ut_ad(ULINT_UNDEFINED == dict_sys_tables_type_validate(
+				      type, DICT_N_COLS_COMPACT));
+		}
+	}
+
 	/* The low order bit of SYS_TABLES.TYPE is always set to 1. But in
 	dict_table_t::flags the low order bit is used to determine if the
 	row format is Redundant (0) or Compact (1) when the format is Antelope.
@@ -1211,7 +1285,8 @@ dict_sys_tables_rec_read(
 		if (!dict_tf2_is_valid(*flags, *flags2)) {
 			ib::error() << "Table " << table_name << " in InnoDB"
 				" data dictionary contains invalid flags."
-				" SYS_TABLES.MIX_LEN=" << *flags2;
+				" SYS_TABLES.TYPE=" << type
+				<< " SYS_TABLES.MIX_LEN=" << *flags2;
 			return(false);
 		}
 
@@ -2541,8 +2616,7 @@ dict_load_table_low(table_name_t& name, const rec_t* rec, dict_table_t** table)
 	ulint		flags2;
 	ulint		n_v_col;
 
-	const char* error_text = dict_sys_tables_rec_check(rec);
-	if (error_text != NULL) {
+	if (const char* error_text = dict_sys_tables_rec_check(rec)) {
 		return(error_text);
 	}
 
@@ -2802,7 +2876,6 @@ dict_load_table_one(
 	const rec_t*	rec;
 	const byte*	field;
 	ulint		len;
-	const char*	err_msg;
 	mtr_t		mtr;
 
 	DBUG_ENTER("dict_load_table_one");
@@ -2859,9 +2932,7 @@ dict_load_table_one(
 		goto err_exit;
 	}
 
-	err_msg = dict_load_table_low(name, rec, &table);
-
-	if (err_msg) {
+	if (const char* err_msg = dict_load_table_low(name, rec, &table)) {
 		if (err_msg != dict_load_table_flags) {
 			ib::error() << err_msg;
 		}
@@ -2885,6 +2956,8 @@ dict_load_table_one(
 
 	mem_heap_empty(heap);
 
+	ut_ad(dict_tf2_is_valid(table->flags, table->flags2));
+
 	/* If there is no tablespace for the table then we only need to
 	load the index definitions. So that we can IMPORT the tablespace
 	later. When recovering table locks for resurrected incomplete
@@ -2920,17 +2993,6 @@ dict_load_table_one(
 		}
 	}
 
-	if (!dict_tf2_is_valid(table->flags, table->flags2)) {
-		ib::error() << "Table " << table->name << " in InnoDB"
-			" data dictionary contains invalid flags."
-			" SYS_TABLES.MIX_LEN=" << table->flags2;
-		table->flags2 &= ~DICT_TF2_TEMPORARY;
-		dict_table_remove_from_cache(table);
-		table = NULL;
-		err = DB_FAIL;
-		goto func_exit;
-	}
-
 	/* Initialize table foreign_child value. Its value could be
 	changed when dict_load_foreigns() is called below */
 	table->fk_max_recusive_level = 0;

storage/innobase/fil/fil0fil.cc

@@ -3819,18 +3819,12 @@ fil_ibd_create(
 		ib::error() << "Cannot create file '" << path << "'";
 
 		if (error == OS_FILE_ALREADY_EXISTS) {
-			ib::error() << "The file '" << path << "'"
+			ib::info() << "The file '" << path << "'"
 				" already exists though the"
 				" corresponding table did not exist"
 				" in the InnoDB data dictionary."
-				" Have you moved InnoDB .ibd files"
-				" around without using the SQL commands"
-				" DISCARD TABLESPACE and IMPORT TABLESPACE,"
-				" or did mysqld crash in the middle of"
-				" CREATE TABLE?"
 				" You can resolve the problem by removing"
-				" the file '" << path
-				<< "' under the 'datadir' of MySQL.";
+				" the file.";
 
 			return(DB_TABLESPACE_EXISTS);
 		}

storage/innobase/fsp/fsp0file.cc

@@ -425,13 +425,10 @@ Datafile::validate_to_dd(ulint space_id, ulint flags)
 	/* else do not use this tablespace. */
 	m_is_valid = false;
 
-	ib::error() << "In file '" << m_filepath << "', tablespace id and"
-		" flags are " << m_space_id << " and " << ib::hex(m_flags)
-		<< ", but in the InnoDB data dictionary they are "
-		<< space_id << " and " << ib::hex(flags)
-		<< ". Have you moved InnoDB .ibd files around without"
-		" using the commands DISCARD TABLESPACE and IMPORT TABLESPACE?"
-		" " << TROUBLESHOOT_DATADICT_MSG;
+	ib::error() << "Refusing to load '" << m_filepath << "' (id="
+		<< m_space_id << ", flags=" << ib::hex(m_flags)
+		<< "); dictionary contains id="
+		<< space_id << ", flags=" << ib::hex(flags);
 
 	return(DB_ERROR);
 }

storage/innobase/handler/ha_innodb.cc

@@ -6624,10 +6624,6 @@ ha_innobase::open(
 					norm_name);
 		}
 
-		ib::warn() << "Cannot open table " << norm_name << " from the"
-			" internal data dictionary of InnoDB though the .frm"
-			" file for the table exists. " << TROUBLESHOOTING_MSG;
-
 		free_share(m_share);
 		set_my_errno(ENOENT);
 

storage/innobase/include/dict0dict.ic

@@ -646,74 +646,54 @@ bool
 dict_tf_is_valid(
 	ulint	flags)
 {
-	bool	compact = DICT_TF_GET_COMPACT(flags);
-	ulint	zip_ssize = DICT_TF_GET_ZIP_SSIZE(flags);
-	bool	atomic_blobs = DICT_TF_HAS_ATOMIC_BLOBS(flags);
-	bool	data_dir = DICT_TF_HAS_DATA_DIR(flags);
-	ulint	unused = DICT_TF_GET_UNUSED(flags);
-	bool	page_compression = DICT_TF_GET_PAGE_COMPRESSION(flags);
-	ulint	page_compression_level = DICT_TF_GET_PAGE_COMPRESSION_LEVEL(flags);
-	bool	flags_corrupt = false;
-
 	/* Make sure there are no bits that we do not know about. */
-	if (unused != 0) {
-		flags_corrupt = true;
+	if (flags >= 1U << DICT_TF_BITS) {
+		return(false);
 	}
 
-	if (atomic_blobs) {
-		/* Barracuda row formats COMPRESSED and DYNAMIC both use
+	const bool	not_redundant = DICT_TF_GET_COMPACT(flags);
+	const bool	atomic_blobs = DICT_TF_HAS_ATOMIC_BLOBS(flags);
+
+	if (atomic_blobs && !not_redundant) {
+		/* ROW_FORMAT=COMPRESSED and ROW_FORMAT=DYNAMIC both use
 		atomic_blobs, which build on the page structure introduced
-		for the COMPACT row format by allowing keys in secondary
+		for ROW_FORMAT=COMPACT by allowing keys in secondary
 		indexes to be made from data stored off-page in the
 		clustered index. */
-
-		if (!compact) {
-			flags_corrupt = true;
-		}
-
-	} else if (zip_ssize) {
-		/* Antelope does not support COMPRESSED row format. */
-		flags_corrupt = true;
+		return(false);
 	}
 
-	if (zip_ssize) {
-
-		/* COMPRESSED row format must have compact and atomic_blobs
-		bits set and validate the number is within allowed range. */
-
-		if (!compact
-		    || !atomic_blobs
-		    || zip_ssize > PAGE_ZIP_SSIZE_MAX) {
-			flags_corrupt = true;
-		}
-	}
+	ulint	zip_ssize = DICT_TF_GET_ZIP_SSIZE(flags);
 
-	if (page_compression || page_compression_level) {
-		/* Page compression format must have compact and
-		atomic_blobs and page_compression_level requires
-		page_compression */
-		if (!compact
-			|| !page_compression
-			|| !atomic_blobs) {
-			flags_corrupt = true;
-		}
+	if (!zip_ssize) {
+		/* Not ROW_FORMAT=COMPRESSED */
+	} else if (!atomic_blobs) {
+		/* ROW_FORMAT=COMPRESSED implies ROW_FORMAT=DYNAMIC
+		for the uncompressed page format */
+		return(false);
+	} else if (zip_ssize > PAGE_ZIP_SSIZE_MAX
+		   || zip_ssize > UNIV_PAGE_SIZE_SHIFT
+		   || UNIV_PAGE_SIZE_SHIFT > UNIV_ZIP_SIZE_SHIFT_MAX) {
+		/* KEY_BLOCK_SIZE is out of bounds, or
+		ROW_FORMAT=COMPRESSED is not supported with this
+		innodb_page_size (only up to 16KiB) */
+		return(false);
 	}
 
-
-	if (flags_corrupt) {
-		ib::error()
-			<< "InnoDB: Error: table unused flags are:" << flags
-			<< " in the data dictionary and are corrupted:"
-			<< " compact:" << compact
-			<< " atomic_blobs:" << atomic_blobs
-			<< " unused:" << unused
-			<< " data_dir:" << data_dir
-			<< " zip_ssize:" << zip_ssize
-			<< " page_compression:" << page_compression
-			<< " page_compression_level:" << page_compression_level;
-		return (false);
-	} else {
-		return(true);
+	switch (DICT_TF_GET_PAGE_COMPRESSION_LEVEL(flags)) {
+	case 0:
+		/* PAGE_COMPRESSION_LEVEL=0 should imply PAGE_COMPRESSED=NO */
+		return(!DICT_TF_GET_PAGE_COMPRESSION(flags));
+	case 1: case 2: case 3: case 4: case 5: case 6: case 7: case 8: case 9:
+		/* PAGE_COMPRESSION_LEVEL requires
+		ROW_FORMAT=COMPACT or ROW_FORMAT=DYNAMIC
+		(not ROW_FORMAT=COMPRESSED or ROW_FORMAT=REDUNDANT)
+		and PAGE_COMPRESSED=YES */
+		return(!zip_ssize && not_redundant
+		       && DICT_TF_GET_PAGE_COMPRESSION(flags));
+	default:
+		/* Invalid PAGE_COMPRESSION_LEVEL value */
+		return(false);
 	}
 }
 
@@ -978,7 +958,6 @@ dict_tf_to_fsp_flags(ulint table_flags)
 	ulint fsp_flags;
 	ulint page_compression_level = DICT_TF_GET_PAGE_COMPRESSION_LEVEL(
 		table_flags);
-	ulint atomic_writes = DICT_TF_GET_ATOMIC_WRITES(table_flags);
 
 	ut_ad((DICT_TF_GET_PAGE_COMPRESSION(table_flags) == 0)
 	      == (page_compression_level == 0));
@@ -1005,7 +984,6 @@ dict_tf_to_fsp_flags(ulint table_flags)
 		fsp_flags |= 1U << FSP_FLAGS_MEM_DATA_DIR;
 	}
 
-	fsp_flags |= atomic_writes << FSP_FLAGS_MEM_ATOMIC_WRITES;
 	fsp_flags |= page_compression_level << FSP_FLAGS_MEM_COMPRESSION_LEVEL;
 
 	return(fsp_flags);