Bug#30865032 WARNING MESSAGES CANNOT CALCULATE STATISTICS WHEN IMPORT…

…ING TABLESPACES

The occurrence of this message is a minor issue fixed by change #1 below.
But during testing, I found that if mysqld is restarted while remote and
local tablespaces are discarded, especially if the tablespaces to be imported
are already in place at startup, then many things can go wrong.  There were
various asserts that occurred depending on timing. During all the testing
and debugging, the following changes were made.

1. Prevent the stats thread from complaining about a missing tablespace.
   See dict_stats_update().
2. Prevent a discarded tablespace from being opened at startup, even if the
   table to be imported is already in place. See Validate_files::check().
3. dd_tablespace_get_state_enum() was refactored to separate the normal
   way to do it in v8.0, which is to use "state" key in
   dd::tablespaces::se_private_date, from the non-standard way which is
   to check undo::spaces or look for the old key value pair of
   "discarded=true". This allowed the new call to this routine by the
   change in fix #2 above.
4. Change thd_tablespace_op() in sql/sql_thd_api.cc such that instead of
   returning 1 if the DDL requires an implicit tablespace, it returns the
   DDL operation flag.  This can still be interpreted as a boolean, but it
   can also be used to determine if the op is an IMPORT or a DISCARD.
5. With that change, the annoying message that a space is discarded can be
   avoided during an import when it needs to be discarded.
6. Several test cases were corrected now that the useless "is discarded"
   warning is no longer being written.
7. Two places where dd_tablespace_set_state() was called to set the state
   to either "discard" or "normal" were consolidated to a new version of
   dd_tablespace_set_state(thd, dd_space_id, space_name, dd_state).
8. This new version of dd_tablespace_set_state() was used in
   dd_commit_inplace_alter_table() to make sure that in all three places
   the dd is changed to identify a discarded tablesapace, it is identified
   in dd:Tablespace::se_private_data as well as dd:Table::se_private_data
   or dd::Partition::se_private_data.  The reason it is necessary to
   record this in dd::Tablespace is that during startup, boot_tablespaces()
   and Validate::files::check() are only traversing dd::Tablespace.
   And that is where fix #2 is done!
9. One of the asserts that occurred was during IMPORT TABLESPACE after a
   restart that found a discarded 5.7 tablespace in the v8.0 discarded
   location. This assert occurred in Fil_shard::get_file_size() just after
   ER_IB_MSG_272.  The 5.7 file did not have the SDI flag, but the v8.0
   space that was discarded did have that flag.  So the flags did not match.
   That crash was fixed by setting the fil_space_t::flags to what it is in
   the tablespace header page.  A descriptive comment was added.
10. There was a section in fil_ibd_open() that checked
   `if (space != nullptr) {` and if true, it would close and free stuff
   then immediately crash.  I think I remember many years ago adding that
   assert because I did not think it actually occurred. Well it did occur
   during my testing before I added fix #2 above.  This made fil_ibd_open()
   assume that the file was NOT already open.
   So fil_ibd_open() is now changed to allow for that possibility by adding
   `if (space != nullptr) {return DB_SUCCESS}` further down.
   Since fil_ibd_open() can be called with a `validate` boolean, the routine
   now attempts to do all the validation whether or not the tablespace is
   already open.

The following are non-functional changes;
- Many code documentation lines were added or improved.
- dict_sys_t::s_space_id renamed to dict_sys_t::s_dict_space_id in order
  to clarify better which space_id it referred to.
- For the same reason, change s_dd_space_id to s_dd_dict_space_id.
- Replaced `table->flags2 & DICT_TF2_DISCARDED`
  with `dict_table_is_discarded(table)` in dict0load.cc
- A redundant call to ibuf_delete_for_discarded_space(space_id) was deleted
  from fil_discard_tablespace() because it is also called higher up in
  the call stack in row_import_for_mysql().
- Deleted the declaration to `row_import_update_discarded_flag()` since
  the definition no longer exists.  It was deleted when we switched from
  `discarded=true` to 'state=discarded' in dd::Tablespace::se_private_data
  early in v8.0 developement.

Approved by Mateusz in RB#26077

mysql-test/suite/innodb/r/choose_tbsp_location-discard.result

@@ -156,8 +156,6 @@ t5980.ibd.bak2
 # Import the tablespace and do some DDL and DML.
 #
 ALTER TABLE t5980 IMPORT TABLESPACE;
-Warnings:
-Warning	1814	InnoDB: Tablespace has been discarded for table 't5980'
 ### files in MYSQLD_DATADIR/test
 ### files in MYSQL_TMP_DIR/alt_dir/test
 t5980.cfg
@@ -409,8 +407,6 @@ t5980.ibd
 t5980.ibd.bak
 t5980.ibd.bak2
 ALTER TABLE t5980 IMPORT TABLESPACE;
-Warnings:
-Warning	1814	InnoDB: Tablespace has been discarded for table 't5980'
 INSERT INTO t5980 VALUES (2, "Insert this record after IMPORT");
 SELECT * FROM t5980;
 a	b

mysql-test/suite/innodb/r/partition_import_57.result

@@ -0,0 +1,77 @@
+#
+# Import regular and partitioned tablespaces exported from 5.7 after a restart.
+#
+CREATE SCHEMA `import80`;
+USE `import80`;
+CREATE TABLE `t1` (`c1` INT);
+CREATE TABLE `t2` (`c1` INT PRIMARY KEY, `c2` INT) DATA DIRECTORY ='MYSQL_TMP_DIR/external_dir';
+CREATE TABLE `t3` (`c1` INT) PARTITION BY KEY (`c1`) PARTITIONS 2;
+CREATE TABLE `t4` (`c1` INT PRIMARY KEY, `c2` INT) DATA DIRECTORY ='MYSQL_TMP_DIR/external_dir' PARTITION BY HASH (`c1`) PARTITIONS 2;
+CREATE TABLE `t5` (`c1` INT);
+INSERT INTO `t5` values (1);
+CREATE TABLE `t6` (`c1` INT PRIMARY KEY, `c2` INT) DATA DIRECTORY ='MYSQL_TMP_DIR/external_dir';
+INSERT INTO `t6` values (1, 1);
+CREATE TABLE `t7` (`c1` INT) PARTITION BY KEY (`c1`) PARTITIONS 2;
+INSERT INTO `t7` values (1);
+CREATE TABLE `t8` (`c1` INT PRIMARY KEY, `c2` INT) DATA DIRECTORY ='MYSQL_TMP_DIR/external_dir' PARTITION BY HASH (`c1`) PARTITIONS 2;
+INSERT INTO `t8` values (1, 1);
+FLUSH TABLES `t5`, `t6`, `t7`, `t8` FOR EXPORT;
+# Copy cfg and ibd files for `t5` to MYSQL_TMP_DIR/export80.
+# Copy cfg and ibd files for `t6` to MYSQL_TMP_DIR/export80.
+# Copy cfg and ibd files for `t7` to MYSQL_TMP_DIR/export80.
+# Copy cfg and ibd files for `t8` to MYSQL_TMP_DIR/export80.
+UNLOCK TABLES;
+ALTER TABLE `t1` DISCARD TABLESPACE;
+ALTER TABLE `t2` DISCARD TABLESPACE;
+ALTER TABLE `t3` DISCARD TABLESPACE;
+ALTER TABLE `t4` DISCARD TABLESPACE;
+ALTER TABLE `t5` DISCARD TABLESPACE;
+ALTER TABLE `t6` DISCARD TABLESPACE;
+ALTER TABLE `t7` DISCARD TABLESPACE;
+ALTER TABLE `t8` DISCARD TABLESPACE;
+# Unzip exported 5.7 partitioned tables to MYSQL_TMP_DIR.
+# Copy exported v5.7 t1 files to MYSQLD_DATADIR/import80.
+# Copy exported v5.7 t2 files to MYSQL_TMP_DIR/external_dir/import80.
+# Copy exported v5.7 t3 files to MYSQLD_DATADIR/import80.
+# Copy exported v5.7 t4 files to MYSQL_TMP_DIR/external_dir/import80.
+# Copy exported v8.0 t5 files to MYSQLD_DATADIR/import80.
+# Copy exported v8.0 t6 files to MYSQL_TMP_DIR/external_dir/import80.
+# Copy exported v8.0 t7 files to MYSQLD_DATADIR/import80.
+# Copy exported v8.0 t8 files to MYSQL_TMP_DIR/external_dir/import80.
+# restart
+ALTER TABLE `t1` IMPORT TABLESPACE;
+select * from t1;
+c1
+1
+ALTER TABLE `t2` IMPORT TABLESPACE;
+select * from t2;
+c1	c2
+1	1
+ALTER TABLE `t3` IMPORT TABLESPACE;
+select * from t3;
+c1
+1
+ALTER TABLE `t4` IMPORT TABLESPACE;
+select * from t4;
+c1	c2
+1	1
+ALTER TABLE `t5` IMPORT TABLESPACE;
+select * from t5;
+c1
+1
+ALTER TABLE `t6` IMPORT TABLESPACE;
+select * from t6;
+c1	c2
+1	1
+ALTER TABLE `t7` IMPORT TABLESPACE;
+select * from t7;
+c1
+1
+ALTER TABLE `t8` IMPORT TABLESPACE;
+select * from t8;
+c1	c2
+1	1
+# Cleanup
+DROP TABLE `t1`, `t2`, t3, t4, `t5`, `t6`, t7, t8;
+USE `test`;
+DROP SCHEMA `import80`;

mysql-test/suite/innodb/t/partition_import_57-master.opt

@@ -0,0 +1 @@
+--innodb-directories=$MYSQL_TMP_DIR/external_dir

mysql-test/suite/innodb/t/partition_import_57.test

@@ -0,0 +1,140 @@
+--echo #
+--echo # Import regular and partitioned tablespaces exported from 5.7 after a restart.
+--echo #
+
+# The following SQL was used to create the tables on 5.7
+# CREATE SCHEMA `export57`;
+# USE `export57`;
+# CREATE TABLE `t1` (`c1` INT);
+# INSERT INTO `t1` values (1);
+# CREATE TABLE `t2` (`c1` INT PRIMARY KEY, `c2` INT) DATA DIRECTORY '/tmp/data57';
+# INSERT INTO `t2` values (1, 1);
+# CREATE TABLE `t3` (`c1` INT) PARTITION BY KEY (`c1`) PARTITIONS 2;
+# INSERT INTO `t3` values (1);
+# CREATE TABLE `t4` (`c1` INT PRIMARY KEY, `c2` INT) DATA DIRECTORY '/tmp/data57' PARTITION BY HASH (`c1`) PARTITIONS 2;
+# INSERT INTO `t4` values (1, 1);
+# FLUSH TABLES `t1`, `t2`, `t3`, `t4` FOR EXPORT;
+#
+# Copy all .cfg and .ibd files for t1 and t3 files to /tmp/data57/export57
+# /tmp/data57> zip -qr data57_exported.zip export57
+# Copy data57_exported.zip to {branch mysql-8.0} mysql-test/std_data/data57_exported.zip
+#
+# UNLOCK TABLES;
+# DROP TABLE `t1`, `t2`, `t3`, `t4`;
+
+--source include/have_innodb_16k.inc
+--let $MYSQLD_DATADIR = `select @@datadir`
+--let $EXTERNAL_DIR = $MYSQL_TMP_DIR/external_dir
+
+CREATE SCHEMA `import80`;
+USE `import80`;
+
+CREATE TABLE `t1` (`c1` INT);
+
+--replace_result $MYSQL_TMP_DIR MYSQL_TMP_DIR
+eval CREATE TABLE `t2` (`c1` INT PRIMARY KEY, `c2` INT) DATA DIRECTORY ='$EXTERNAL_DIR';
+
+CREATE TABLE `t3` (`c1` INT) PARTITION BY KEY (`c1`) PARTITIONS 2;
+
+--replace_result $MYSQL_TMP_DIR MYSQL_TMP_DIR
+eval CREATE TABLE `t4` (`c1` INT PRIMARY KEY, `c2` INT) DATA DIRECTORY ='$EXTERNAL_DIR' PARTITION BY HASH (`c1`) PARTITIONS 2;
+
+CREATE TABLE `t5` (`c1` INT);
+INSERT INTO `t5` values (1);
+
+--replace_result $MYSQL_TMP_DIR MYSQL_TMP_DIR
+eval CREATE TABLE `t6` (`c1` INT PRIMARY KEY, `c2` INT) DATA DIRECTORY ='$EXTERNAL_DIR';
+INSERT INTO `t6` values (1, 1);
+
+CREATE TABLE `t7` (`c1` INT) PARTITION BY KEY (`c1`) PARTITIONS 2;
+INSERT INTO `t7` values (1);
+
+--replace_result $MYSQL_TMP_DIR MYSQL_TMP_DIR
+eval CREATE TABLE `t8` (`c1` INT PRIMARY KEY, `c2` INT) DATA DIRECTORY ='$EXTERNAL_DIR' PARTITION BY HASH (`c1`) PARTITIONS 2;
+INSERT INTO `t8` values (1, 1);
+
+FLUSH TABLES `t5`, `t6`, `t7`, `t8` FOR EXPORT;
+--mkdir $MYSQL_TMP_DIR/export80/
+--echo # Copy cfg and ibd files for `t5` to MYSQL_TMP_DIR/export80.
+--copy_files_wildcard  $MYSQLD_DATADIR/import80/   $MYSQL_TMP_DIR/export80/  t5*
+--echo # Copy cfg and ibd files for `t6` to MYSQL_TMP_DIR/export80.
+--copy_files_wildcard  $EXTERNAL_DIR/import80/     $MYSQL_TMP_DIR/export80/  t6*
+--echo # Copy cfg and ibd files for `t7` to MYSQL_TMP_DIR/export80.
+--copy_files_wildcard  $MYSQLD_DATADIR/import80/   $MYSQL_TMP_DIR/export80/  t7*
+--echo # Copy cfg and ibd files for `t8` to MYSQL_TMP_DIR/export80.
+--copy_files_wildcard  $EXTERNAL_DIR/import80/     $MYSQL_TMP_DIR/export80/  t8*
+UNLOCK TABLES;
+
+ALTER TABLE `t1` DISCARD TABLESPACE;
+ALTER TABLE `t2` DISCARD TABLESPACE;
+ALTER TABLE `t3` DISCARD TABLESPACE;
+ALTER TABLE `t4` DISCARD TABLESPACE;
+ALTER TABLE `t5` DISCARD TABLESPACE;
+ALTER TABLE `t6` DISCARD TABLESPACE;
+ALTER TABLE `t7` DISCARD TABLESPACE;
+ALTER TABLE `t8` DISCARD TABLESPACE;
+
+--echo # Unzip exported 5.7 partitioned tables to MYSQL_TMP_DIR.
+--copy_file $MYSQLTEST_VARDIR/std_data/data57_exported.zip $MYSQL_TMP_DIR/data57_exported.zip
+--file_exists $MYSQL_TMP_DIR/data57_exported.zip
+--exec unzip -qo $MYSQL_TMP_DIR/data57_exported.zip -d $MYSQL_TMP_DIR
+
+--echo # Copy exported v5.7 t1 files to MYSQLD_DATADIR/import80.
+--copy_files_wildcard  $MYSQL_TMP_DIR/export57/  $MYSQLD_DATADIR/import80/  t1*
+
+--echo # Copy exported v5.7 t2 files to MYSQL_TMP_DIR/external_dir/import80.
+--copy_files_wildcard  $MYSQL_TMP_DIR/export57/  $EXTERNAL_DIR/import80/  t2*
+
+--echo # Copy exported v5.7 t3 files to MYSQLD_DATADIR/import80.
+--copy_files_wildcard  $MYSQL_TMP_DIR/export57/  $MYSQLD_DATADIR/import80/  t3*
+
+--echo # Copy exported v5.7 t4 files to MYSQL_TMP_DIR/external_dir/import80.
+--copy_files_wildcard  $MYSQL_TMP_DIR/export57/  $EXTERNAL_DIR/import80/  t4*
+
+--echo # Copy exported v8.0 t5 files to MYSQLD_DATADIR/import80.
+--copy_files_wildcard  $MYSQL_TMP_DIR/export80/  $MYSQLD_DATADIR/import80/  t5*
+
+--echo # Copy exported v8.0 t6 files to MYSQL_TMP_DIR/external_dir/import80.
+--copy_files_wildcard  $MYSQL_TMP_DIR/export80/  $EXTERNAL_DIR/import80/  t6*
+
+--echo # Copy exported v8.0 t7 files to MYSQLD_DATADIR/import80.
+--copy_files_wildcard  $MYSQL_TMP_DIR/export80/  $MYSQLD_DATADIR/import80/  t7*
+
+--echo # Copy exported v8.0 t8 files to MYSQL_TMP_DIR/external_dir/import80.
+--copy_files_wildcard  $MYSQL_TMP_DIR/export80/  $EXTERNAL_DIR/import80/  t8*
+
+--source include/restart_mysqld.inc
+
+ALTER TABLE `t1` IMPORT TABLESPACE;
+select * from t1;
+
+ALTER TABLE `t2` IMPORT TABLESPACE;
+select * from t2;
+
+ALTER TABLE `t3` IMPORT TABLESPACE;
+select * from t3;
+
+ALTER TABLE `t4` IMPORT TABLESPACE;
+select * from t4;
+
+ALTER TABLE `t5` IMPORT TABLESPACE;
+select * from t5;
+
+ALTER TABLE `t6` IMPORT TABLESPACE;
+select * from t6;
+
+ALTER TABLE `t7` IMPORT TABLESPACE;
+select * from t7;
+
+ALTER TABLE `t8` IMPORT TABLESPACE;
+select * from t8;
+
+--echo # Cleanup
+DROP TABLE `t1`, `t2`, t3, t4, `t5`, `t6`, t7, t8;
+USE `test`;
+DROP SCHEMA `import80`;
+--force-rmdir $EXTERNAL_DIR/import80
+--force-rmdir $EXTERNAL_DIR
+--force-rmdir $MYSQL_TMP_DIR/export57
+--force-rmdir $MYSQL_TMP_DIR/export80
+--remove_file $MYSQL_TMP_DIR/data57_exported.zip

sql/sql_thd_api.cc

@@ -310,13 +310,18 @@ int thd_tablespace_op(const MYSQL_THD thd) {
     statement, so this function must check both the SQL command
     code and the Alter_info::flags.
   */
-  if (thd->lex->sql_command != SQLCOM_ALTER_TABLE) return 0;
-  assert(thd->lex->alter_info != nullptr);
+  int ret = 0;
+
+  if (thd->lex->sql_command == SQLCOM_ALTER_TABLE) {
+    if (thd->lex->alter_info->flags & Alter_info::ALTER_DISCARD_TABLESPACE) {
+      ret = Alter_info::ALTER_DISCARD_TABLESPACE;
+    }
+    if (thd->lex->alter_info->flags & Alter_info::ALTER_IMPORT_TABLESPACE) {
+      ret = Alter_info::ALTER_IMPORT_TABLESPACE;
+    }
+  }
 
-  return (thd->lex->alter_info->flags & (Alter_info::ALTER_DISCARD_TABLESPACE |
-                                         Alter_info::ALTER_IMPORT_TABLESPACE))
-             ? 1
-             : 0;
+  return (ret);
 }
 
 static void set_thd_stage_info(MYSQL_THD thd, const PSI_stage_info *new_stage,