Bug#32416819 ASSERTION "UT_LIST_GET_LEN.TRX_SYS->MYSQL_TRX_LIST. ==

0" ON SHUTDOWN

Background:
-----------
  If the pseudo_replica_mode is enabled and an internal binlog command
is run, the current thread is treated as an applier thread. If an XA
START query is issued at this stage, the applier thread will detach the
native transaction thread from all handlertons and store the backup in
hton->slot's ha_ptr_backup via rpl_detach_engine_ha_data. The detached
transaction is returned back during a commit or rollback via
rpl_reattach_engine_ha_data.

Issue:
------
  If a shutdown is issued in the same session which is marked as an
applier, immediately after an XA START, then the call to reattach via
rpl_reattach_engine_ha_data is missed. The shutdown is handled in
ha_rollback_low. Since no statements were run after the XA START,
ha_list is empty, skipping over the checks to reattach the engine data.
As the shutdown progresses, innodb asserts that the
trx_sys->mysql_trx_list (list of trx_t's created for mysql) is not empty
during trx_sys_close().
  A similar issue is seen in ha_commit_low. If ha_list is empty,
thd->rpl_reattach_engine_ha_data will be skipped. If a new XA START is
issued at this stage, then an assert is hit stating that ha_engine_data
is already detached

Fix:
----
  Ensure that the checks to reattach engine ha data happens even if
ha_list is empty in both ha_commit_low and ha_rollback_low

Change-Id: I13cbd289314437175e5813edc44978f027d97798

Author: ram1048

Diff for: mysql-test/include/xa_applier_shutdown.inc

@@ -0,0 +1,135 @@
+# This test checks the behavior of applier thread when shutdown is issued at
+# various stages of the XA transaction. The session thread is marked as an
+# applier thread by setting the pseudo_replica_mode and executing BINLOG''
+# query.
+#
+# References:
+#     Bug#32416819: ASSERTION "UT_LIST_GET_LEN.TRX_SYS->MYSQL_TRX_LIST. == 0" ON
+#     SHUTDOWN
+#
+# The include file lists the scenarios to be checked before running shutdown.
+# Enable the following flags to check different scenarios:
+# $create_table:       Create the table for the test and drop it at the end
+# $insert_table:       Insert 3 rows into the table containing single int col
+# $need_xa_end:        Includes the XA END statement
+# $need_xa_commit_one: Includes the XA COMMIT..ONE PHASE statement followed by
+#                      the previous set of statements
+# $need_xa_prepare:    Includes the XA PREPARE statement
+# $need_xa_commit:     Includes the XA COMMIT statement
+# $need_xa_rollback:   Includes the XA ROLLBACK statement
+# $restart_connection: If set, create new connection and restart the server
+#                      using the new connection
+#                      If unset, restart the server on the connection that
+#                      executes the transaction
+#
+# Note:
+# During shutdown, Innodb closes trx_sys. At this stage, it is expected that all
+# the trx_t's created for mysql is freed (mysql_trx_list - list of trx_t created
+# for mysql is empty). Whenever a `USE <database>` command is issued, Innodb
+# will create a trx_t (associated with the corresponding session THD) and adds
+# it to mysql_trx_list
+#
+# During replication, if the replication applier sees an 'XA START', it will
+# call detach_native_trx on all handlertons. In Innodb, this function will store
+# a backup of the above trx_t in ha_data, and replace it will nullptr
+#
+# In an ideal scenario, the applier will be able to finish the XA transaction,
+# and during the commit, ha_commit_low will restore the backup from ha_data
+# using reattach_native_trx. In case the applier is not able to finish the XA
+# transaction, then ha_rollback_low will reattach_native_trx
+#
+# In the bug scenario, this was happening only when ha_list was non-empty.
+# Hence, if shutdown is issued at this point, then the trx_t will never be
+# returned from ha_data causing the assertion in Innodb
+#
+# The current connection can simulate a replication applier by issuing:
+#     SET @@SESSION.pseudo_replica_mode=1;
+#     BINLOG '0';
+#
+# The BINLOG statement will initialize thd->rli_fake, even though the statement
+# fails. The thd->rli_fake will simulate the replication applier. '0' is used
+# for better readability
+
+if ($create_table) {
+  CREATE TABLE t1 (c1 INT);
+  INSERT INTO t1 VALUES (10);
+}
+
+SET @@SESSION.pseudo_replica_mode=1;
+--error ER_BASE64_DECODE_ERROR
+BINLOG '0';
+
+XA START 'test0';
+
+if ($insert_table) {
+  INSERT INTO t1 VALUES (1);
+  INSERT INTO t1 VALUES (2);
+  INSERT INTO t1 VALUES (3);
+}
+
+if ($need_xa_end) {
+  XA END 'test0';
+}
+
+if ($need_xa_commit_one) {
+  XA COMMIT 'test0' ONE PHASE;
+
+  XA START 'test0';
+  if ($insert_table) {
+    INSERT INTO t1 VALUES (4);
+    INSERT INTO t1 VALUES (5);
+    INSERT INTO t1 VALUES (6);
+  }
+
+  if ($need_xa_end) {
+    XA END 'test0';
+  }
+}
+
+if ($need_xa_prepare) {
+  XA PREPARE 'test0';
+}
+
+XA RECOVER;
+
+if ($restart_connection) {
+  # When restarting from a different connection, default connection must be
+  # handled separately. In this case, the default connection simply waits until
+  # it can reconnect after the server starts.
+
+  --connection default
+  --disable_reconnect
+
+  --connect ($restart_connection, localhost, root,,)
+  --connection $restart_connection
+  --source include/restart_mysqld.inc
+  --disconnect $restart_connection
+
+  --connection default
+  --enable_reconnect
+  --source include/wait_until_connected_again.inc
+}
+
+if (!$restart_connection) {
+  # The `connection default` is used to indicate that restart is issued in the
+  # current connection
+
+  --connection default
+  --source include/restart_mysqld.inc
+  --connection default
+}
+
+XA RECOVER;
+
+if ($need_xa_rollback) {
+  XA ROLLBACK 'test0';
+}
+
+if ($need_xa_commit) {
+  XA COMMIT 'test0';
+}
+
+if ($create_table) {
+  SELECT * FROM t1 ORDER BY c1;
+  DROP TABLE t1;
+}