Bug#35277407 InnoDB：trx hangs due to wrong trx->in_innodb value

This commit backports the fix to 8.0

This patch will solve the following duplicates of this bug:
  Bug #112425: trx_t might be Use-After-Free in innobase_commit_by_xid
  Bug #99643:  innobase_commit_by_xid/innobase_rollback_by_xid is not
thread safe
  Bug #105036: trx would be used after free in `innobase_commit_by_xid`
and rollback

Background:
  TrxInInnoDB is a RAII wrapper for trx_t object used to track if the
transaction's thread is currently executing within InnoDB code. It is
acquired on all entry points, and as Innodb can be entered "recursively",
the trx->in_depth is used to track the balance of enters and exits.
  On the outermost enter, the thread additionally checks if
trx->in_innodb has the TRX_FORCE_ROLLBACK (0x8000 0000) flag set, which
means a high priority transaction is attempting an asynchronous rollback
of this transaction, so to avoid races, this thread should wait for the
rollback to complete.

Issue:
  TrxInInnoDB's destructor calls exit which resets in_depth and
in_innodb increased by enter. However innobase_commit_by_xid and
innobase_rollback_by_xid calls trx_free_for_background which returns the
trx back to the pool, before the destructor is called. If this trx is
being reused by another thread, it can lead to data-race and corrupted
value of in_depth and in_innodb. If in_depth gets the value of -1,
subsequent calls to enter and exit will bump in_innodb by one. This can
lead to indefinite wait if in_innodb reaches TRX_FORCE_ROLLBACK.

Fix:
  Ensure that TrxInInnoDB calls exit before returning the trx object to
the pool. Further add checks to catch corrupt values of in_depth when
freeing trx. Trx state validation before free was missed in
trx_free_prepared_or_active_recovered

Thanks to Shaohua Wang (Alibaba, Ex-Innodb) for the contribution

Change-Id: Ibf79bec85ffa0eaf65f565c169db61536bff10a2

Author: ram1048

storage/innobase/handler/ha_innodb.cc

@@ -20073,9 +20073,11 @@ static xa_status_code innobase_commit_by_xid(
   trx_t *trx = trx_get_trx_by_xid(xid);
 
   if (trx != nullptr) {
-    TrxInInnoDB trx_in_innodb(trx);
+    {
+      TrxInInnoDB trx_in_innodb(trx);
 
-    innobase_commit_low(trx);
+      innobase_commit_low(trx);
+    }
     ut_ad(trx->mysql_thd == nullptr);
     /* use cases are: disconnected xa, slave xa, recovery */
     trx_deregister_from_2pc(trx);
@@ -20101,9 +20103,12 @@ static xa_status_code innobase_rollback_by_xid(
   trx_t *trx = trx_get_trx_by_xid(xid);
 
   if (trx != nullptr) {
-    TrxInInnoDB trx_in_innodb(trx);
+    int ret;
+    {
+      TrxInInnoDB trx_in_innodb(trx);
 
-    int ret = innobase_rollback_trx(trx);
+      ret = innobase_rollback_trx(trx);
+    }
 
     trx_deregister_from_2pc(trx);
     ut_ad(!trx->will_lock);

storage/innobase/include/trx0types.h

@@ -64,7 +64,7 @@ static const uint32_t TRX_FORCE_ROLLBACK_DISABLE = 1 << 29;
 /** Mark the transaction for forced rollback */
 static const uint32_t TRX_FORCE_ROLLBACK = 1U << 31;
 
-/** For masking out the above four flags */
+/** For masking out the above flags */
 static const uint32_t TRX_FORCE_ROLLBACK_MASK = 0x1FFFFFFF;
 
 /** Transaction execution states when trx->state == TRX_STATE_ACTIVE */

storage/innobase/trx/trx0trx.cc

@@ -265,6 +265,9 @@ struct TrxFactory {
 
     trx->dict_operation_lock_mode = 0;
 
+    ut_a(trx->in_innodb == 0);
+    ut_a(trx->in_depth == 0);
+
     trx->xid = ut::new_withkey<xid_t>(UT_NEW_THIS_FILE_PSI_KEY);
 
     trx->detailed_error = reinterpret_cast<char *>(
@@ -501,11 +504,12 @@ static void trx_free(trx_t *&trx) {
 
   ut_ad(trx->read_view == nullptr);
   ut_ad(trx->is_dd_trx == false);
+  ut_ad(trx->in_depth == 0);
 
   /* trx locking state should have been reset before returning trx
   to pool */
   ut_ad(trx->will_lock == 0);
-
+  trx->in_innodb &= TRX_FORCE_ROLLBACK_MASK;
   trx_pools->mem_free(trx);
 
   trx = nullptr;
@@ -620,6 +624,7 @@ void trx_free_prepared_or_active_recovered(trx_t *trx) {
   trx->state.store(TRX_STATE_NOT_STARTED, std::memory_order_relaxed);
   trx->will_lock = 0;
 
+  trx_validate_state_before_free(trx);
   trx_free(trx);
 }