MDEV-14985 innodb_undo_log_truncate may be blocked if transactions were recovered at startup

The field trx_rseg_t::trx_ref_count that was added in WL#6965 in
MySQL 5.7.5 is being incremented twice if a recovered transaction
includes both undo log partitions insert_undo and update_undo.

This reference count is being used in trx_purge(), which invokes
trx_purge_initiate_truncate() to try to truncate an undo tablespace
file. Because of the double-increment, the trx_ref_count would never
reach 0.

It is possible that after the failed truncation attempt, the undo
tablespace would be disabled for logging any new transactions until
the server is restarted (hopefully after committing or rolling back
all transactions, so that no transactions would be recovered
on the next startup).

trx_resurrect_insert(), trx_resurrect_update(): Do not increment
trx_ref_count. Instead, let the caller do that.

trx_lists_init_at_db_start(): Increment rseg->trx_ref_count only
once for each recovered transaction. Adjust comments.
Finally, if innodb_force_recovery prevents the undo log scan,
do not bother iterating the empty lists.

Author: dr-m

storage/innobase/trx/trx0trx.cc

@@ -824,10 +824,6 @@ trx_resurrect_insert(
 	ut_d(trx->start_line = __LINE__);
 
 	trx->rsegs.m_redo.rseg = rseg;
-	/* For transactions with active data will not have rseg size = 1
-	or will not qualify for purge limit criteria. So it is safe to increment
-	this trx_ref_count w/o mutex protection. */
-	++trx->rsegs.m_redo.rseg->trx_ref_count;
 	*trx->xid = undo->xid;
 	trx->id = undo->trx_id;
 	trx->rsegs.m_redo.insert_undo = undo;
@@ -934,10 +930,6 @@ trx_resurrect_update(
 	trx_rseg_t*	rseg)	/*!< in/out: rollback segment */
 {
 	trx->rsegs.m_redo.rseg = rseg;
-	/* For transactions with active data will not have rseg size = 1
-	or will not qualify for purge limit criteria. So it is safe to increment
-	this trx_ref_count w/o mutex protection. */
-	++trx->rsegs.m_redo.rseg->trx_ref_count;
 	*trx->xid = undo->xid;
 	trx->id = undo->trx_id;
 	trx->rsegs.m_redo.update_undo = undo;
@@ -991,19 +983,22 @@ trx_lists_init_at_db_start()
 
 	purge_sys = UT_NEW_NOKEY(purge_sys_t());
 
-	if (srv_force_recovery < SRV_FORCE_NO_UNDO_LOG_SCAN) {
-		trx_rseg_array_init();
+	if (srv_force_recovery >= SRV_FORCE_NO_UNDO_LOG_SCAN) {
+		return;
 	}
 
+	trx_rseg_array_init();
+
 	/* Look from the rollback segments if there exist undo logs for
 	transactions. */
 
 	for (ulint i = 0; i < TRX_SYS_N_RSEGS; ++i) {
 		trx_undo_t*	undo;
 		trx_rseg_t*	rseg = trx_sys->rseg_array[i];
 
-		/* At this stage non-redo rseg slots are all NULL as they are
-		re-created on server start and existing slots are not read. */
+		/* Some rollback segment may be unavailable,
+		especially if the server was previously run with a
+		non-default value of innodb_undo_logs. */
 		if (rseg == NULL) {
 			continue;
 		}
@@ -1013,6 +1008,11 @@ trx_lists_init_at_db_start()
 		     undo != NULL;
 		     undo = UT_LIST_GET_NEXT(undo_list, undo)) {
 
+			/* trx_purge() will not run before we return,
+			so we can safely increment this without
+			holding rseg->mutex. */
+			++rseg->trx_ref_count;
+
 			trx_t*	trx;
 
 			trx = trx_resurrect_insert(undo, rseg);
@@ -1037,6 +1037,7 @@ trx_lists_init_at_db_start()
 
 			if (trx == NULL) {
 				trx = trx_allocate_for_background();
+				++rseg->trx_ref_count;
 
 				ut_d(trx->start_file = __FILE__);
 				ut_d(trx->start_line = __LINE__);