[YSQL] ERROR: The catalog snapshot used for this transaction has been invalidated: MISMATCHED_SCHEMA, DDL is taking more time to execute

[agsh-yb]

Jira Link: DB-9470

Description

Error observed in 2.14.14.0-b17 quite frequently in testtabletsplittingwithpitrcopybrupgradesindexmatview-aws-rf3-upgrade-2.14.14.0-b17, While performing some DDL’s (Alter table) commands

(   123.480s) Execute DDLs' after Create BACKUP_001 : >>> Integration Test Failed <<< 
Query error: The catalog snapshot used for this transaction has been invalidated: MISMATCHED_SCHEMA
CONTEXT:  Catalog Version Mismatch: A DDL occurred while processing this query. Try again.

Note: No parallel DDL are there, Also if we add wait of 1 second after every DDL command this error is not observed
I assume alter table commands are taking time here, similar error issue #12621
Steps test is following is:

Create tables : Success
	testtabletsplittingwithpitrcopybrupgradesindexmatview-aws-rf3-upgrade-2.14.13.0-b13: Start
	(     0.667s) User Login : Success
	(     0.284s) Refresh YB Version : Success
	(    94.395s) Setup Provider : Success
	(     0.193s) Enable RBAC Flag : Success
	(     3.562s) Download old release: 2.14.13.0-b13 : Success
	(     0.311s) Updating Health Check Interval to 60000 sec : Success
	(   519.630s) Create universe sagr-isd9902-9a0cb6806c-20240103-112936 : Success
	(    25.500s) Create Database tablet_splitting_pitr : Success
	(    22.357s) Create tables : Success
	(   163.076s) Load data to employees_hash : Success
	(   163.145s) Load data to employees_range : Success
	(   156.700s) Load data to co_table_1 : Success
	(     3.442s) Create Secondary Index. : Success
	(     3.381s) Create Unique Index. : Success
	(     3.474s) Create Partial Index. : Success
	(    23.481s) Create Secondary Index. : Success
	(     3.406s) Create Unique Index. : Success
	(     3.483s) Create Partial Index. : Success
	(    14.788s) Create Secondary Index. : Success
	(     4.353s) Create Unique Index. : Success
	(     3.382s) Create Partial Index. : Success
	(    41.881s) Create Snapshot Schedule : Success
	(    41.169s) Create Snapshot Schedule colocated : Success
	(     3.486s) Execute COPY TO COPY FROM : Success
	(    31.603s) Drop Colocated tables : Success
	(   230.939s) Load data to employees_hash : Success
	(   290.376s) Load data to employees_range : Success
	(   370.444s) PITR at time(1) : Success
	(    60.097s) PITR at time(1) : Success
	(     9.570s) Create Materialized Views : Success
	(   155.701s) Create Backups : Success
	(   241.338s) Create Backups : Success
	(   126.847s) Execute DDLs' after Create BACKUP_001 : >>> Integration Test Failed <<< 
Query error: The catalog snapshot used for this transaction has been invalidated: MISMATCHED_SCHEMA
CONTEXT:  Catalog Version Mismatch: A DDL occurred while processing this query. Try again.

Observing failure from version2.14.13.0-b13

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

• I confirm this issue does not contain any sensitive information.