Daily major freeze fails to launch due to DDL conflict, reaching retry limit (errcode=-4023)

[15640940228-ctrl]

Environment

• Version: observer (OceanBase seekdb)
• Build Info:
  • Source revision: 100000232026041319-614e895bcf06c9b9d092115dc7d16219dc95d96e
  • REVISION: 100000...

Description

A daily major freeze operation is not being launched because it is blocked by a DDL conflict. The process has reached its configured retry time limit, resulting in error code -4023.

Impact

The scheduled major freeze, which is critical for storage management and performance, is failing to execute. This could lead to unmerged data, increased storage usage, and potential performance degradation over time.

[footka]

​

Thread 238540 (T1_RCSrv0)

#0 0x00007ff4038b29cd in ???

#1 0x0000563a850dfe43 in futex_hook from /obdata/data/ddltmp/obm.z1.obs0/bin/observer

#2 0x0000563a964c5c9c in int oceanbase::common::ObLatchWaitQueue::wait(oceanbase::common::ObWaitProc&, unsigned int, unsigned int, oceanbase::common::ObLatch::LowTryRDLock&, oceanbase::common::ObLatch::LowTryRDLock&, long) from /obdata/data/ddltmp/obm.z1.obs0/bin/observer

#3 0x0000563a84fe8c2f in oceanbase::common::ObLatch::rdlock(unsigned int, long) from /obdata/data/ddltmp/obm.z1.obs0/bin/observer

#4 0x0000563a93ef3f1e in oceanbase::share::ObPluginVectorIndexLoadScheduler::switch_to_leader from /obdata/data/ddltmp/obm.z1.obs0/bin/observer

#5 0x0000563a8a1f03e0 in oceanbase::logservice::ObRoleChangeHandler::switch_to_leader(oceanbase::logservice::RCDiagnoseInfo&) from /obdata/data/ddltmp/obm.z1.obs0/bin/observer

#6 0x0000563a8a1fc762 in oceanbase::logservice::ObRoleChangeService::switch_follower_to_leader_(long, oceanbase::storage::ObLS*, oceanbase::logservice::ObRoleChangeService::RetrySubmitRoleChangeEventCtx&) from /obdata/data/ddltmp/obm.z1.obs0/bin/observer

#7 0x0000563a8a1f3d69 in oceanbase::logservice::ObRoleChangeService::handle_role_change_event_(oceanbase::logservice::RoleChangeEvent const&, oceanbase::logservice::ObRoleChangeService::RetrySubmitRoleChangeEventCtx&) from /obdata/data/ddltmp/obm.z1.obs0/bin/observer

#8 0x0000563a8a1f31ff in oceanbase::logservice::ObRoleChangeService::handle(void*) from /obdata/data/ddltmp/obm.z1.obs0/bin/observer

#9 0x0000563a854f0979 in oceanbase::common::ObSimpleThreadPoolBase::run1 from /obdata/data/ddltmp/obm.z1.obs0/bin/observer

#10 0x0000563a964b04ff in oceanbase::lib::Threads::run(long) from /obdata/data/ddltmp/obm.z1.obs0/bin/observer

#11 0x0000563a964c0425 in oceanbase::lib::Thread::__th_start(void*) from /obdata/data/ddltmp/obm.z1.obs0/bin/observer

#12 0x00007ff403a5c3fb in ???

#13 0x00007ff4038b2e83 in ???

[footka]

[2026-04-14 23:33:01.127018] INFO [SHARE] operator (ob_plugin_vector_index_serialize.cpp:417) [239133][T1_VectorAsyncT][T1][Y0-0000000000000000-0-0] [lt=11] [vec index debug] success write one data into lob tablet(src_lob={ptr:0x7ff1e6c04050, size:89204, has_lob_header:false, compatable inrow data:""}, lob_param.lob_meta_tablet_id_={id:1152921504607035715}, lob_param.tx_desc_={this:0x7ff327fe4030, tx_id:{txid:36041983}, state:2, addr:":16607", tenant_id:1, session_id:0, assoc_session_id:0, client_sid:0, xid:NULL, access_mode:0, tx_consistency_type:0, isolation:1, snapshot_version:{val:18446744073709551615, v:3}, snapshot_scn:0, active_scn:{branch:0, seq:1}, op_sn:2, alloc_ts:1776180780786013, active_ts:1776180780786013, commit_ts:-1, finish_ts:-1, timeout_us:86400000000, lock_timeout_us:-1, expire_ts:1776267180786013, coord_id:{id:-1}, parts:[{id:{id:1}, addr:":16607", epoch:31880350314535883, first_scn:{branch:0, seq:55}, last_scn:{branch:0, seq:55}, last_touch_ts:1776152396635206, flag_.flag_bit_:{is_dup_ls:0, is_clean:false}}], exec_info_reap_ts:1776152396635205, commit_version:{val:18446744073709551615, v:3}, commit_times:0, commit_cb:null, cluster_id:1, cluster_version:4295098368, seq_base:1776152396635129, flags_.SHADOW:false, flags_.INTERRUPTED:false, flags_.BLOCK:false, flags_.REPLICA:false, conflict_txs:, abort_cause:0, commit_expire_ts:0, commit_task_.is_registered:false, modified_tables:, last_rc_snapshot_version:{val:1776180780784944000, v:0}, ref:2})

[2026-04-14 23:35:14.243142] WDIAG [STORAGE.TRANS] get_tx_ctx (ob_trans_ctx_mgr_v4.cpp:1899) [239133][T1_VectorAsyncT][T1][Y0-0000000000000000-0-0] [lt=9706][errcode=-4038] get transaction context error(ret=-4038, ret="OB_NOT_MASTER", ls_id={id:1}, tx_id={txid:36041983})

[2026-04-14 23:35:14.243176] WDIAG [STORAGE.TRANS] local_ls_commit_tx_ (ob_trans_service_v4.cpp:1794) [239133][T1_VectorAsyncT][T1][Y0-0000000000000000-0-0] [lt=33][errcode=-4038] get coordinator tx context fail(ret=-4038, tx_id={txid:36041983}, coord={id:1})

​

[footka]

[2026-04-14 23:35:04.826064] INFO [STANDBY] do_change_ls_access_mode_ (ob_tenant_role_transition_service.cpp:844) [493086][T1_L0_G0][T1][Y40DF7F000001-00064F66B60658D8-0-0] [lt=8] [ROLE_TRANSITION] change ls access mode success(ret=0, ret="OB_SUCCESS", info={tenant_id:1, ls_id:{id:1}, mode_version:113, access_mode:1, ref_scn:{val:1776180345352703000, v:0}, sys_ls_end_scn:{val:1776180904810852000, v:0}}, target_access_mode=2, ref_scn={val:1, v:0})

[footka]

Thread 239133: Holds the write lock for adapter_map_rwlock_.

↓ Waits for commit to complete.

↓ Commit RPC → get_tx_ctx → OB_NOT_MASTER (LS tx ctx mgr is a follower).

↓ Requires LS tx ctx mgr to restore leader status.

↓

RCSrv0: Responsible for handling the role change callback → calls TRANS_SERVICE.switch_to_leader to restore LS tx ctx mgr.

↓ However, within the switch_to_leader method of the vector index scheduler...

↓ Waits for a read lock on adapter_map_rwlock_.

↓

↓ ← Blocked by the write lock held by Thread 239133.

[footka]

The direct cause is that during the role change to primary on the standby, the role_change callback task was blocked by ObPluginVectorIndexLoadScheduler::switch_to_leader. This prevented ObTenantMajorFreeze::switch_to_leader from executing, which in turn caused subsequent freeze and merge_scheduler processes to fail to proceed normally.

[footka]

Time A (Primary) B (Standby)

---

04-14 17:00 Vector background thread (Thread 239133)
Holds lock M (adapter_map_rwlock_ write lock)
Starts transaction T (txid: 36041983)
Timeout: 24h
Transaction T writes persistent redo log

04-14 23:35:04 Switches to standby
switch_to_follower_forcedly
Transaction T has persistent logs, not killed
23:35:xx Switches to primary
Transaction T synced from A already has some redo logs, new primary chases abort log

04-14 23:35:04 ~ 23:43 A acts as standby, syncs abort log from B
Participant exits normally after replaying abort
But scheduler cannot finish
Continuously fails to send messages to participant
(trans_service's is_master_ check fails, returns OB_NOT_MASTER)
23:43:xx Switches to standby

04-14 23:43:25 Switches to primary
RCSrv0 processes FOLLOWER_2_LEADER
Calls sub-handler's switch_to_leader in cursor order

04-14 23:43:25.081 cursor=1 TRANS_SERVICE
switch_to_leader completes
LS tx ctx mgr recovers to L_WORKING
Scheduler successfully sends message to participant
But RPC framework has a bug
Callback not initialized as expected
Causing transaction processing logic to be missing
Scheduler still cannot exit
(cb_executed=false)

04-14 23:43:25.083 cursor=46 VECTOR_INDEX
switch_to_leader executes
refresh_adapter_rb_flag
Attempts to acquire lock M (read lock)
-> Blocked by Thread 239133's write lock
-> RCSrv0 HANGS
*** Deadlock formed ***

04-15 02:00:03 daily_launcher triggers major freeze
merge_scheduler_.is_paused=true
Returns OB_LEADER_NOT_EXIST
Begins retry (every 120 seconds)
(merge_scheduler not recovered, because subsequent role change events cannot be processed)

04-15 04:01:01 Retry times out after 2h
EDIAG: "daily major freeze is not launched due to ddl conflict"

04-15 17:00:40 Transaction T times out after 24h and exits
Thread 239133 releases lock M
*** Deadlock resolved ***