Fix scheduler/triggerer deadlock on task_instance for deferrable tasks#65836
Open
rapsealk wants to merge 1 commit intoapache:mainfrom
Open
Fix scheduler/triggerer deadlock on task_instance for deferrable tasks#65836rapsealk wants to merge 1 commit intoapache:mainfrom
rapsealk wants to merge 1 commit intoapache:mainfrom
Conversation
boring-cyborg
Bot
added
area:Scheduler
The scheduler's check_trigger_timeouts() and the triggerer's Trigger.clean_unused() both issued bulk UPDATEs against task_instance rows referenced via different index paths (state+trigger_timeout vs trigger_id), letting InnoDB acquire row+gap locks in different orders and producing classic A-B / B-A deadlocks under HA scheduler deployments — especially with multiple scheduler replicas. Both writers now select candidate ids in primary-key order with SELECT ... FOR UPDATE SKIP LOCKED, then UPDATE ... WHERE id IN (...) in bounded batches. The deterministic PK lock order eliminates the cross-index deadlock; SKIP LOCKED keeps concurrent scheduler replicas from blocking each other. closes: apache#65818 Assisted-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
rapsealk
force-pushed
the
fix-65818-trigger-id-deadlock
branch
from
e35be28 to
aa571ff
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On HA scheduler deployments backed by MySQL, the scheduler's
SchedulerJobRunner.check_trigger_timeouts()and the triggerer'sTrigger.clean_unused()both issue bulk UPDATEs againsttask_instancerows but reach those rows via different indexes —(state, trigger_timeout)vs(trigger_id). InnoDB therefore acquires row + gap locks in different orders for the two queries, producing classic A→B / B→A deadlocks once a deferrable workload has overlapping rows in flight (especially with multiple scheduler replicas).This change reshapes both writers to:
SELECT id ... ORDER BY id LIMIT N FOR UPDATE SKIP LOCKEDUPDATE task_instance ... WHERE id IN (...)so that both paths take row locks in deterministic primary-key order, eliminating the cross-index deadlock.
SKIP LOCKEDadditionally keeps concurrent scheduler replicas from blocking on each other. Behaviour is unchanged: same predicates, same SET clauses, just batched and locked.Two regression tests cover convergence when more rows match than fit in a single batch (the batch size is monkeypatched to 2 in the tests).
closes: #65818
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Opus 4.7 (1M context) following the guidelines