fix: prevent head-of-line blocking in RedisWorker fetch_task() (#7900) - #7987
Merged
gerrod3 merged 1 commit intoAug 18, 2026
Merged
Conversation
gerrod3
requested changes
Aug 17, 2026
…7900) When many waiting tasks need the same blocked exclusive resource, fetch_task() now excludes known-blocked resources from subsequent DB queries using reserved_resources_record__overlap, leveraging the existing partial GIN index. The taken_exclusive/taken_shared sets accumulate across iterations to preserve FIFO ordering. Closes pulp#7900
dkliban
force-pushed
the
fix/7900-fetch-task-head-of-line-blocking
branch
from
August 17, 2026 18:27
30a1e9b to
f405316
Compare
Member
Author
|
@gerrod3 I addressed your feedback. PLease re-review. Thank you! |
Backport to 3.116: 💚 backport PR created✅ Backport PR branch: Backported as #7989 🤖 @patchback |
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When the queue head is dominated by tasks needing the same blocked exclusive resource (e.g., 17,000+
general_createtasks for the same Python repository),fetch_task()scanned from position 0 on every iteration, discarding knowledge of blocked resources and starving tasks with free resources for hours.Fix: Track resources reported as blocked by
acquire_locksand exclude them from subsequent DB queries using PostgreSQL's array overlap operator (&&), leveraging the existing GIN index (pulp_task_resources_index) onreserved_resources_record.Key changes
blocked_resourcesset acrossfetch_task()iterationsreserved_resources_record__overlaptaken_exclusive/taken_sharedFIFO knowledge across iterationsProperties
acquire_locksLua script unchangedhandle_tasks()interface --fetch_task()still returns Task or Noneblocked_resourcesis local to eachfetch_task()call -- starts fresh after each task completesshared:prefixed)Test plan
acquire_lockscalled <= 21 times (was 51)Closes: #7900