Skip to content

fix: prevent head-of-line blocking in RedisWorker fetch_task() (#7900) - #7987

Merged
gerrod3 merged 1 commit into
pulp:mainfrom
dkliban:fix/7900-fetch-task-head-of-line-blocking
Aug 18, 2026
Merged

fix: prevent head-of-line blocking in RedisWorker fetch_task() (#7900)#7987
gerrod3 merged 1 commit into
pulp:mainfrom
dkliban:fix/7900-fetch-task-head-of-line-blocking

Conversation

@dkliban

@dkliban dkliban commented Aug 17, 2026

Copy link
Copy Markdown
Member

Summary

When the queue head is dominated by tasks needing the same blocked exclusive resource (e.g., 17,000+ general_create tasks for the same Python repository), fetch_task() scanned from position 0 on every iteration, discarding knowledge of blocked resources and starving tasks with free resources for hours.

Fix: Track resources reported as blocked by acquire_locks and exclude them from subsequent DB queries using PostgreSQL's array overlap operator (&&), leveraging the existing GIN index (pulp_task_resources_index) on reserved_resources_record.

Key changes

  • Accumulate blocked_resources set across fetch_task() iterations
  • Exclude tasks with blocked resources at DB level via reserved_resources_record__overlap
  • Preserve taken_exclusive/taken_shared FIFO knowledge across iterations
  • Use fixed-size batches instead of exponential doubling from position 0

Properties

  • FIFO fairness within same resource preserved -- tasks for the same resource still execute in creation order
  • No changes to Redis lock mechanism -- acquire_locks Lua script unchanged
  • No changes to handle_tasks() interface -- fetch_task() still returns Task or None
  • blocked_resources is local to each fetch_task() call -- starts fresh after each task completes
  • Conservative resource exclusion -- only raw resource names added (not shared: prefixed)

Test plan

  • Unit test: 200 tasks on 10 blocked resources + 1 free task -- verifies acquire_locks called <= 21 times (was 51)
  • Functional test: 30 blocked tasks + 1 free-resource task -- verifies free task completes within 30 seconds
  • Test discrimination verified (fails on original code, passes on fix)
  • Regression analysis: 0 regressions
  • Adversarial review: 0 findings

Closes: #7900

Comment thread CHANGES/7900.bugfix Outdated
Comment thread pulpcore/tasking/redis_worker.py Outdated
Comment thread pulpcore/tasking/redis_worker.py Outdated
Comment thread pulpcore/tests/unit/test_fetch_task_hol_blocking.py Outdated
Comment thread pulpcore/tasking/redis_worker.py Outdated
Comment thread pulpcore/tests/unit/test_fetch_task_hol_blocking.py Outdated
Comment thread pulpcore/tests/unit/test_fetch_task_hol_blocking.py Outdated
Comment thread pulpcore/tests/functional/api/test_fetch_task_hol_blocking.py Outdated
…7900)

When many waiting tasks need the same blocked exclusive resource,
fetch_task() now excludes known-blocked resources from subsequent DB
queries using reserved_resources_record__overlap, leveraging the
existing partial GIN index. The taken_exclusive/taken_shared sets
accumulate across iterations to preserve FIFO ordering.

Closes pulp#7900
@dkliban
dkliban force-pushed the fix/7900-fetch-task-head-of-line-blocking branch from 30a1e9b to f405316 Compare August 17, 2026 18:27
@dkliban

dkliban commented Aug 18, 2026

Copy link
Copy Markdown
Member Author

@gerrod3 I addressed your feedback. PLease re-review. Thank you!

@gerrod3 gerrod3 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks!

@gerrod3
gerrod3 merged commit ff9930d into pulp:main Aug 18, 2026
34 of 40 checks passed
@patchback

patchback Bot commented Aug 18, 2026

Copy link
Copy Markdown

Backport to 3.116: 💚 backport PR created

✅ Backport PR branch: patchback/backports/3.116/ff9930d1996a4da8f54f7dd97d88aca56479b332/pr-7987

Backported as #7989

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RedisWorker fetch_task() head-of-line blocking: workers starved when queue head is serialized on one resource (WORKER_TYPE=redis)

2 participants