feat!: prevent stale jobs queue jobs#17220
Open
AlessioGr wants to merge 34 commits into
Open
Conversation
Contributor
📦 esbuild Bundle Analysis for payloadThis analysis was generated by esbuild-bundle-analyzer. 🤖
Largest pathsThese visualization shows top 20 largest paths in the bundle.Meta file: packages/next/meta_index.json, Out file: esbuild/index.js
Meta file: packages/payload/meta_index.json, Out file: esbuild/index.js
Meta file: packages/payload/meta_shared.json, Out file: esbuild/exports/shared.js
Meta file: packages/richtext-lexical/meta_client.json, Out file: esbuild/exports/client_optimized/index.js
Meta file: packages/ui/meta_client.json, Out file: esbuild/exports/client_optimized/index.js
Meta file: packages/ui/meta_shared.json, Out file: esbuild/exports/shared_optimized/index.js
DetailsNext to the size is how much the size has increased or decreased compared with the base branch of this PR.
|
AlessioGr
changed the title
AlessioGr
commented
Jul 23, 2026
| remainingJobsFromQueried: 0, | ||
| }) | ||
| expect(consoleCount).toHaveBeenCalledTimes(14) // Should be 14 sql calls if the optimizations are used. If not, this would be 23 calls | ||
| expect(consoleCount).toHaveBeenCalledTimes(16) |
Member
Author
There was a problem hiding this comment.
+2 queries, because of the new where queries that prevent updated when processingUntil is expired. These cannot go through the useOptimizedUpsertRow drizzle fast path, because:
- when updating just the job log, we now need to check 2 tables in order to verify
processingUntilinstead of one - even when not updating the job log,
useOptimizedUpsertRowcurrently does not allowwherequeries.
We can optimize again in the future. Getting it down to 14 again would require complex new logic in upsertRow
AlessioGr
marked this pull request as ready for review
July 23, 2026 18:43
AlessioGr
enabled auto-merge (squash)
July 23, 2026 18:43
r1tsuu
reviewed
Jul 24, 2026
Comment on lines
-240
to
-243
| admin: { | ||
| hidden: true, | ||
| readOnly: true, | ||
| }, |
Member
There was a problem hiding this comment.
Is this intentional or you removed it temporary for testing?
Member
Author
There was a problem hiding this comment.
It's intentional, added it by accident in my last PR.
The Jobs collection already is hidden by default as a whole. If you go ahead and unhide it, you usually do so for debugging purposes. And for debugging purposes, you'll want to see this field in the admin panel
Comment on lines
+289
to
+291
| processingLease: { | ||
| duration: 5 * 60 * 1000, | ||
| safetyBuffer: 60 * 1000, |
Member
There was a problem hiding this comment.
Should we be able to configure duration on a task/workflow level? or having it as global-only is fine? In your project you might have very different jobs in terms of complexity
Member
Author
There was a problem hiding this comment.
I'd think about this for the future if there is a need for this, no need to bloat the config if not necessary right now.
Generally I would say global-only is enough. The complexity of a job usually doesnt really change what processingLease duration you'll want to use. The processing lease continuously renews itself, so it automatically scales with the duration of a job. It's not a job timeout.
What does affect this is things like lambda duration, if you're hosting on lambdas. E.g. if the worker process shuts down after 4 minutes, there is no point keeping a 20 minute processing lease. And that would be something that takes effect globally.
Where it does maybe make sense is configuration based on the queue name, if different queues use different workers and worker configurations. But yea, I'd worry about that in the future. And even then, different workers for different queues can control this configuration individually through environment variables
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR lets Payload recover Jobs Queue jobs after a worker crashes. It replaces the permanent
processingflag with a renewableprocessingUntillease and builds on #17441, which makes the initial job claim safe when multiple workers poll the same queue.Why
Previously, a worker set
processing: truewhen it claimed a job. If that process crashed, nothing reset the flag, so Payload treated the job as running forever.For example, a schedule runs every 10 minutes but does not queue a new job while the previous one is processing. If its worker crashes, the old job stays at
processing: true, cannot be picked up again, and can block every future scheduled run.How it works
A job is claimable when
processingUntilis missing or expired. Claiming it atomically writes:processingUntil: how long Payload should consider the worker activeprocessingToken: which worker attempt currently owns the jobWhile the job runs, a heartbeat renews
processingUntilevery third of the lease duration. A healthy job can therefore run longer than one lease; this is not a task timeout.Worker-owned updates - including heartbeats, task logs, completion, and handled errors - must still match the worker's token and have enough lease time remaining. If an update no longer matches, Payload throws an internal
JobRunAbortedErrorand stops accepting results from that worker. This prevents a timed-out worker from overwriting a newer attempt.When a worker disappears, its heartbeat stops. After the lease expires, the next normal
payload.jobs.run()orrunByID()call can claim the job directly with a new token and lease. No separate cleanup step is required.A crash intentionally does not increment
totalTriedor consume the handler's retry configuration. Completed task logs are kept, so recovery continues from the remaining tasks.Configuration
The default lease is 20 minutes and the default safety buffer is 30 seconds:
durationcontrols how long a job remains owned without a successful heartbeat, and will delay recovery. 20 minutes was chosen as the default, because recovery usually is rare and not time-sensitive.safetyBufferis the minimum time that must remain before a worker starts a job update. It gives db adapters that read and write separately time to finish before the lease expires. It should be longer than a normal database update and shorter than the lease duration.Why the token and safety buffer are both needed
#17441 protects claim time: only one worker can change a pending job into a claimed job.
This PR must also protect later writes. After a lease expires, Worker B can recover a job while Worker A is still finishing old work.
processingUntilalone only says that some worker has an active lease. The stableprocessingTokenidentifies whether the update belongs to Worker A or Worker B.Some db adapters first find a matching job and then write it. Without a safety window, this race is possible:
The safety buffer prevents A from starting that update near lease expiry. This is a cross-db-adapter safety margin, not an atomic database guarantee. It can be increased for slower remote adapters, if you expect an update to take longer than 30 seconds.
MongoDB applies the ownership query and update atomically with
findOneAndUpdate()orupdateMany(), so it does not rely on the buffer for correctness. The buffer can be set to0for a db adapter like mongodb with the same guarantee.Breaking change
The
processingfield onpayload-jobsis replaced by the nullable, indexedprocessingUntildate. Custom collection overrides and direct database queries that useprocessingmust be updated.Existing SQL databases need a migration that replaces this column. Migration files will be added in a separate PR; MongoDB does not require a schema migration.