[13.x] Add optional disk storage for large SQS queue payloads#59734
Open
Orrison wants to merge 4 commits intolaravel:13.xfrom
Open
[13.x] Add optional disk storage for large SQS queue payloads#59734Orrison wants to merge 4 commits intolaravel:13.xfrom
Orrison wants to merge 4 commits intolaravel:13.xfrom
Conversation
Signed-off-by: Kevin Ullyott <ullyott.kevin@gmail.com>
Signed-off-by: Kevin Ullyott <ullyott.kevin@gmail.com>
Signed-off-by: Kevin Ullyott <ullyott.kevin@gmail.com>
Signed-off-by: Kevin Ullyott <ullyott.kevin@gmail.com>
79eb26e to
1dcb06d
Compare
|
This feature would be a welcome addition to the framework! We're currently evaluating adoption of SQS as our queue driver, and this would alleviate one of our concerns with doing so. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When using the SQS queue driver, sending a job with a payload that exceeds the queue's maximum message size causes AWS to reject it with an
InvalidParameterValueerror:This adds native support for automatically offloading large payloads to a configured filesystem disk (e.g. S3) and sending a small pointer through SQS instead. The worker then fetches the full payload from disk when processing the job.
This is a well-known and AWS-recommended strategy for working around this limitation. And Java and Python implementations exist doing it: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-managing-large-messages.html
This is fully opt-in and backwards compatible, existing SQS users are unaffected. The feature is controlled by a new
extended_store_optionsblock in the SQS queue connection config, withenableddefaultingto
false:How it works
The @pointer key uses an @ prefix which cannot be a PHP class property, so there is no risk of collision with normal job payloads.