[13.x] Optionally flush the SQS overflow store on queue:clear#60138
Merged
Conversation
|
Thanks for submitting a PR! Note that draft PRs are not reviewed. If you would like a review, please mark your pull request as ready for review in the GitHub user interface. Pull requests that are abandoned in draft may be closed due to inactivity. |
b26cb51 to
59c5d26
Compare
Signed-off-by: Kevin Ullyott <ullyott.kevin@gmail.com>
Signed-off-by: Kevin Ullyott <ullyott.kevin@gmail.com>
Signed-off-by: Kevin Ullyott <ullyott.kevin@gmail.com>
59c5d26 to
bd17a8e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Right now,
queue:clearpurges SQS but leaves the offloaded payloads sitting in the cache store. Even if a TTL got added later, when someone clears the queue, they may want the storage reclaimed now rather than waiting for expiration (matters especially on S3, where leftover objects keep costing money or persistence of payloads with sensitive data could be an issue).Adds an opt-in
flush_on_clearflag under theoverflowconfig. When on,SqsQueue::clear()callsflush()on the configured cache store after purging SQS. Defaults to off.Worth flagging: for most cache stores
flush()wipes the whole store, not just the overflow entries. I considered some options to narrow its scope. Like tags or maybe a manifest file/entry pattern. Tags didn't seem right (though maybe that is still an option), and a manifest adds per-push overhead and concurrency issues that felt heavier than the feature needs. So this is opt-in and separate fromdelete_after_processing, with the recommendation to pointoverflow.storeat a dedicated cache store (the newstoragedriver with its own disk and/or path is a clean fit).Tests cover flush runs when enabled, don't run when disabled or when overflow is off, and forward the configured store name.
Has any user-facing documentation been written for the overflow feature yet? Happy to put together a docs PR if not.