Restructure for better large bucket support (Fixes #138, #128. #100, #80, #54, and #14) #249
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
eherot
force-pushed
the
improve-large-bucket-performance
branch
2 times, most recently
from
8822db0
to
ce1edd1
Compare
* Use a synchronous queue to handle the processing of events in parallel * Switch to aws-sdk v3 * Add a lot of logging * Enable the use of the SDK's start_after parameter call to fetch only new events (useful in cases where objects are stored in alphabetical order by time, such as S3 access logs) * Limit the batch size of the S3 request
eherot
force-pushed
the
improve-large-bucket-performance
branch
from
ce1edd1
to
c32c13f
Compare
eherot
force-pushed
the
improve-large-bucket-performance
branch
from
3bcd04f
to
a38b42a
Compare
eherot
force-pushed
the
improve-large-bucket-performance
branch
15 times, most recently
from
e83ec3d
to
bbbf9c1
Compare
eherot
force-pushed
the
improve-large-bucket-performance
branch
from
119d4a6
to
9d85332
Compare
This is critical for unit testing since RSpec does not tolerate background threads
eherot
force-pushed
the
improve-large-bucket-performance
branch
from
fc28c9b
to
209e919
Compare
eherot
force-pushed
the
improve-large-bucket-performance
branch
from
209e919
to
68081ff
Compare
It's going to be overwritten anyway so no sense wasting the time, especially if the goal is to clear out an excessively large db
eherot
force-pushed
the
improve-large-bucket-performance
branch
from
68081ff
to
05b4a65
Compare
…_size) with use_start_after
eherot
force-pushed
the
improve-large-bucket-performance
branch
from
beaa532
to
2aae3bc
Compare
|
Hi @eherot The Logstash S3 input is now been shipped as part of https://github.com/logstash-plugins/logstash-integration-aws , rather than the individual plugin. Not sure if you want to re-target your PR against it? |
|
Thanks for the heads up. Let me see how much work that's going to be... |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
(This is a reworking of the now defunct #84)
What this changes
May/Should fix a number of open issues either directly or by virtue of replacing the problem code entirely
Thanks for contributing to Logstash! If you haven't already signed our CLA, here's a handy link: https://www.elastic.co/contributor-agreement/