Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(aws service): Allow zero size compressed messages #8069

Merged
merged 7 commits into from Jul 2, 2021

Conversation

ktff
Copy link
Contributor

@ktff ktff commented Jun 29, 2021

Closes #7710

Enable for individually compressed messages to be empty for sources:

  • aws_kinesis_firehose
  • aws_s3

Other sources either don't have sub messages or they can't be individually compressed.

ktff added 3 commits June 29, 2021 13:25
Signed-off-by: ktf <krunotf@gmail.com>
Signed-off-by: ktf <krunotf@gmail.com>
Signed-off-by: ktf <krunotf@gmail.com>
@ktff ktff added source: aws_kinesis_firehose Anything `aws_kinesis_firehose` source related source: aws_s3 Anything `aws_s3` source related labels Jun 29, 2021
@ktff ktff self-assigned this Jun 29, 2021
ktff added 2 commits June 29, 2021 14:45
Signed-off-by: ktf <krunotf@gmail.com>
Signed-off-by: ktf <krunotf@gmail.com>
@ktff ktff marked this pull request as ready for review June 30, 2021 16:30
@ktff ktff requested a review from a team June 30, 2021 16:30
Copy link
Member

@jszwedko jszwedko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thanks! @ktff

src/sources/aws_kinesis_firehose/mod.rs Show resolved Hide resolved
Copy link
Member

@bruceg bruceg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks reasonable, though peekable might simplify one part.

src/sources/aws_s3/mod.rs Show resolved Hide resolved
ktff added 2 commits July 2, 2021 12:35
Signed-off-by: ktf <krunotf@gmail.com>
Signed-off-by: ktf <krunotf@gmail.com>
@ktff ktff merged commit 0d39819 into master Jul 2, 2021
@ktff ktff deleted the ktff/zero_byte_deco branch July 2, 2021 17:18
001wwang pushed a commit to 001wwang/vector that referenced this pull request Jul 7, 2021
)

* AWS S3

Signed-off-by: ktf <krunotf@gmail.com>

* aws kinesis firehose

Signed-off-by: ktf <krunotf@gmail.com>

* Empty message

Signed-off-by: ktf <krunotf@gmail.com>

* Use vec

Signed-off-by: ktf <krunotf@gmail.com>

* Correct tests

Signed-off-by: ktf <krunotf@gmail.com>

* Bump

Signed-off-by: ktf <krunotf@gmail.com>

* Bump

Signed-off-by: ktf <krunotf@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
source: aws_kinesis_firehose Anything `aws_kinesis_firehose` source related source: aws_s3 Anything `aws_s3` source related
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Avoid trying to decompress 0-byte files
3 participants