Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[v22.3.x] Backport #9598 (Compression fixes) #9722

Merged
merged 6 commits into from
Apr 13, 2023
Merged

[v22.3.x] Backport #9598 (Compression fixes) #9722

merged 6 commits into from
Apr 13, 2023

Conversation

jcsp
Copy link
Contributor

@jcsp jcsp commented Mar 29, 2023 •
edited
Loading

Backport #9598
Backport #9563

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v23.1.x
  • v22.3.x
  • v22.2.x

Release Notes

Bug Fixes

  • Fix cases where sending very large batches to compacted topics using LZ4 or gzip compression could result in bad_alloc errors.

jcsp added 6 commits March 29, 2023 20:35
@jcsp
compression: use fragmented allocation in LZ4 decompress
76dbd27 
Fixes redpanda-data#9521

(cherry picked from commit 214a2dc)
@jcsp
compression/tests: improve edge case coverage
9616435 
Test either side of allocation chunk boundary.

(cherry picked from commit 423199e)
@jcsp
compression: fix allocation in gzip compressor
b07b17a 
...and as a bonus, avoid fully reading buffer twice
when decompressing (previously this was done to
calculate size of monolithic output buffer)

(cherry picked from commit 14562d7)
@jcsp
compression: fix LZ4 allocation
d1111a5 
* Avoid monolithic allocation in compression
* Avoid linearizing input buffer in decompression

(cherry picked from commit b847f0c)
@jcsp
compression/tests: extend microbenchmark to lz4 & gzip
0722086 
(cherry picked from commit 01b9962)
@jcsp
compression: skip empty chunks in inputs
0c2d2e3 
The previous code worked in practice, but in the case
of gzip at least, the docs claim that it is allowed
to return an error if given an empty buffer, so let's
not do that.

(cherry picked from commit 3a2ee83)
@jcsp jcsp marked this pull request as ready for review March 30, 2023 07:58
@jcsp jcsp merged commit a7fb3da into redpanda-data:v22.3.x Apr 13, 2023
@jcsp jcsp deleted the v22.3.x-backport-9598 branch April 13, 2023 14:09
@BenPope BenPope added this to the v22.3.16 milestone Apr 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
area/redpanda
Projects
None yet
Milestone
v22.3.16
Development

Successfully merging this pull request may close these issues.
2 participants
@jcsp @BenPope