Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Go] IPC LZ4 Decompressor does not return bytes to pool on Close #38728

Closed
asubiotto opened this issue Nov 15, 2023 · 0 comments · Fixed by #38729
Closed

[Go] IPC LZ4 Decompressor does not return bytes to pool on Close #38728

asubiotto opened this issue Nov 15, 2023 · 0 comments · Fixed by #38729
Assignees
@asubiotto
Labels
Component: Go Type: bug
Milestone
15.0.0

Comments

@asubiotto
Copy link
Contributor

Describe the bug, including details regarding any error messages, version, and platform.

Based on a memory profile of our application, there seem to be higher allocations than expected given a sync.Pool is used for LZ4 IPC decompression buffers. Here is a link to the profile: https://pprof.me/d5122d09b9dae81844c3068f75e31b76/?profileType=profile%3Aalloc_space%3Abytes%3Aspace%3Abytes

Component(s)

Go
@asubiotto asubiotto mentioned this issue Nov 15, 2023
Merged
zeroshade pushed a commit that referenced this issue Nov 27, 2023
@asubiotto
GH-38728: [Go] ipc: put lz4 decompression buffers back into sync.Pool (…
b0e1f74 
…#38729)

The lz4 decompressor was not calling Reset on the underlying writer in its Close method. This could cause buffers not to be released back to the pool and defeating the purpose of the sync.Pool in the lz4 package.

Additionally, a call to Close was missing in readDictionary.

Closes #38728 
* Closes: #38728

Authored-by: Alfonso Subiotto Marques <alfonso.subiotto@polarsignals.com>
Signed-off-by: Matt Topol <zotthewizard@gmail.com>
@zeroshade zeroshade added this to the 15.0.0 milestone Nov 27, 2023
asubiotto added a commit to asubiotto/arrow that referenced this issue Nov 29, 2023
@asubiotto
apacheGH-38728: [Go] ipc: put lz4 decompression buffers back into syn…
e312603 
…c.Pool (apache#38729)

The lz4 decompressor was not calling Reset on the underlying writer in its Close method. This could cause buffers not to be released back to the pool and defeating the purpose of the sync.Pool in the lz4 package.

Additionally, a call to Close was missing in readDictionary.

Closes apache#38728 
* Closes: apache#38728

Authored-by: Alfonso Subiotto Marques <alfonso.subiotto@polarsignals.com>
Signed-off-by: Matt Topol <zotthewizard@gmail.com>
dgreiss pushed a commit to dgreiss/arrow that referenced this issue Feb 19, 2024
@asubiotto @dgreiss
apacheGH-38728: [Go] ipc: put lz4 decompression buffers back into syn…
9594bf4 
…c.Pool (apache#38729)

The lz4 decompressor was not calling Reset on the underlying writer in its Close method. This could cause buffers not to be released back to the pool and defeating the purpose of the sync.Pool in the lz4 package.

Additionally, a call to Close was missing in readDictionary.

Closes apache#38728 
* Closes: apache#38728

Authored-by: Alfonso Subiotto Marques <alfonso.subiotto@polarsignals.com>
Signed-off-by: Matt Topol <zotthewizard@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@asubiotto asubiotto

Labels
Component: Go Type: bug
Projects
None yet
Milestone
15.0.0
Development

Successfully merging a pull request may close this issue.

GH-38728: [Go] ipc: put lz4 decompression buffers back into sync.Pool asubiotto/arrow
2 participants
@zeroshade @asubiotto