Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

archive/zip: reading many small files allocates a lot of memory #14289

Closed
dziemba opened this issue Feb 10, 2016 · 3 comments
Closed

archive/zip: reading many small files allocates a lot of memory #14289

dziemba opened this issue Feb 10, 2016 · 3 comments
Milestone

Comments

@dziemba
Copy link
Contributor

@dziemba dziemba commented Feb 10, 2016

archive/zip creates a new compress/flate.Reader for each file read from the archive. If there are many small files in an archive, this allocates a lot of memory.

compress/flate allows re-using of Readers, see https://codereview.appspot.com/97140043/

Using this in archive/zip reduces the amount of memory allocated/GCed. In a real-world scenario when reading ~ 500 small files from a ZIP archive this gives a speedup of 1.5x-2x.

Version: go1.6rc2

@ianlancetaylor ianlancetaylor added this to the Go1.7 milestone Feb 10, 2016
@dziemba
Copy link
Contributor Author

@dziemba dziemba commented Feb 10, 2016

This should fix it: https://go-review.googlesource.com/#/c/19416/

But I'm not 100% sure if the implementaion is correct. Please review with care ;)

@gopherbot
Copy link

@gopherbot gopherbot commented Feb 18, 2016

CL https://golang.org/cl/19416 mentions this issue.

@dziemba
Copy link
Contributor Author

@dziemba dziemba commented May 4, 2016

Can we merge this soon so it will be included in Go 1.7?

@gopherbot gopherbot closed this in e155bf0 May 4, 2016
@golang golang locked and limited conversation to collaborators May 4, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.