New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

compress/flate: add Reset method to let callers avoid garbage #6138

Closed
bradfitz opened this Issue Aug 13, 2013 · 9 comments

Comments

Projects
None yet
5 participants
@bradfitz
Member

bradfitz commented Aug 13, 2013

Related to issue #6136, flate.NewWriter generates a lot of garbage which can't be
controlled by the calling application.

Options:

-- internal memory reuse in the flate package
-- Reset method (like bufio.Reader and bufio.Writer)
-- let buffers be passed in

Attachments:

  1. k0EYnXJ_LT.0.svg (15027 bytes)
@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Aug 13, 2013

Member

Comment 1:

Each call to flate.NewWriter allocates 1.4 MB of memory, and Close doesn't reuse any of
it.
Member

bradfitz commented Aug 13, 2013

Comment 1:

Each call to flate.NewWriter allocates 1.4 MB of memory, and Close doesn't reuse any of
it.
@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Aug 13, 2013

Contributor

Comment 2:

Reset method please.
(Rationale: Having the reset method means that the buffer is attached to the Writer, not
magically moving around, so it gives the caller more control and also keeps the reuse
local to a specific package.)
Contributor

rsc commented Aug 13, 2013

Comment 2:

Reset method please.
(Rationale: Having the reset method means that the buffer is attached to the Writer, not
magically moving around, so it gives the caller more control and also keeps the reuse
local to a specific package.)
@rsc

This comment has been minimized.

Show comment
Hide comment
@rsc

rsc Aug 13, 2013

Contributor

Comment 3:

I wish it did generate a "log of garbage". That would make things easier.
Contributor

rsc commented Aug 13, 2013

Comment 3:

I wish it did generate a "log of garbage". That would make things easier.
@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Aug 13, 2013

Member

Comment 4:

This issue was updated by revision c7d352c.

TestOver65kFiles spends all its time garbage collecting.
Removing the 1.4 MB of allocations per each of the 65k
files brings this from 34 seconds to 0.23 seconds.
R=golang-dev, rsc
CC=golang-dev
https://golang.org/cl/12894043
Member

bradfitz commented Aug 13, 2013

Comment 4:

This issue was updated by revision c7d352c.

TestOver65kFiles spends all its time garbage collecting.
Removing the 1.4 MB of allocations per each of the 65k
files brings this from 34 seconds to 0.23 seconds.
R=golang-dev, rsc
CC=golang-dev
https://golang.org/cl/12894043
@robpike

This comment has been minimized.

Show comment
Hide comment
@robpike

robpike Aug 16, 2013

Contributor

Comment 5:

Looks fixed to me. Reopen if you disagree.

Status changed to Fixed.

Contributor

robpike commented Aug 16, 2013

Comment 5:

Looks fixed to me. Reopen if you disagree.

Status changed to Fixed.

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Aug 16, 2013

Member

Comment 6:

Per comment #2, this bug focused to be about adding a Reset method on flate.Writer so
callers can avoid 1.4 MB of garbage per writer if it affects them.

Owner changed to @bradfitz.

Status changed to Accepted.

Member

bradfitz commented Aug 16, 2013

Comment 6:

Per comment #2, this bug focused to be about adding a Reset method on flate.Writer so
callers can avoid 1.4 MB of garbage per writer if it affects them.

Owner changed to @bradfitz.

Status changed to Accepted.

@remyoudompheng

This comment has been minimized.

Show comment
Hide comment
@remyoudompheng

remyoudompheng Aug 24, 2013

Contributor

Comment 7:

Should we create separate issues for compress/zlib and compress/gzip ? Most people never
use compress/flate directly, and the most common demand is probably Resetting a
gzip.Writer.
Contributor

remyoudompheng commented Aug 24, 2013

Comment 7:

Should we create separate issues for compress/zlib and compress/gzip ? Most people never
use compress/flate directly, and the most common demand is probably Resetting a
gzip.Writer.
@remyoudompheng

This comment has been minimized.

Show comment
Hide comment
@remyoudompheng

remyoudompheng Aug 29, 2013

Contributor

Comment 8:

This issue was closed by revision f5f0e40.

Status changed to Fixed.

Contributor

remyoudompheng commented Aug 29, 2013

Comment 8:

This issue was closed by revision f5f0e40.

Status changed to Fixed.

@bradfitz

This comment has been minimized.

Show comment
Hide comment
@bradfitz

bradfitz Aug 30, 2013

Member

Comment 9:

This issue was updated by revision db12f9d.

R=golang-dev, iant
CC=golang-dev
https://golang.org/cl/13435043
Member

bradfitz commented Aug 30, 2013

Comment 9:

This issue was updated by revision db12f9d.

R=golang-dev, iant
CC=golang-dev
https://golang.org/cl/13435043

@bradfitz bradfitz self-assigned this Aug 30, 2013

@rsc rsc added this to the Go1.2 milestone Apr 14, 2015

@rsc rsc removed the go1.2maybe label Apr 14, 2015

@golang golang locked and limited conversation to collaborators Jun 24, 2016

This issue was closed.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.