net/http: reuse gzip reader from pool for transport

[Sunyue]

go/src/net/http/transport.go

Line 2858 in 089e37a

gz.zr, gz.zerr = gzip.NewReader(gz.body)

Is it possible to reuse this gzip reader from a sync.Pool to reduce the memory allocation cause by flate.NewReader

[bcmills]

gzip.Reader does have a Reset method, but we would also need some way to know when the gzip.Reader is safe to reuse.

Hmm. (*http.gzipReader).Read already has some locking in Read. Perhaps we could do something like:

func (gz *gzipReader) Read(p []byte) (n int, err error) {
	gz.body.mu.Lock()
	if gz.body.closed {
		err = errReadOnClosedResBody
	} else if gz.zr == nil {
		if gz.zerr == nil {
			zr := gzipPool.Get().(*gzip.Reader)
			gz.zerr = zr.Reset(gz.body)
			if gz.zerr == nil {
				gz.zr = zr
			} else {
				gzipPool.Put(zr)
			}
		}
		err = gz.zerr
	}
	gz.body.mu.Unlock()

	if err != nil {
		return 0, err
	}
	return gz.zr.Read(p)
}

func (gz *gzipReader) Close() error {
	err := gz.body.Close()

	gz.body.mu.Lock()
	if gz.zr != nil {
		gz.zr.Reset(emptyReader{})  // drop reference to gz.body
		gzipPool.Put(gz.zr)
		gz.zr = nil
	}
	gz.body.mu.Unlock()

	return err
}

var (
	gzipPool = sync.Pool{
		New: func() any { return new(gzip.Reader) },
	}
	_ flate.Reader = emptyReader{}
)

type emptyReader struct{}

func (emptyReader) Read([]byte) (int, error) { return 0, io.EOF }
func (emptyReader) ReadByte() (byte, error) { return 0, io.EOF }

Of course, such a change would also need a supporting benchmark. @Sunyue, want to send it for Go 1.22?

(CC @neild)

[neild]

Seems like something we could do; I'm happy to review a CL if someone wants to send me one.

[gopherbot]

Change https://go.dev/cl/510115 mentions this issue: net/http: pool transport gzip readers

[gopherbot]

Change https://go.dev/cl/510255 mentions this issue: net/http: pool transport gzip readers

[AlexanderYastrebov]

Created #61390