zstd.Encoder consumes at least 8 MB of memory

[marten-seemann]

First of all, thank you for this amazing library! I'm using zstd to compress QUIC log files in libp2p. Switching from gzip to zstd allowed me to speed up the compression by roughly 6x (at comparable file sizes).

I'm running into problems regarding the memory consumption of the zstd implementation though. A node can easily have hundreds of active connections, each of them logging through their own zstd.Encoder.
As far as I can tell, the reason for this is that a zstd.Encoder is initialized with a massive buffer of 8 MB in

compress/zstd/enc_base.go

Lines 131 to 138 in b5cd744

	if (!singleBlock || d.DictContentSize() > 0) && cap(e.hist) < int(e.maxMatchOff*2)+d.DictContentSize() {
	l := e.maxMatchOff*2 + int32(d.DictContentSize())
	// Make it at least 1MB.
	if l < 1<<20 {
	l = 1 << 20
	}
	e.hist = make([]byte, 0, l)
	}

This is because e.maxMatchOff is 4 MB.

I have to admit that I don't understand a lot of the internals of zstd, so I'm not sure if this is something that can be optimized. Ideally, a zstd.Encoder would consume << 1 MB on initialization, and only allocate more memory if and when it's actually needed.

[klauspost]

@marten-seemann You can use the WithWindowSize to adjust the window size of the encoder.