Compression improvement. #689
Labels
component/loki
stale
A stale issue or PR that will automatically be closed.
type/feature
Something new we should do
When looking at memory comsumption we can see that a big part of it is due to the compression.
1 - we should look into pooling
CompressionWriter
creation. (https://github.com/grafana/loki/blob/master/pkg/chunkenc/gzip.go#L149) Currently one is created for every chunk, gzip is known to be a huge memory eater see and golang/go#32371 golang/go#186252 - introduce benchmark with more compression options to have different tradeoff.
@gouthamve started the work here https://github.com/gouthamve/logish-explore/tree/master/chunkenc we should bring it to Loki.
The text was updated successfully, but these errors were encountered: