Improve lz4 compression

[cyriltovena]

• Move to v4.
• Remove not required checksumming. (we already checksum in the chunk)
• Default will now write 4M blocks, it's backward compatible for reads.

Performance wise I can see a 20% improvement compare to snappy (for reads, write are slower) but also a better compression ratio. At the expense of more cpu and memory used.

see benchmark and compression test result below.

bench:

go test -benchmem -run=BenchmarkRead -bench BenchmarkRead -v ./pkg/chunkenc
goos: darwin
goarch: amd64
pkg: github.com/grafana/loki/pkg/chunkenc
BenchmarkRead
BenchmarkRead/none
    memchunk_test.go:605: bytes per second  691 MB
    memchunk_test.go:606: n= 1
    memchunk_test.go:605: bytes per second  716 MB
    memchunk_test.go:606: n= 100
BenchmarkRead/none-4         	     100	  10566322 ns/op	 7761962 B/op	   28476 allocs/op
BenchmarkRead/gzip
    memchunk_test.go:605: bytes per second  370 MB
    memchunk_test.go:606: n= 1
    memchunk_test.go:605: bytes per second  322 MB
    memchunk_test.go:606: n= 3
BenchmarkRead/gzip-4         	       3	 341573597 ns/op	112990232 B/op	  413237 allocs/op
BenchmarkRead/lz4-64k
    memchunk_test.go:605: bytes per second  1.1 GB
    memchunk_test.go:606: n= 1
    memchunk_test.go:605: bytes per second  792 MB
    memchunk_test.go:606: n= 15
BenchmarkRead/lz4-64k-4      	      15	  99341841 ns/op	80775355 B/op	  296551 allocs/op
BenchmarkRead/lz4-256k
    memchunk_test.go:605: bytes per second  593 MB
    memchunk_test.go:606: n= 1
    memchunk_test.go:605: bytes per second  821 MB
    memchunk_test.go:606: n= 8
    memchunk_test.go:605: bytes per second  722 MB
    memchunk_test.go:606: n= 10
BenchmarkRead/lz4-256k-4     	      10	 116166896 ns/op	86147066 B/op	  316208 allocs/op
BenchmarkRead/lz4-1M
    memchunk_test.go:605: bytes per second  856 MB
    memchunk_test.go:606: n= 1
    memchunk_test.go:605: bytes per second  682 MB
    memchunk_test.go:606: n= 12
BenchmarkRead/lz4-1M-4       	      12	 125007983 ns/op	87994374 B/op	  321232 allocs/op
BenchmarkRead/lz4
    memchunk_test.go:605: bytes per second  758 MB
    memchunk_test.go:606: n= 1
    memchunk_test.go:605: bytes per second  799 MB
    memchunk_test.go:606: n= 9
    memchunk_test.go:605: bytes per second  951 MB
    memchunk_test.go:606: n= 10
    memchunk_test.go:605: bytes per second  825 MB
    memchunk_test.go:606: n= 13
BenchmarkRead/lz4-4          	      13	 103320290 ns/op	89072964 B/op	  321186 allocs/op
BenchmarkRead/snappy
    memchunk_test.go:605: bytes per second  785 MB
    memchunk_test.go:606: n= 1
    memchunk_test.go:605: bytes per second  621 MB
    memchunk_test.go:606: n= 15
BenchmarkRead/snappy-4       	      15	  97019479 ns/op	61859299 B/op	  226125 allocs/op
PASS
ok  	github.com/grafana/loki/pkg/chunkenc	17.788s

compression test:

=== RUN   TestChunkSize
=== RUN   TestChunkSize/none
    memchunk_test.go:435: Chunk size 1.5 MB
    memchunk_test.go:436: characters  1516641
=== RUN   TestChunkSize/gzip
    memchunk_test.go:435: Chunk size 1.3 MB
    memchunk_test.go:436: characters  22029822
=== RUN   TestChunkSize/lz4-64k
    memchunk_test.go:435: Chunk size 1.3 MB
    memchunk_test.go:436: characters  15734244
=== RUN   TestChunkSize/lz4-256k
    memchunk_test.go:435: Chunk size 1.3 MB
    memchunk_test.go:436: characters  16774337
=== RUN   TestChunkSize/lz4-1M
    memchunk_test.go:435: Chunk size 1.3 MB
    memchunk_test.go:436: characters  17039875
=== RUN   TestChunkSize/lz4
    memchunk_test.go:435: Chunk size 1.3 MB
    memchunk_test.go:436: characters  17039875
=== RUN   TestChunkSize/snappy
    memchunk_test.go:435: Chunk size 1.3 MB
    memchunk_test.go:436: characters  12059524
--- PASS: TestChunkSize (0.48s)
    --- PASS: TestChunkSize/none (0.00s)
    --- PASS: TestChunkSize/gzip (0.19s)
    --- PASS: TestChunkSize/lz4-64k (0.03s)
    --- PASS: TestChunkSize/lz4-256k (0.04s)
    --- PASS: TestChunkSize/lz4-1M (0.17s)
    --- PASS: TestChunkSize/lz4 (0.03s)
    --- PASS: TestChunkSize/snappy (0.01s)
PASS
ok  	github.com/grafana/loki/pkg/chunkenc	1.064s

[codecov-commenter]

Codecov Report

Merging #2614 into master will decrease coverage by 0.00%.
The diff coverage is 83.33%.

@@            Coverage Diff             @@
##           master    #2614      +/-   ##
==========================================
- Coverage   62.87%   62.86%   -0.01%     
==========================================
  Files         170      170              
  Lines       15049    15045       -4     
==========================================
- Hits         9462     9458       -4     
  Misses       4826     4826              
  Partials      761      761              

Impacted Files	Coverage Δ
pkg/chunkenc/pool.go	87.27% <81.81%> (-2.21%)	⬇️
pkg/chunkenc/interface.go	87.50% <100.00%> (ø)
pkg/logql/evaluator.go	92.88% <0.00%> (+0.40%)	⬆️

[owen-d]

LGTM, sorry for the delay. Nice work.