s2: Add LZ4 block converter (#748) · klauspost/compress@7e21226

Commit

s2: Add LZ4 block converter (#748)

This allows converting compressed LZ4 blocks to S2 (or snappy) blocks without decompression.

LZ4 -> S2 seems to be same size on average.
LZ4 -> Snappy is usually worse.

## Single threaded performance

Speed (excluding LZ4 encoding):

```
BenchmarkLZ4Converter_ConvertBlock/html-32         	   28237	     42827 ns/op	2390.99 MB/s	       559.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/urls-32         	    2138	    541816 ns/op	1295.80 MB/s	     -3943 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/jpg-32          	  514826	      2328 ns/op	52874.24 MB/s	       482.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/jpg_200b-32     	34821668	        33.48 ns/op	5973.00 MB/s	         2.000 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/pdf-32          	  198241	      5975 ns/op	17136.81 MB/s	       136.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/html4-32        	    7002	    173440 ns/op	2361.63 MB/s	      1840 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1-32         	    5940	    196951 ns/op	 772.22 MB/s	       106.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt2-32         	    6656	    177228 ns/op	 706.32 MB/s	     -1427 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt3-32         	    2355	    510435 ns/op	 836.06 MB/s	       384.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt4-32         	    1700	    694444 ns/op	 693.88 MB/s	     -9125 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/pb-32           	   37118	     32141 ns/op	3689.60 MB/s	         1.000 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/gaviota-32      	    6961	    172253 ns/op	1070.05 MB/s	      9303 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1_128b-32    	19923691	        59.82 ns/op	2139.66 MB/s	         0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1_1000b-32   	 3180837	       375.2 ns/op	2665.40 MB/s	        16.00 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1_10000b-32  	  184214	      6350 ns/op	1574.70 MB/s	        90.00 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1_20000b-32  	   74031	     15521 ns/op	1288.54 MB/s	        -5.000 b_saved	       0 B/op	       0 allocs/op
```

Assembly speed (amd64)
```
BenchmarkLZ4Converter_ConvertBlock/html-32         	   47457	     24463 ns/op	4185.89 MB/s	       559.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/urls-32         	    3506	    330277 ns/op	2125.75 MB/s	     -3943 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/jpg-32          	  450177	      2718 ns/op	45294.89 MB/s	       482.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/jpg_200b-32     	76887589	        15.52 ns/op	12887.16 MB/s	         2.000 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/pdf-32          	  279540	      4322 ns/op	23694.21 MB/s	       136.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/html4-32        	   10000	    107485 ns/op	3810.75 MB/s	      1840 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1-32         	   10000	    117764 ns/op	1291.47 MB/s	       106.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt2-32         	   10000	    100578 ns/op	1244.60 MB/s	     -1427 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt3-32         	    3793	    313021 ns/op	1363.34 MB/s	       384.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt4-32         	    2988	    399888 ns/op	1204.99 MB/s	     -9125 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/pb-32           	   57486	     19277 ns/op	6151.76 MB/s	         1.000 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/gaviota-32      	   10000	    115641 ns/op	1593.90 MB/s	      9303 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1_128b-32    	38400122	        31.21 ns/op	4101.10 MB/s	         0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1_1000b-32   	 6509028	       179.9 ns/op	5559.38 MB/s	        16.00 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1_10000b-32  	  368212	      3244 ns/op	3082.28 MB/s	        90.00 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1_20000b-32  	  141013	      8303 ns/op	2408.69 MB/s	        -5.000 b_saved	       0 B/op	       0 allocs/op
```


Reference compression speed:

```
BenchmarkCompressBlockReference/html/default-32                    14070         82449 ns/op    1241.98 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/urls/default-32                     1215        970890 ns/op     723.14 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/jpg/default-32                    193770          5904 ns/op    20849.30 MB/s          0 B/op          0 allocs/op
BenchmarkCompressBlockReference/jpg_200b/default-32              8297767           144.1 ns/op  1387.77 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/pdf/default-32                     94203         12694 ns/op    8066.76 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/html4/default-32                   12174         97969 ns/op    4180.90 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt1/default-32                     3613        333851 ns/op     455.56 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt2/default-32                     4683        260579 ns/op     480.39 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt3/default-32                     1268        947209 ns/op     450.54 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt4/default-32                     1083       1097426 ns/op     439.08 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/pb/default-32                      18357         64771 ns/op    1830.90 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/gaviota/default-32                  3942        295275 ns/op     624.23 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt1_128b/default-32            11448295           105.7 ns/op  1210.45 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt1_1000b/default-32            1000000          1021 ns/op     979.26 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt1_10000b/default-32            116739         10114 ns/op     988.68 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt1_20000b/default-32             49216         23409 ns/op     854.39 MB/s           0 B/op          0 allocs/op

BenchmarkCompressBlockReference/html/better-32                      6649        174667 ns/op     586.26 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/urls/better-32                       627       1905706 ns/op     368.41 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/jpg/better-32                      52425         22783 ns/op    5402.88 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/jpg_200b/better-32               2772865           433.3 ns/op   461.61 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/pdf/better-32                       9210        127051 ns/op     805.97 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/html4/better-32                     5835        201146 ns/op    2036.33 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt1/better-32                      2034        566702 ns/op     268.38 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt2/better-32                      2386        500580 ns/op     250.07 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt3/better-32                       758       1556541 ns/op     274.17 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt4/better-32                       591       2013515 ns/op     239.31 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/pb/better-32                        7836        155117 ns/op     764.51 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/gaviota/better-32                   2473        484975 ns/op     380.06 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt1_128b/better-32              4322678           275.5 ns/op   464.59 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt1_1000b/better-32              468687          2533 ns/op     394.76 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt1_10000b/better-32              49606         23720 ns/op     421.59 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt1_20000b/better-32              14823         81300 ns/op     246.00 MB/s           0 B/op          0 allocs/op


```

Size comparisons (using Go lz4 encoder):

```
=== RUN   TestLZ4Converter_ConvertBlock/html
    lz4convert_test.go:42: input size: 102400
    lz4convert_test.go:43: lz4 size: 21195
    lz4convert_test.go:60: lz4->snappy size: 21828
    lz4convert_test.go:79: lz4->s2 size: 20636
    lz4convert_test.go:91: s2 (default) size: 20865
    lz4convert_test.go:95: s2 (better) size: 18969
    lz4convert_test.go:97: lz4 -> s2 bytes saved: 559
    lz4convert_test.go:98: lz4 -> snappy bytes saved: -633
    lz4convert_test.go:99: data -> s2 (default) bytes saved: 330
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 2226
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: -229
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 1667
    --- PASS: TestLZ4Converter_ConvertBlock/html (0.00s)
=== RUN   TestLZ4Converter_ConvertBlock/urls
    lz4convert_test.go:42: input size: 702087
    lz4convert_test.go:43: lz4 size: 292514
    lz4convert_test.go:60: lz4->snappy size: 297926
    lz4convert_test.go:79: lz4->s2 size: 296457
    lz4convert_test.go:91: s2 (default) size: 286538
    lz4convert_test.go:95: s2 (better) size: 248076
    lz4convert_test.go:97: lz4 -> s2 bytes saved: -3943
    lz4convert_test.go:98: lz4 -> snappy bytes saved: -5412
    lz4convert_test.go:99: data -> s2 (default) bytes saved: 5976
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 44438
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: 9919
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 48381
    --- PASS: TestLZ4Converter_ConvertBlock/urls (0.01s)
=== RUN   TestLZ4Converter_ConvertBlock/jpg
    lz4convert_test.go:42: input size: 123093
    lz4convert_test.go:43: lz4 size: 123522
    lz4convert_test.go:60: lz4->snappy size: 123040
    lz4convert_test.go:79: lz4->s2 size: 123040
    lz4convert_test.go:91: s2 (default) size: 123097
    lz4convert_test.go:95: s2 (better) size: 123097
    lz4convert_test.go:97: lz4 -> s2 bytes saved: 482
    lz4convert_test.go:98: lz4 -> snappy bytes saved: 482
    lz4convert_test.go:99: data -> s2 (default) bytes saved: 425
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 425
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: -57
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: -57
    --- PASS: TestLZ4Converter_ConvertBlock/jpg (0.00s)
=== RUN   TestLZ4Converter_ConvertBlock/jpg_200b
    lz4convert_test.go:42: input size: 200
    lz4convert_test.go:43: lz4 size: 155
    lz4convert_test.go:60: lz4->snappy size: 153
    lz4convert_test.go:79: lz4->s2 size: 153
    lz4convert_test.go:91: s2 (default) size: 153
    lz4convert_test.go:95: s2 (better) size: 147
    lz4convert_test.go:97: lz4 -> s2 bytes saved: 2
    lz4convert_test.go:98: lz4 -> snappy bytes saved: 2
    lz4convert_test.go:99: data -> s2 (default) bytes saved: 2
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 8
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: 0
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 6
    --- PASS: TestLZ4Converter_ConvertBlock/jpg_200b (0.00s)
=== RUN   TestLZ4Converter_ConvertBlock/pdf
    lz4convert_test.go:42: input size: 102400
    lz4convert_test.go:43: lz4 size: 83152
    lz4convert_test.go:60: lz4->snappy size: 83428
    lz4convert_test.go:79: lz4->s2 size: 83016
    lz4convert_test.go:91: s2 (default) size: 84199
    lz4convert_test.go:95: s2 (better) size: 82884
    lz4convert_test.go:97: lz4 -> s2 bytes saved: 136
    lz4convert_test.go:98: lz4 -> snappy bytes saved: -276
    lz4convert_test.go:99: data -> s2 (default) bytes saved: -1047
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 268
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: -1183
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 132
    --- PASS: TestLZ4Converter_ConvertBlock/pdf (0.00s)
=== RUN   TestLZ4Converter_ConvertBlock/html4
    lz4convert_test.go:42: input size: 409600
    lz4convert_test.go:43: lz4 size: 81908
    lz4convert_test.go:60: lz4->snappy size: 84886
    lz4convert_test.go:79: lz4->s2 size: 80068
    lz4convert_test.go:91: s2 (default) size: 20867
    lz4convert_test.go:95: s2 (better) size: 18979
    lz4convert_test.go:97: lz4 -> s2 bytes saved: 1840
    lz4convert_test.go:98: lz4 -> snappy bytes saved: -2978
    lz4convert_test.go:99: data -> s2 (default) bytes saved: 61041
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 62929
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: 59201
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 61089
    --- PASS: TestLZ4Converter_ConvertBlock/html4 (0.00s)
=== RUN   TestLZ4Converter_ConvertBlock/txt1
    lz4convert_test.go:42: input size: 152089
    lz4convert_test.go:43: lz4 size: 79672
    lz4convert_test.go:60: lz4->snappy size: 79567
    lz4convert_test.go:79: lz4->s2 size: 79566
    lz4convert_test.go:91: s2 (default) size: 85931
    lz4convert_test.go:95: s2 (better) size: 71608
    lz4convert_test.go:97: lz4 -> s2 bytes saved: 106
    lz4convert_test.go:98: lz4 -> snappy bytes saved: 105
    lz4convert_test.go:99: data -> s2 (default) bytes saved: -6259
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 8064
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: -6365
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 7958
    --- PASS: TestLZ4Converter_ConvertBlock/txt1 (0.00s)
=== RUN   TestLZ4Converter_ConvertBlock/txt2
    lz4convert_test.go:42: input size: 125179
    lz4convert_test.go:43: lz4 size: 70801
    lz4convert_test.go:60: lz4->snappy size: 72231
    lz4convert_test.go:79: lz4->s2 size: 72228
    lz4convert_test.go:91: s2 (default) size: 79572
    lz4convert_test.go:95: s2 (better) size: 65938
    lz4convert_test.go:97: lz4 -> s2 bytes saved: -1427
    lz4convert_test.go:98: lz4 -> snappy bytes saved: -1430
    lz4convert_test.go:99: data -> s2 (default) bytes saved: -8771
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 4863
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: -7344
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 6290
    --- PASS: TestLZ4Converter_ConvertBlock/txt2 (0.00s)
=== RUN   TestLZ4Converter_ConvertBlock/txt3
    lz4convert_test.go:42: input size: 426754
    lz4convert_test.go:43: lz4 size: 207038
    lz4convert_test.go:60: lz4->snappy size: 206693
    lz4convert_test.go:79: lz4->s2 size: 206654
    lz4convert_test.go:91: s2 (default) size: 220380
    lz4convert_test.go:95: s2 (better) size: 184936
    lz4convert_test.go:97: lz4 -> s2 bytes saved: 384
    lz4convert_test.go:98: lz4 -> snappy bytes saved: 345
    lz4convert_test.go:99: data -> s2 (default) bytes saved: -13342
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 22102
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: -13726
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 21718
    --- PASS: TestLZ4Converter_ConvertBlock/txt3 (0.01s)
=== RUN   TestLZ4Converter_ConvertBlock/txt4
    lz4convert_test.go:42: input size: 481861
    lz4convert_test.go:43: lz4 size: 277731
    lz4convert_test.go:60: lz4->snappy size: 286863
    lz4convert_test.go:79: lz4->s2 size: 286856
    lz4convert_test.go:91: s2 (default) size: 318193
    lz4convert_test.go:95: s2 (better) size: 264987
    lz4convert_test.go:97: lz4 -> s2 bytes saved: -9125
    lz4convert_test.go:98: lz4 -> snappy bytes saved: -9132
    lz4convert_test.go:99: data -> s2 (default) bytes saved: -40462
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 12744
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: -31337
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 21869
    --- PASS: TestLZ4Converter_ConvertBlock/txt4 (0.01s)
=== RUN   TestLZ4Converter_ConvertBlock/pb
    lz4convert_test.go:42: input size: 118588
    lz4convert_test.go:43: lz4 size: 19003
    lz4convert_test.go:60: lz4->snappy size: 21130
    lz4convert_test.go:79: lz4->s2 size: 19002
    lz4convert_test.go:91: s2 (default) size: 18603
    lz4convert_test.go:95: s2 (better) size: 17686
    lz4convert_test.go:97: lz4 -> s2 bytes saved: 1
    lz4convert_test.go:98: lz4 -> snappy bytes saved: -2127
    lz4convert_test.go:99: data -> s2 (default) bytes saved: 400
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 1317
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: 399
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 1316
    --- PASS: TestLZ4Converter_ConvertBlock/pb (0.00s)
=== RUN   TestLZ4Converter_ConvertBlock/gaviota
    lz4convert_test.go:42: input size: 184320
    lz4convert_test.go:43: lz4 size: 71749
    lz4convert_test.go:60: lz4->snappy size: 63392
    lz4convert_test.go:79: lz4->s2 size: 62446
    lz4convert_test.go:91: s2 (default) size: 65016
    lz4convert_test.go:95: s2 (better) size: 55395
    lz4convert_test.go:97: lz4 -> s2 bytes saved: 9303
    lz4convert_test.go:98: lz4 -> snappy bytes saved: 8357
    lz4convert_test.go:99: data -> s2 (default) bytes saved: 6733
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 16354
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: -2570
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 7051
    --- PASS: TestLZ4Converter_ConvertBlock/gaviota (0.00s)
=== RUN   TestLZ4Converter_ConvertBlock/txt1_128b
    lz4convert_test.go:42: input size: 128
    lz4convert_test.go:43: lz4 size: 84
    lz4convert_test.go:60: lz4->snappy size: 84
    lz4convert_test.go:79: lz4->s2 size: 84
    lz4convert_test.go:91: s2 (default) size: 80
    lz4convert_test.go:95: s2 (better) size: 76
    lz4convert_test.go:97: lz4 -> s2 bytes saved: 0
    lz4convert_test.go:98: lz4 -> snappy bytes saved: 0
    lz4convert_test.go:99: data -> s2 (default) bytes saved: 4
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 8
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: 4
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 8
    --- PASS: TestLZ4Converter_ConvertBlock/txt1_128b (0.00s)
=== RUN   TestLZ4Converter_ConvertBlock/txt1_1000b
    lz4convert_test.go:42: input size: 1000
    lz4convert_test.go:43: lz4 size: 807
    lz4convert_test.go:60: lz4->snappy size: 791
    lz4convert_test.go:79: lz4->s2 size: 791
    lz4convert_test.go:91: s2 (default) size: 772
    lz4convert_test.go:95: s2 (better) size: 744
    lz4convert_test.go:97: lz4 -> s2 bytes saved: 16
    lz4convert_test.go:98: lz4 -> snappy bytes saved: 16
    lz4convert_test.go:99: data -> s2 (default) bytes saved: 35
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 63
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: 19
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 47
    --- PASS: TestLZ4Converter_ConvertBlock/txt1_1000b (0.00s)
=== RUN   TestLZ4Converter_ConvertBlock/txt1_10000b
    lz4convert_test.go:42: input size: 10000
    lz4convert_test.go:43: lz4 size: 6969
    lz4convert_test.go:60: lz4->snappy size: 6879
    lz4convert_test.go:79: lz4->s2 size: 6879
    lz4convert_test.go:91: s2 (default) size: 6931
    lz4convert_test.go:95: s2 (better) size: 6216
    lz4convert_test.go:97: lz4 -> s2 bytes saved: 90
    lz4convert_test.go:98: lz4 -> snappy bytes saved: 90
    lz4convert_test.go:99: data -> s2 (default) bytes saved: 38
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 753
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: -52
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 663
    --- PASS: TestLZ4Converter_ConvertBlock/txt1_10000b (0.00s)
=== RUN   TestLZ4Converter_ConvertBlock/txt1_20000b
    lz4convert_test.go:42: input size: 20000
    lz4convert_test.go:43: lz4 size: 12750
    lz4convert_test.go:60: lz4->snappy size: 12755
    lz4convert_test.go:79: lz4->s2 size: 12755
    lz4convert_test.go:91: s2 (default) size: 13513
    lz4convert_test.go:95: s2 (better) size: 11489
    lz4convert_test.go:97: lz4 -> s2 bytes saved: -5
    lz4convert_test.go:98: lz4 -> snappy bytes saved: -5
    lz4convert_test.go:99: data -> s2 (default) bytes saved: -763
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 1261
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: -758
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 1266
```

Loading branch information

klauspost committed Feb 17, 2023

1 parent 0793ca1 commit 7e21226

huff0/decompress.go

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -61,7 +61,7 @@ func ReadTable(in []byte, s *Scratch) (s2 *Scratch, remain []byte, err error) {
  
    		b, err := fse.Decompress(in[:iSize], s.fse)

    		s.fse.Out = nil

    		if err != nil {

    			return s, nil, err

    			return s, nil, fmt.Errorf("fse decompress returned: %w", err)

    		}

    		if len(b) > 255 {

    			return s, nil, errors.New("corrupt input: output table too large")

internal/lz4ref/LICENSE

-Original file line number
+Diff line change
@@ -0,0 +1,28 @@
+    Copyright (c) 2015, Pierre Curto
+    All rights reserved.
+    Redistribution and use in source and binary forms, with or without
+    modification, are permitted provided that the following conditions are met:
+    * Redistributions of source code must retain the above copyright notice, this
+      list of conditions and the following disclaimer.
+    * Redistributions in binary form must reproduce the above copyright notice,
+      this list of conditions and the following disclaimer in the documentation
+      and/or other materials provided with the distribution.
+    * Neither the name of xxHash nor the names of its
+      contributors may be used to endorse or promote products derived from
+      this software without specific prior written permission.
+    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+    AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+    IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+    DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+    FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+    DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+    SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+    CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+    OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+    OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

0 comments on commit `7e21226`

Please sign in to comment.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Commit

There are no files selected for viewing

0 comments on commit `7e21226`

Commit

There are no files selected for viewing

0 comments on commit 7e21226

0 comments on commit `7e21226`