Question: dictionary construction for LZ4

[LeeYoung624]

Hi there,
I'm a new guy in compression. Recently I have found that the latest version of LZ4 and Zstd (both created by Yann:) ) have mentioned about dictionary compression.
I have tried with Zstd, which achieve a better compression ratio with higher round speed when I use a dictionary trained by the compressed data. But for the LZ4 case, I found that LZ4 doesn't offer any API for constructing a dictionary(as it's said in LZ4 v1.8.1.2 release description 'lz4 doesn't specify how to build a dictionary. All it says is that it can be any file up to 64 KB.' ). And I tried LZ4 compression with the 64KB dictionary trained by Zstd, but it seems doesn't work.
I tried with -b1e9 -B4096, and here is my result:
without dictionary:
Benchmarking levels from 1 to 9
1#oot-framework.oat : 26750624 -> 15195120 (1.760), 431.0 MB/s ,1724.7 MB/s
2#oot-framework.oat : 26750624 -> 15195120 (1.760), 431.9 MB/s ,1760.0 MB/s
3#oot-framework.oat : 26750624 -> 14559095 (1.837), 79.7 MB/s ,1780.2 MB/s
4#oot-framework.oat : 26750624 -> 14537315 (1.840), 75.2 MB/s ,1724.6 MB/s
5#oot-framework.oat : 26750624 -> 14525203 (1.842), 76.1 MB/s ,1755.5 MB/s
6#oot-framework.oat : 26750624 -> 14519434 (1.842), 71.6 MB/s ,1753.4 MB/s
7#oot-framework.oat : 26750624 -> 14516259 (1.843), 71.0 MB/s ,1799.4 MB/s
8#oot-framework.oat : 26750624 -> 14514780 (1.843), 70.9 MB/s ,1797.1 MB/s
9#oot-framework.oat : 26750624 -> 14514142 (1.843), 69.9 MB/s ,1782.7 MB/s

with dictionary:
Benchmarking levels from 1 to 9
1#oot-framework.oat : 26750624 -> 15195120 (1.760), 424.0 MB/s ,1753.8 MB/s
2#oot-framework.oat : 26750624 -> 15195120 (1.760), 423.2 MB/s ,1759.9 MB/s
3#oot-framework.oat : 26750624 -> 14559095 (1.837), 77.2 MB/s ,1765.4 MB/s
4#oot-framework.oat : 26750624 -> 14537315 (1.840), 77.9 MB/s ,1822.4 MB/s
5#oot-framework.oat : 26750624 -> 14525203 (1.842), 76.9 MB/s ,1765.5 MB/s
6#oot-framework.oat : 26750624 -> 14519434 (1.842), 72.8 MB/s ,1769.7 MB/s
7#oot-framework.oat : 26750624 -> 14516259 (1.843), 70.2 MB/s ,1763.9 MB/s
8#oot-framework.oat : 26750624 -> 14514780 (1.843), 68.7 MB/s ,1764.0 MB/s
9#oot-framework.oat : 26750624 -> 14514142 (1.843), 68.0 MB/s ,1751.9 MB/s
As you can see, the ratio is exactly the same, and the speed seems didn't get any boost.
My question is:
1, Is there some instructions about the construction of dictionary for LZ4? If not, any suggestions?
2, My compression data is some specific binary files(in block size of 4KB), and I hope to have a good ratio and decompress speed(I'm not concerned about compression speed). In my opinion, this should work much better if I use a dictionary trained by the compressed file itself, with a high level of LZ4. Any other suggestion for my case?
As I said at first, I'm a new guy in compression, so feel free to tell me if I'm wrong.
Regards,
Lee

[Cyan4973]

Hi @LeeYoung624 ,

LZ4 dictionaries can be created using zstd dictionary builder, as specified in the original release note :

lz4 doesn't specify how to build a dictionary. All it says is that it can be any file up to 64 KB.
This approach is compatible with zstd dictionary builder, which can be instructed to create a 64 KB dictionary with this command :
zstd --train dirSamples/* -o dictName --maxdict=64KB

Since release notes are not necessarily read, and not part of the repository, maybe we could duplicate this instruction somewhere else, in the man page for example.

Dictionary work the same way for LZ4 as they do for zstd.
But the benchmark program included inside lz4 is not compatible with dictionaries, and transparently ignore them. We should probably fix that, at a minimum output a meaningful error message, or at best, make it compatible with dictionaries.

To test lz4 with dictionaries, you need to actually compress those small files. Therefore, cut input data into small blocks manually, typically using split command, then instruct to compress all files in the target directory, using typically lz4 -m -r command, and then collect result by counting bytes of produced *.lz4 files.

Example :

$ ./lz4 silesia4K/aaaa -c | wc -c
    1075
$ ./lz4 silesia4K/aaaa -c -D dictSilesia4K | wc -c
     982

$ ./lz4 silesia4K/bbbb -c | wc -c
    1268
$ ./lz4 silesia4K/bbbb -c -D dictSilesia4K | wc -c
    1084

$ ./lz4 githubUsers/user.01F0Ikhr36.json -c | wc -c
     400
$ ./lz4 githubUsers/user.01F0Ikhr36.json -c -D dictGithubUsers | wc -c
      41

[LeeYoung624]

Hi @Cyan4973
I didn't expect that your response so fast! That's really hardworking!
Thank you for your help, and the benchmark is indeed confusing:)