A Curious Case with bzip3 compression

[pete4abw]

Data compression wonks are always testing and analyzing results of different files, file types, methods, etc.
While evaluating results with the Silesia corpus, I noticed that the results for bzip3 were completely opposite of expectations. For some reason (yet unexplained), the results for level 1 were the best, and level 7 was the worst. I also tested the file using the bzip3 binary with different block sizes and again, BS = 16MB had the best. @kspalaiologos , I would be interested to get your impression on this. Thank you!

lrzip-next version: 0.12.4
Debian: 12.2
bzip3 library: 1.2.2-2 (stable)
system ram: 16GB

See the following table.

Comp Level	Compressed Size	Compress Time	Decompress Time
1	47,358,600	6.39	4.66
2	47,725,672	6.44	5.33
3	48,081,688	6.58	4.80
4	48,223,286	6.63	5.07
5	48,990,143	8.21	6.37
6	49,524,341	10.58	8.02
7	49,558,337	13.11	8.13
8	49,518,016	15.00	7.82
9	49,380,508	24.19	7.79

As defined in lrzip-next, bzip3 uses block sizes according to compression level from 32MB to 511MB.

Comp Level	BZIP3 Block Size (MB)	Note
1	32
2	48
3	64
4	96
5	128
6	192	Max on 16MB system
7	256
8	384
9	511

[kspalaiologos]

Hello!

My personal suspicion: the files in the silesia corpus are largely unrelated (also of different data types), so compressing them separately (i.e. approx. with a small block size) yields better results.

[kspalaiologos]

E.g. consider this:

 [13:16] ~/silesia % for F in *; do bzip3 $F; done
 [13:16] ~/silesia % wc -c *.bz3
 2233954 dickens.bz3
15881570 mozilla.bz3
 2119990 mr.bz3
 1425956 nci.bz3
 2526882 ooffice.bz3
 2261872 osdb.bz3
  980475 reymont.bz3
 3976068 samba.bz3
 4673313 sao.bz3
 6787036 webster.bz3
  405877 xml.bz3
 3657090 x-ray.bz3
46930083 total
 [13:17] ~/silesia % rm -rf *.bz3
 [13:17] ~/silesia % tar -cf silesia.tar *
 [13:17] ~/silesia % bzip3 -veb16 silesia.tar
 silesia.tar:	211957760 -> 47237109 bytes, 22.29%, 1.78 bpb

Compressing the files separately gives an even better result.

[pete4abw]

Interesting. Thank you for the explanation. Since lrzip-next is not a journaling compressor, for now, at least, we're left with this anomaly. I suppose we'll always find some contradictory results if you look hard enough! bzip3 however yields great results and very fast times.