lrzip-bz2 performs worse than just bz2 for large json file

[phiresky]

See here:

 880M 14. Mai 21:25 files.tar
  11M 14. Mai 21:25 files.tar.bz2
  21M 14. Mai 21:31 files.tar.lrz-bz2
  22M 14. Mai 21:33 files.tar.lrz-lzma
  51M 14. Mai 21:29 files.tar.lrz-nocompress
  21M 14. Mai 21:29 files.tar.lrz-nocompress.bz2
  19M 14. Mai 21:35 files.tar.lrz-zpaq

Using just bzip2, the 880MB file compresses to 11MB. Using lrzip -b, it is 21MB.
How can this happen / should this happen?

Here is the file (.zip so github accepts the upload):
files.tar.zip

[pete4abw]

BZIP operates much better on small files. Lrzip on large ones. There are
1000 or so files and all are small. The long range compression routine
does not work as well on smaller files. Even though you tar the files
together, the bzip compression window is much smaller and operates more
efficiently with this type.

Interestingly, if you tar up the Kernel source tree, you get much better
results.

On 05/14/2016 02:41 PM, phiresky wrote:

See here:

|880M 14. Mai 21:25 files.tar 11M 14. Mai 21:25 files.tar.bz2 21M 14.
Mai 21:31 files.tar.lrz-bz2 22M 14. Mai 21:33 files.tar.lrz-lzma 51M
14. Mai 21:29 files.tar.lrz-nocompress 21M 14. Mai 21:29
files.tar.lrz-nocompress.bz2 19M 14. Mai 21:35 files.tar.lrz-zpaq |

Using just bzip2, the 880MB file compresses to 11MB. Using |lrzip -b|,
it is 21MB.
How can this happen / should this happen?

Here is the file (.zip so github accepts the upload):
files.tar.zip https://github.com/ckolivas/lrzip/files/264669/files.zip

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#51

Peter Hyman
http://www.linkedin.com/in/peterh
C: +1(609)598-0262
C: +1(612)440-7383 (Pete)

[phiresky]

The json files are very similar, they can basically be seen as a single file (same file structure, same words, just different order). The result stays the same when instead of tarring them up using cat *.json.

[ckolivas]

"Should this happen?" - on rare occasions the underlying compression does a better job on the file due to the tightly packed redundancy, and separating out the dictionary from the matches is counterproductive. The only thing lrzip offers for this particular archive is much faster compression and decompression.