Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
Possibly integrate zlib-ng #416
I've heard good things about pigz (Chris Rorden is a big fan). I'm initially a bit concerned that the source code is only available as a tarball from 2 years ago.
It also feels like it may work well in posix environments, but I don't see how well it is supported across platforms.
You might find the my comparison of different GZ compression strategies relevant. For that test I intentionally used a slow hard disk. With zlib you can write a compressed file direct to disk. Traditionally, with pigz we need to write the raw data to disk and then pigz loads this file and compresses it. Therefore, one needs to wonder if the parallel performance of pigz can be offset by the disk IO. The link also shows that if you have a modern version of pigz on Unix you can use a named-pipe to avoid writing the whole file to a slow disk. In my experience, pigz works fine on Windows, though I don't think you can use the named-pipe trick. The fact that it has not changed much in the last years may just be a sign it is mature. The original deflate/gz format dates back to a time when memory was limited and multiple cores was exotic. It has achieved widespread use, but innovations will really come from new standards that leverage modern computers. zstd is extremely impressive, in particular for medical datasets when it is combined with a byte-shuffling filter.
In my testing, I always found Cloudflare's zlib faster than zlib-ng. However, this may have changed or may have been due to compiler settings. Cloudflare seems stuck at an older version of zlib (1.2.8).