[WIP] Replace gzip/bzip2/xz with pigz/pbzip2/pxz for multicore

[leemgs]

This PR is to fix issue #113. And It can be used as an alternative of PR #117.
In modern generation, most of the developers are using a machine based on
multi-core architecture. From now on, Let's prepare to support parallelism.

pigz is compatible with the existing gzip because pigz is using gzip's library.
$ ldd /usr/bin/pigz | grep -e 'libz|pthread'
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fe318f23000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fe318d09000)

pbzip2 is compatible with the existing pbzip2 because pbzip2 is using bzip2's library.
$ ldd /usr/bin/pbzip2 | grep -e 'libbz2|pthread'
libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007fa9d089a000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fa9d067d000)

pxz is compatible with the existing xz because pxz is using xz's library.
$ ldd /usr/bin/pxz |grep -e 'liblzma|pthread'
liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f27d3fd5000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f27d3b95000)

Signed-off-by: Geunsik Lim geunsik.lim@samsung.com

[pmatilai]

After wasting time looking at the obviously completely untested patch against an imaginary library in PR #117 ... to put it really really politely, I'm not in a very positive mood towards this work.

I'm not going to look at a single patch until you show me some numbers to back up the change, for each compression type/program separately on traditional spinning harddisk, SSD numbers as an added bonus. In rpmbuild context to show you've actually tested it.

[ldv-alt]

The idea is wrong: you cannot replace gzip, bzip2, and xz because they are much more widespread. The implementation is wrong: it does not achieve the declared goal.

[leemgs]

@pmatilai , Sorry for the inconvenience caused to you. I will propose new PR after finding the complete solution to support gzip/bzip2 as well as xz. This WIP will be closed.

[proyvind]

As the multithreaded (de)compression support for pigz isn't implemented in the binary, the benefits from aren't really there as it won't be possible to implement multithreaded (de)compression for using in the rpmio library.
Same goes for pbzip2.

For xz, multithreaded compression support has been implemented into liblzma and since added to librpmio xz payload support. Unless pxz supports parallell decompression (IIRC it does't, I borrowed the code for the initial python liblzma extension, where the support for was implemented through use of openmp.
Then again, as support was implemented by compressing every block in separate thread, one might imagine that multithreaded compression would be possibly by doing decompression of each block in separate threads.. But I don't think it did..
Once xz upstream has implemented multithreaded decompression in liblzma, this will be a considerate improvement in time spent on decompression packages for installing..

For pbzip2 & gzip I think they do parallel decompression, don't they.

Anyways, to benefit from the reduction in size, the source which the packager usually aren't the one responsible for producing the compressed tarball releases would have to be compressed using by upstream projects. And if I'm not mistaken, it's required to be compressed with this multithreaded compression to benefit from multithreaded decompression..

If you're eager to implement support for parallel gzip or bzip2 compression anywhere in rpm, you would have to implement it in bzip2 & gzip payload support in librpmio.

So in essence, this just isn't very useful to rpm unless the support is provided by a library, and then it would be of interest to librpmio itself, not decompression of tarballs for package building.