Exports are bottlenecked by gzip compression which cannot be disabled

[dralley]

Version
Any

Describe the bug

We identified a severe bottleneck in the way hammer exports work and found the code in upstream Pulp.

Line 406 of https://github.com/pulp/pulpcore/blob/main/pulpcore/app/tasks/export.py

"with tarfile.open(tarfile_fp, "w|gz", fileobj=split_process.stdin)"

The tarfile Python module creates tar files using the gizp Python module.

Data compression for the gzip Python module is provided by the zlib Python module.

The zlib Python module calls the zlib library.

If defaults are used the whole way through this series of events, the result is a single threaded pulp process doing compression of a tarball containing a massive content library. This bottleneck and can make large hammer exports take several days.

Modifying the lines that tell the tarfile.open function to NOT use compression ( change "w|gz" to "w" ) dramatically speeds up the hammer export. In our testing it reduced the time from days to just hours. The drawback is the file size was significantly larger, but the trade-off is worthwhile given we have tight timeframes and plentiful disk capacity.

Can this bottleneck be addressed with multi-threaded gzip compression?

and/or

Can a hammer command line option for no compression be implemented?

Run a hammer export and monitor Pulp processes. One process with run at 100% CPU. Modify the abovementioned Python script to NOT use gzip encryption, and an uncompressed tarball will be created instead much quicker and with multiple Pulp processes.

Steps to Reproduce:

1. Run a "hammer export". Monitor the Pulp process CPU usage and time taken to complete export.
2. Change the abovementioned Python code in Pulp.
3. Run a "hammer export" again and note performance improvement.

Expected behavior
A clear and concise description of what you expected to happen.

Additional context
https://bugzilla.redhat.com/show_bug.cgi?id=2188504

[dralley]

I don't know that disabling compression entirely is a good idea. However, the default compression level used is level 9, the most computationally expensive (slow) one. Given level 9 is (roughly) 6x slower than level 1, but only compresses about 20% better, this is probably a poor tradeoff.

We should look at using levels 1-3 instead.

[pulpbot]

https://bugzilla.redhat.com/show_bug.cgi?id=2238353