[23.2] Use python-isal for fast zip deflate compression in rocrate export #17342
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mvdbeek
force-pushed
the
isal_celery_export
branch
from
5c08f00
to
7e9c02b
Compare
mvdbeek
force-pushed
the
isal_celery_export
branch
from
8ee3b56
to
a4b14be
Compare
jdavcs
approved these changes
Jan 23, 2024
nsoranzo
reviewed
Jan 23, 2024
| @@ -2753,7 +2753,7 @@ def _finalize(self) -> None: | |||
| out_file = out_file_name[: -len(".zip")] | |||
| else: | |||
| out_file = out_file_name | |||
| rval = shutil.make_archive(out_file, "zip", self.export_directory) | |||
| rval = shutil.make_archive(out_file, "fastzip", self.export_directory) | |||
There was a problem hiding this comment.
I think this works only because this files also imports from galaxy.util.compression_utils import CompressedFile. Should we re-export shutil (maybe as gx_shutil) in lib/galaxy/util/compression_utils.py and import it in this file from there?
There was a problem hiding this comment.
I think this works only because this files also imports
from galaxy.util.compression_utils import CompressedFile
yes
Should we re-export
shutil(maybe asgx_shutil) inlib/galaxy/util/compression_utils.pyand import it in this file from there?
makes sense, sure
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We're currently archiving a lot of old and unused histories to tape by exporting histories to RO-crates, and CPU utilization appears to be the bottleneck.
I've switched out the default zlib DEFLATE compression with the more efficient DEFLATE implementation provided by https://github.com/intel/isa-l via https://pypi.org/project/isal/. I've also tried zlib-ng (https://github.com/zlib-ng/zlib-ng) but that was only marginally faster.
This is about 3 times faster on 8GB sampled from /dev/urandom (~160 MB/s vs ~56 MB/s), both locally and on galaxy07 under heavy load.
How to test the changes?
(Select all options that apply)
License