-
-
Notifications
You must be signed in to change notification settings - Fork 72
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
provide method for extracting specific files from an archive #58
Comments
py7zr was designed to do it in future, but current implementation does not have the function. An extraction main function is built with five code blocks.
Currently step.1 works for all of items. It calls 'self. worker.register_filelike()' against each target items. when calling it with 'None' as a target path, then decompress function will skip it. |
A branch https://github.com/miurahr/py7zr/commits/topic-extraction-filter try to realize it with dirty hack. Please see test case to know how to specify files. Lines 372 to 383 in c3f580e
There are two items to extract in this test case, one is 'scripts' directory and another is 'scripts/py7zr' file. Other files 'setup.py' and 'setup.cfg' is skipped. Any feedback? |
I nearly implemented a similar solution,
I only wasn't sure if it's the intended solution, because I only had a brief look on the Not sure why you find your solution "dirty"? |
Your code does almost as same as I tried. Lines 737 to 739 in c3f580e
There are several design considerations for API.
Lines 263 to 274 in c3f580e
|
Here is my idea how to utilize Lines 391 to 397 in e51772a
If it is convenient, I'd like to add new method such as |
When trying to run with your changes, I get an error:
called the
Also an error occurs, if I provide an output path...
|
It is what I mentioned at
|
very specifically asked, because the 2nd sentence confuses me... |
means you should call with
|
Please see Lines 374 to 379 in e51772a
That is not |
Thanks for the hint! I am reading in the 7z file BEFORE a loop.
My code for the extraction looks like this:
|
strange...
in the loop, then it works. |
7-zip format is basically use 'solid' archive, that all files are compressed in single archive stream. Both extract() and extractall() method have to process all the archive data, read all the archive data, even that is several giga bytes. After you called extract() method, an internal file pointer has positioned to end of data. You want to process large archive (30Gb) and looping method, if it is twice, you read 30Gb x 2 = 60Gb from disk. If it is ten times of loop, you read 300Gb from disk! Solid 7-zip format does not support random access by its nature, but optimized to compression ratio. Users are recommended to construct a list of files to extract, you can use loop there, then call extact() only once. |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
Thanks @michaelfecher for testing. Lines 405 to 412 in ae9e76a
|
Is your feature request related to a problem? Please describe.
For a serverless batch process, I have big 7z files floating around (2.5Gb+), which I don't want to extract completely (30Gb+).
Otherwise the serverless aproach would be very expensive.
Describe the solution you'd like
Instead I want to selectively extract files.
The argument of the method could be of a list of strings, which should match with the given file list of the archive and only extract the matches.
This would enable a leaner solution and focus on the files I am interested in.
Describe alternatives you've considered
There are currently no alternatives, because I already tried to do it via the 7z binary provisioning via copying the binaries + libraries. :)
It happens, that there is a permission denied error - regardless of what I am doing.
The support also couldn't assist here.
The text was updated successfully, but these errors were encountered: