7z-based (optional) replacement for patool #4041

mih · 2020-01-17T13:00:05Z

Sitting on top of #4039

This is a complete draft implementation of a 7z-based file compression/decompression that should be as capable as the patool-based one.

TODO

Further related TODOs (for subsequent PRs) surfaced by this PR:

Need platform abstraction for cmdline argument quoting #4048 -> Replace shlex.quote() with quote_cmdlinearg() #4050
add p7zip as dependency/recommends to Debian package

mih · 2020-01-17T13:01:16Z

Grrrr, this was supposed to be in draft mode. I see no way to achieve that post-factum.

yarikoptic

Automagically fallback to patool if 7z command is not available. I would have even may be wrapped *comparess_file so if exception would happen, announce in the log (before reraising) to try an alternative method by changing that config setting. Otherwise it is burried and nobody would discover that there is a config setting they could tune (until they ask)

datalad/support/archives.py

yarikoptic · 2020-01-17T14:11:24Z

expand tests for additional archive formats and compressors

would also be nice to

make one of the travis runs use alternative (non default) archives handler (i.e. patool instead of 7z) by providing config var in the env.

mih · 2020-01-17T15:23:48Z

Holy cow, this is a rabbit hole... I am so deep in add_archive_content.py and I don't even know why...

codecov · 2020-01-17T15:49:52Z

Codecov Report

Merging #4041 into master will decrease coverage by 0.08%.
The diff coverage is 89.9%.

@@            Coverage Diff            @@
##           master   #4041      +/-   ##
=========================================
- Coverage   89.79%   89.7%   -0.09%     
=========================================
  Files         273     274       +1     
  Lines       36577   36620      +43     
=========================================
+ Hits        32843   32851       +8     
- Misses       3734    3769      +35

Impacted Files	Coverage Δ
datalad/interface/tests/test_download_url.py	`100% <ø> (ø)`	⬆️
datalad/tests/test_archives.py	`94.92% <100%> (+4.21%)`	⬆️
datalad/interface/download_url.py	`97.89% <100%> (+0.16%)`	⬆️
datalad/support/external_versions.py	`91.03% <58.33%> (-2.96%)`	⬇️
datalad/support/archives.py	`89.47% <84.21%> (+0.05%)`	⬆️
datalad/support/archive_utils_7z.py	`88% <88%> (ø)`
datalad/support/archive_utils_patool.py	`40.25% <0%> (-42.86%)`	⬇️
datalad/utils.py	`87.26% <0%> (ø)`	⬆️
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fca2866...25d7c34. Read the comment docs.

mih · 2020-01-17T15:58:45Z

Automagically fallback to patool if 7z command is not available.

I am reluctant to do this. We are pretty much in the same situation with patool. We can pull patool in via dependency, but we cannot pull in its dependencies (tar, zip, ...). Moreover, the reason why I am doing this to begin with, is that this code is broken, and it should be removed, or fixed. Hence, I went for a dedicated switch.

yarikoptic · 2020-01-17T17:24:36Z

We are pretty much in the same situation with patool. We can pull patool in via dependency, but we cannot pull in its dependencies (tar, zip, ...).

patool has fallbacks from 7z to e.g. zip, jar, etc, so it could work on systems without 7z. You might fix some bugs (on windows) but would introduce datalad crashes on others (without 7z, which is generally not installed by default, whenever zip etc are more known and more likely to be installed). so I would strongly appreciate (could buy a bottle) if we could avoid such crashes with a cost of a single if condition.

mih · 2020-01-19T12:56:57Z

I am planning to merge this relatively quickly: On Monday, or whenever the test pass (whichever happens last). It is the foundation for more work on windows, which will go into different directions, hence should not be added here.

make one of the travis runs use alternative (non default) archives handler (i.e. patool instead of 7z) by providing config var in the env.

I would leave this out here, for now, and as a topic for further debate. I think we should not test this code for every PR. Maybe a cron job would suffice.

@yarikoptic

…taladgh-4019) Keep the old implementation around (by request of @yarikoptic), but require a switch (datalad.runtime.use-patool) to turn it on.

Also enable all tests on Windows.

The reason is the same as what made us use MD5E instead MD5 as the default backend. If we purposefully remove an extension, we have mimetype inspection left is the only option to decided on some further processing (e.g. archive type). Possible, but needlessly complicated. An alternative fix would have been to not purposefully use an extension-less archive name in the associated tests.

@yarikoptic

Via @yarikoptic: > would remain a preferred way to simply reraise exception, since otherwise it would use this line as the source of exception while also mentioning original one (in python3 only)

@yarikoptic

Thanks @yarikoptic for pointing this out.

Now required on import.

Should be turned into a common helper dataladgh-4048

Required for switching between 7z and patool. Also make it reported by `wtf()`

…ves too XZ is now being tested, and 7z has built-in support.

…tion

All of the checks of the test generator fail on a system without p7zip. Before dataladgh-4041 (in particular, before beb5d04), the subset of checks that existed at that time was skipped with (see dataladgh-3176): SKIP: cmd:7z is missing. (Not) Funny enough but ATM we need p7zip installation to handle .gz files extraction 'correctly' However, with the change to the archive file name in 05e9353 (TST: Expand test coverage to additional relevant compression formats, 2020-01-17), the MissingExternalDependency exception that led to the skip for gzip formats is no longer triggered. And even if it were, the other file formats tested would fail. Mark these tests as known failures when p7zip isn't available.

mih force-pushed the patoolreplacement branch 3 times, most recently from cd028b7 to d5c1939 Compare Jan 17, 2020

yarikoptic requested changes Jan 17, 2020

View changes

datalad/support/archives.py Outdated Show resolved Hide resolved

datalad/support/archives.py Outdated Show resolved Hide resolved

mih mentioned this pull request Jan 18, 2020

RF: Move all patool code into a dedicated code file #4039

Merged

mih force-pushed the patoolreplacement branch from 2c727d4 to fd22c4f Compare Jan 19, 2020

This was referenced Jan 19, 2020

Use-case "encrypted archive" #2555

Open

support.archives.unixify_path() has type issue on Windows #2572

Closed

mih added the merge-if-ok label Jan 19, 2020

mih added this to Major issues in Windows Jan 19, 2020

mih mentioned this pull request Jan 20, 2020

Replace shlex.quote() with quote_cmdlinearg() #4050

Merged

1 task

mih added 12 commits Jan 20, 2020

RF: Beautify imports and a few other spots

5e1c0b1

RF: Reimplement patool functionality with straight 7z calls (fixes da…

beb5d04

…taladgh-4019) Keep the old implementation around (by request of @yarikoptic), but require a switch (datalad.runtime.use-patool) to turn it on.

RF: Normalize imports

58cbc4c

TST: Expand test coverage to additional relevant compression formats

05e9353

Also enable all tests on Windows.

BF: Revert previous change to improve error reporting

6c99b9a

Via @yarikoptic: > would remain a preferred way to simply reraise exception, since otherwise it would use this line as the source of exception while also mentioning original one (in python3 only)

ENH: Proper dependency test on 7z, and clearly suggest alternative

d674922

Thanks @yarikoptic for pointing this out.

BF: Install p7zip for doc build

dc62286

Now required on import.

RF: Enable fall-back on patool if possible, disable 7z-check on Windows

7457645

BF: Use more appropriate cmdline arg quoting on windows

98aa1ac

Should be turned into a common helper dataladgh-4048

DOC: Fix docstring

f61c261

BF+TST: Adjust after implementation change

6cfaf97

mih added 5 commits Jan 20, 2020

TST: Enable more tests on appveyor

d01413e

BF: Implement test of 7z version

7fb806a

Required for switching between 7z and patool. Also make it reported by `wtf()`

TST: Pull in xz-utils on travis, so that patool can handle such archi…

d9fec82

…ves too XZ is now being tested, and 7z has built-in support.

BF: Prevent misinterpretation of filenames starting with '-' as an op…

5a15a92

…tion

RF: Use quote_cmdlinearg() helper instead of custom implementation

25d7c34

mih force-pushed the patoolreplacement branch from 66d95f1 to 25d7c34 Compare Jan 20, 2020

mih merged commit 790ab49 into datalad:master Jan 21, 2020
17 checks passed

mih deleted the patoolreplacement branch Jan 21, 2020

mih mentioned this pull request Jan 21, 2020

datalad.support.archives fundamentally broken on windows #4018

Closed

mih moved this from Major issues to Done in Windows Mar 12, 2020

kyleam mentioned this pull request May 11, 2020

TST: archive: Skip tests that depend on p7zip #4512

Merged

7z-based (optional) replacement for patool #4041

7z-based (optional) replacement for patool #4041

mih commented Jan 17, 2020 •

edited

mih commented Jan 17, 2020

yarikoptic left a comment

yarikoptic commented Jan 17, 2020

mih commented Jan 17, 2020

codecov bot commented Jan 17, 2020 •

edited

mih commented Jan 17, 2020

yarikoptic commented Jan 17, 2020

mih commented Jan 19, 2020 •

edited

7z-based (optional) replacement for patool #4041

7z-based (optional) replacement for patool #4041

Conversation

mih commented Jan 17, 2020 • edited

mih commented Jan 17, 2020

yarikoptic left a comment

yarikoptic commented Jan 17, 2020

mih commented Jan 17, 2020

codecov bot commented Jan 17, 2020 • edited

Codecov Report

mih commented Jan 17, 2020

yarikoptic commented Jan 17, 2020

mih commented Jan 19, 2020 • edited

mih commented Jan 17, 2020 •

edited

codecov bot commented Jan 17, 2020 •

edited

mih commented Jan 19, 2020 •

edited