decompress.ps1: Rewrite extract_xxx functions #3204

niheaven · 2019-03-10T11:29:02Z

PART II of #3169

Combine lessmsi_config, extract_lessmsi into extract_msi (unique function, switch by MSIEXTRACT_USE_LESSMSI)
Rewrite unpack_inno to extract_inno and handle InnoSetup in dl_urls with extract_inno
Use Expand-Zipfile function in scoopinstaller/install as new extract_zip
Use &(file_path 7zip 7z.exe) instead of 7z shim to avoid error caused by scoop install 7zip 7zip-zstd; scoop uninstall 7zip-zstd
extract_xxx now have similar params ($path, $to, $recurse)
Remove unused condition in depends.ps1

Test Cases are as follow:

Describing Decompression function

  Context 7zip extractionLoading 7z1900-x64.msi from cache
Checking hash of 7z1900-x64.msi ... ok.
Extracting 7z1900-x64.msi ... done.
Linking D:\Scoop\apps\7zip\current => D:\Scoop\apps\7zip\19.00
'7zip' (19.00) was installed successfully!
    [+] extract normal compressed file 179ms
    [+] extract nested compressed file 495ms
    [+] extract nested compressed file with different inner name 235ms
    [+] works with '$recurse' param 89ms

  Context msi extraction
Loading lessmsi-v1.6.3.zip from cache
Checking hash of lessmsi-v1.6.3.zip ... ok.
Extracting lessmsi-v1.6.3.zip ... done.
Linking D:\Scoop\apps\lessmsi\current => D:\Scoop\apps\lessmsi\1.6.3
'lessmsi' (1.6.3) was installed successfully!
    [+] extract normal MSI file 1.71s
    [+] extract empty MSI file using lessmsi 317ms
    [+] works with '$recurse' param 1.11s

  Context inno extraction
Loading innounp048.rar from cache
Checking hash of innounp048.rar ... ok.
Extracting innounp048.rar ... done.
Linking D:\Scoop\apps\innounp\current => D:\Scoop\apps\innounp\0.48
'innounp' (0.48) was installed successfully!
    [+] extract Inno Setup file 246ms
    [+] works with '$recurse' param 169ms

  Context zip extraction
    [+] extract compressed file 117ms
    [+] works with '$recurse' param 112ms

Please test and review. @r15ch13 @rasa @Ash258 @h404bi

PS. After merge, this PR will fix #3120, fix #3198, fix #3287, fix #3230, fix #3080, fix #3374, fix #3339, fix #3388

niheaven · 2019-03-17T16:44:10Z

Fix #3080 in 079c4a5

chawyehsu · 2019-03-18T11:05:46Z

Could you add more tests to increase the test coverage of the decompress functionality? Such as extracting .tar.*, etc.

niheaven · 2019-03-18T11:15:27Z

I'm trying to, but further test cases need 7zip, lessmsi, innounp (and maybe dark?). I'm not sure appveyor could install these tools just for testing. If it could, I could add more. Otherwise, I would try to seperate some function except for actually extraction for testing param passthru.

r15ch13 · 2019-03-18T12:59:28Z

7-Zip 18.05 is available on Appveyor: https://www.appveyor.com/docs/windows-images-software/#tools
Will try adding lessmsi and innounp (test branch: https://github.com/lukesampson/scoop/tree/appveyor-tests)

Ash258 · 2019-03-18T13:06:53Z

You could always use scoop installing as backup plan, if there is no better solution. Create the last test case in which you just do something like bin\scoop.ps1 refresh; bin\scoop.ps1 install <whatever you like> in beginning. And then test these extractions.

niheaven · 2019-03-18T13:19:59Z

It's better to use scoop installed 7zip lessmsi and innounp, since decompress.ps1 use these ones. Using just 7z x will lead to some errors in practice, so I change it to &(file_path 7zip 7z.exe).

niheaven · 2019-03-18T13:22:39Z

You could always use scoop installing as backup plan, if there is no better solution. Create the last test case in which you just do something like bin\scoop.ps1 refresh; bin\scoop.ps1 install <whatever you like> in beginning. And then test these extractions.

I'll try to make some more test cases.

r15ch13 · 2019-03-18T13:42:04Z

Just downloads lessmsi and innounp: 1facb58
Will not work by using file_path() though.

https://ci.appveyor.com/project/lukesampson/scoop/builds/23155191/job/ij3b30cvq9as63hk

niheaven · 2019-03-19T10:15:11Z

Sorry for so many force-push, I've tested CI with installing tools by scoop and achieved the goal with a simplified install_app_ci function forked from install_app.

Now I could add more test cases with these installed tools.

PS. I'm not familiar with Pester's Tag, and I want to make some tweak in test\bin\test.ps1 that it wouldn't test Decompress.Tests.ps1 while decompress.ps1 is not changed. Please review the two test files and make sure it works.

Ash258 · 2019-04-18T08:16:49Z

Is it all?
I should have time today in office to test some manifests installation and today / tomorrow it could be merged

SideNote: Could you look into ScoopInstaller/Extras#2021 (comment) in next PR? Edit Expand-7zip function with some optional [ValueFromRamainingArguments] $Arguments # Or just string whatever and then call 7z x .. @Arguments

niheaven · 2019-04-18T08:29:13Z

It should be enough for a single PR.

I'll add some universal exe unpacker functions in the next PR, and check the commit for NSIS unpacker.

Ash258 · 2019-04-18T09:33:24Z

Tested:

niheaven · 2019-04-18T16:11:51Z

Thanks, will you do more tests on it?

Ash258 · 2019-04-18T18:13:36Z

I don't think so. I would put it into wild.

Ash258

LGTM

Ash258 · 2019-04-21T10:49:39Z

Could you please adopt deprecation warning to old functions (extract_7z, ..) after #3341 is merged??
#3341 (comment)

Just merge master and add `Show-DeprecationMessage $MyInvocation 'New-Name'

niheaven · 2019-04-21T11:34:38Z

Okay, let's move on.

niheaven · 2019-04-21T11:51:03Z

It's ready to merge. @r15ch13 @Ash258

r15ch13 · 2019-04-26T18:01:47Z

@niheaven some of the tests are failing
https://ci.appveyor.com/project/lukesampson/scoop/builds/24128381/job/msr3s7k9kld9qa4g

r15ch13 · 2019-04-26T18:05:52Z

Also install_app_ci should check if the program is already available. Otherwise it will always download (or load the file from cache) and install it on each run of Invoke-Pester

r15ch13 · 2019-04-26T18:37:55Z

Fixed with 79a6d5c f919a3c ff15d1d

niheaven · 2019-04-27T00:31:29Z

@r15ch13 Thanks, I'm not familliar with Pester...

ref: ScoopInstaller/Extras#2021 (comment), #3204 (comment) - Add new param `Overwrite` to `Expand-7ZipArchive`, with following values - "All": 7z param `-aoa`, overwrite All existing files without prompt, behave the same with default - "Skip": 7z param `-aos`, Skip extracting of existing files - "Rename": 7z param `-aou`, aUto rename extracting file - Add new param `Switches` to `Expand-7ZipArchive` and `Expand-InnoArchive` - It could pass all unrecognized params to `7z` and `innounp` - Patch `extract_7zip` and `unpack_inno` to adapt to `Expand-XXX` Usage: ScoopInstaller/Extras#2021, ScoopInstaller/Extras#2070

- The main bucket of Scoop has been separated to https://github.com/ScoopInstaller/Main - Requires PowerShell 5 and newer #3330 - Improve and refactor decompression functions #3204 #3409 #3411 - Move scoop config from `~\.scoop` to `~\.config\scoop\config.json` #3242 - Add `scoop hold` and `scoop unhold` for excluding programs from updates #3444 - Refactored some Core functions #3314 #3416 - Small fix for starting processes on Windows 7 #3415 Co-authored-by: Hsiao-nan Cheung <niheaven@gmail.com> Co-authored-by: Chawye Hsu <h404bi@outlook.com> Co-authored-by: Jakub Čábera <cabera.jakub@gmail.com>

niheaven mentioned this pull request Mar 17, 2019

Scoop doesn't extract .tar.* archives fully #3080

Closed

niheaven force-pushed the rewrite-decompress branch 9 times, most recently from df1d165 to 5c75413 Compare March 19, 2019 10:05

niheaven force-pushed the rewrite-decompress branch 11 times, most recently from 3390bed to 7fd9c2b Compare March 21, 2019 03:46

Fix wrapper

4c53b2e

Ash258 approved these changes Apr 18, 2019

View reviewed changes

Ash258 mentioned this pull request Apr 21, 2019

Adopt new function naming after decompress refactor Ash258/Shovel-Ash258#18

Merged

Add 'Show-DeprecationMessage'

8c6dcfc

This was referenced Apr 22, 2019

7zip: Use github for download or add a switch -source to scoop install or use 7zip installed elsewhere #3287

Closed

Pritunl Client: Add version 1.0.1953.32 ScoopInstaller/Extras#2055

Merged

niheaven changed the base branch from master to develop April 25, 2019 09:49

This was referenced Apr 25, 2019

innosetup logs Permission Denied, but still succeeds #3339

Closed

7zip@19.00: extract_dir error #3388

Closed

r15ch13 merged commit 72730e0 into ScoopInstaller:develop Apr 26, 2019

niheaven deleted the rewrite-decompress branch April 29, 2019 03:31

niheaven mentioned this pull request Apr 29, 2019

Patch(decompress): Allow other args to be passthrough #3411

Merged

r15ch13 mentioned this pull request May 11, 2019

Bucket extraction and refactoring #3449

Merged

Ash258 mentioned this pull request May 16, 2019

feature(decompress): Add 'ExtractDir' to 'Expand-...' functions #3466

Merged

Ash258 mentioned this pull request Oct 18, 2020

Fixed upstream issues Ash258/Scoop-Core#63

Open

Ash258 mentioned this pull request Jan 31, 2021

refactor(versions) Ash258/Scoop-Core#23

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

decompress.ps1: Rewrite extract_xxx functions #3204

decompress.ps1: Rewrite extract_xxx functions #3204

niheaven commented Mar 10, 2019 •

edited

Loading

niheaven commented Mar 17, 2019

chawyehsu commented Mar 18, 2019

niheaven commented Mar 18, 2019

r15ch13 commented Mar 18, 2019

Ash258 commented Mar 18, 2019

niheaven commented Mar 18, 2019 •

edited

Loading

niheaven commented Mar 18, 2019 •

edited

Loading

r15ch13 commented Mar 18, 2019

niheaven commented Mar 19, 2019

Ash258 commented Apr 18, 2019

niheaven commented Apr 18, 2019

Ash258 commented Apr 18, 2019

niheaven commented Apr 18, 2019 •

edited

Loading

Ash258 commented Apr 18, 2019

Ash258 left a comment

Ash258 commented Apr 21, 2019

niheaven commented Apr 21, 2019

niheaven commented Apr 21, 2019

r15ch13 commented Apr 26, 2019

r15ch13 commented Apr 26, 2019

r15ch13 commented Apr 26, 2019

niheaven commented Apr 27, 2019

decompress.ps1: Rewrite extract_xxx functions #3204

decompress.ps1: Rewrite extract_xxx functions #3204

Conversation

niheaven commented Mar 10, 2019 • edited Loading

niheaven commented Mar 17, 2019

chawyehsu commented Mar 18, 2019

niheaven commented Mar 18, 2019

r15ch13 commented Mar 18, 2019

Ash258 commented Mar 18, 2019

niheaven commented Mar 18, 2019 • edited Loading

niheaven commented Mar 18, 2019 • edited Loading

r15ch13 commented Mar 18, 2019

niheaven commented Mar 19, 2019

Ash258 commented Apr 18, 2019

niheaven commented Apr 18, 2019

Ash258 commented Apr 18, 2019

niheaven commented Apr 18, 2019 • edited Loading

Ash258 commented Apr 18, 2019

Ash258 left a comment

Choose a reason for hiding this comment

Ash258 commented Apr 21, 2019

niheaven commented Apr 21, 2019

niheaven commented Apr 21, 2019

r15ch13 commented Apr 26, 2019

r15ch13 commented Apr 26, 2019

r15ch13 commented Apr 26, 2019

niheaven commented Apr 27, 2019

niheaven commented Mar 10, 2019 •

edited

Loading

niheaven commented Mar 18, 2019 •

edited

Loading

niheaven commented Mar 18, 2019 •

edited

Loading

niheaven commented Apr 18, 2019 •

edited

Loading