Tar on Windows is very slow when extracting many small files from tar files

[warpdesign]

Environment

Item	Value
OS, Version / Build	Windows 10 version 10.0.19041.388
Processor Architecture	AMD64
Processor Type & Model	Core i5 (Skylake)
Memory	8GB
Storage Type, free / capacity (e.g. C: SSD 128GB / 512GB)	SSD 256GB
Relevant apps installed	___

Description

Creating multiple small files is very slow: Windows Defender's real-time protection appears to makes things even slower but even when it's disabled Windows appears to lag behind Linux and macOS when handling multiple small files.

To show how slow Windows is at creating lots of small files, I downloaded the Firefox sources. Since it's a tar.bz2 file, I first uncompressed the .bz2 file to get the .tar file and all tests were done using this file as a source. It's a 852mb file that flattens to a directory containing 119 954 files.

I know it's an extreme case, but it's not rare having to deal with thousand of small files when working with npm, git, development, etc..

I added macOS results (running on a slow MacBook with a core m3) as a comparison.

Operation	Windows 10 (protection on)	Windows 10 (protection on, folder excluded)	Windows 10 (protection disabled)	Ubuntu (WSL2)	macOS
flatten tar file	28m38s	3m14s	1m29s	15s	50s
delete directory	58s	57s	45s	5s	14s

The commands I used were:

• Mac & Linux: time tar -xf firefox-40.0.source.tar and time rm -Rf mozilla-release
• Windows (PowerShell): Measure-Command { tar xf .\firefox-40.0.source.tar } and Measure-Command { rm -r -fo .\mozilla-release\ }

I am sure that improving these file operations will benefit to a lot of different Windows use cases:

• installing Windows apps
• installing dev packages, like npm, ruby gems,...
• git operations on large repositories
• building apps/websites (webpack, C/C++ compilers,...)
• apps like VSCode that need to deal with lots of files in the background
• installing Windows updates
• even browsers like Chrome/Edge have caches with lots of files in it
• ...

Windows & Linux (WSL2) tests were run on a Surface Book 1/core i5/8gb/256gb SSD.
Mac tests were run on a 2017 MacBook 12" with a core m3/256gb SSD/8gb and Catalina 10.15.3

Applications used:

App	Windows 10	Ubuntu (WSL2)	macOS
tar	bsdtar 3.3.2 - libarchive 3.3.2 zlib/1.2.5.F-ipp (tar.exe that's included in Windows 10)	tar (GNU tar) 1.30	bsdtar 3.3.2 - libarchive 3.3.2 zlib/1.2.11 liblzma/5.0.5 bz2lib/1.0.6
delete tool	rm -r -fo (powershell alias)	rm (GNU coreutils) 8.30	rm from Catalina 10.15.3

[dada051]

You can use Measure-Command in Powershell as a Linux's time equivalent

[bitcrazed]

Thanks for filing this @warpdesign. It's a well known issue and one we're already hard at work measuring and figuring out how best to deliver meaningful improvements.

As soon as we have analysis, measurements, plans etc., we'll update this and related threads.

Bear with us - exciting stuff on the way 😀

[lukeblevins]

I certainly notice when extracting a 1 GB+ ZIP file for Flutter SDK that speeds are throttled by Windows Security. Turning off Realtime Protection for the duration of the operation does make a difference.

[bitcrazed]

Hey @duke7553 Thanks for sharing. Yes, this is a known issue and something we're actively working to address.

Could you please update your comment and add a few details as to which tool you're using to extract the zip, link to specific zip file, measure-command { ... } output, etc. The more info & specifics you share, the more repro's we can analyze, and the greater likelihood that we can make meaningful improvements.

Thanks.

[warpdesign]

@bitcrazed The link to the tar.bz2 was already there (you first need to uncompress the bz2 file to get the tar file I used in the tests).

I added the version of the apps I used to flatten the tar file and to delete the folder.

The exact command line used for all systems was already there, only thing is that I measured using my phone's stopwatch function on Windows since I was using cmd and didn't know about measure-command of powershell.

[lukeblevins]

The Realtime Protection Tax Sits at 48%

When unzipping the 610 MB stable Flutter SDK 1.17.5 compressed archive, I get the following results in this order:

Windows Security Realtime Protection Disabled:
Completed in 3 minutes 25 seconds (205000ms)

Windows Security Realtime Protection Enabled:
Completed in 6 minutes 32 seconds (392000ms)

Other observations worth mentioning:

• In both tests, the progress graph is very inaccurate, as 50% of the extraction operation completed in ~10 sec
• Clicking "Extract" in the extract dialog occasionally results in a ~10 sec hang before any kind of work started

[orcmid]

Installers pausing at 100% is even more disturbing :).

Non-expert observations: Depending on the size of the directory for the archive, I imagine it may take a while to isolate and load it (equivalent) before anything else starts happening. There's also a fair amount of integrity checking on a Zip apart from security scanning the individual parts. An interesting side-experiment is to do a virus scan of the unexpanded archive and see how long it takes on your system. (I confirm that just opening [not extracting] the 1.20.1-stable.zip in FIle Explorer is definitely not instantaneous although not particularly lengthy as long as I am expecting to see an empty directory for a time.)

[dada051]

I tried tar and 7-zip. 7z is much more efficient:

[nmoinvaz]

I think that Windows Realtime Protection prevents file handles from closing until after it can scan the contents. That is why the delay for installers at 100% - waiting to close the files.

[bitcrazed]

Thanks @warpdesign & others.

FWIW re. TAR/ZIP archives

Tar and/or compressed/archive files are not signed and can contain arbitrary files. As such, in order to protect users from malicious files, anti-malware tools like Defender have to scan the contents of each file extracted from an archive before it is copied to disk.

This is doubly-so for files copied from potentially untrusted websites, and/or branded with the mark of the web.

This is just one of the reasons we encourage vendors of, apps, tools, SDKs and libraries to package their files in signed installers, rather than tar/zip files: When extracting the contents of a signed installer, Defender et al. can relax a little since the provenance of the installer and its contents are more easily determined.

I am preparing guidance on just this as I type and hope to have it published soon on this site's wiki/docs.

[asklar]

@warpdesign @duke7553 I'd be interested to learn whether you see Defender slowing things down even if you add an exclusion for the folder the file is on?

In Windows Security, click on Manage Settings under Virus & Threat protection settings

Then scroll down to the Exclusions section and click on Add or remove exclusions to add the folder to the excluded list

[warpdesign]

@asklar I updated my issue with timings when the source folder is excluded. It's faster but Windows Defender still appears to be slowing things down. I also re-run all Windows benchmarks with Powershell's Measure-Command.