"checking available disk space" step lasts absurd time

[fabiangreffrath]

Hi,

when upgrading packages in MSYS2 I noticed that the "checking available disk space" step takes an absurd amount of time. Today, it upgraded 9 packages:

Total Download Size:    21.45 MiB
Total Installed Size:  127.10 MiB
Net Upgrade Size:        0.15 MiB

It took several minutes (!) to check if there was enough space for the additional 150 kB on a hard disk that has 77 GB empty space left. The whole step takes about 3/4 of the time for the entire package upgrade.

[abdulbadii]

Exactly the same point I've been about to post !

[pedro2555]

what a bummer. almost a year no response.

[elieux]

IIRC this step is very I/O heavy as it checks the mount point for each file. It'll most likely get slower with the number of (un)installed files rather than their size.

[assertivist]

I still have this problem with the newest msys2 mingw64 pacman, to the point where I avoid system upgrades if at all possible. It sucks when I have to install a new package. Anyone know if there's an upstream issue for this? I doubt that it has to do with this fork of pacman itself.

[epigramx]

What I don't get is why does it do it for each package separately. E.g. it's slow to go from 0/11 to 1/11 and then it still takes time for the rest.
I guess it's a pacman limitation from the porting to windows that doesn't fit well at all with the win32 api.

[fabiangreffrath]

Could this get changed to call GetDiskFreeSpaceExA() on the MSYS2 install directory?

https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-getdiskfreespaceexa

[fabiangreffrath]

There is the CheckSpace option in /etc/pacman.conf that we should probably comment out.

[orlp]

I would like to +1 this issue, still a problem as of today. Several minutes wasted checking disk space.

[marcelotduarte]

from https://github.com/msys2/MINGW-packages/blob/master/ci-build.sh

# reduce time required to install packages by disabling pacman's disk space checking
sed -i 's/^CheckSpace/#CheckSpace/g' /etc/pacman.conf

[krizzodil]

I just experienced a similar/the same problem. In my case it was caused by my disconnected vpn, which uses FortiClient. When I am not connected there are frequent problems of that nature, e.g. the file dialog of firefox when downloading something does not show up until I connect the vpn. (Yeah, I should solve that problem sometime)

[roflcopter4]

Why does Pacman even do this? As far as I'm concerned, it's on the user to determine whether there's enough space to install a package; or at least to ensure there is enough space to allow packages to be updated. It's not as though updates typically use all that much more space. Don't treat me like a helpless baby. If someone didn't have enough space, tried to update anyway, and thereby corrupted their whole installation that's just their just desserts. If they complain they ought to be laughed out of the room. Don't waste my time!

Yes, I'm aware this can be disabled. It should be disabled by default.

[eduarddejong]

It's really a weird problem and definitely an issue that needs to be fixed!
I am facing this issue with MSYS2 on Windows 10 on my desktop pc. The diskspace check really takes a LOT of time indeed, but also comes with a LOT of CPU usage at the same time, even on this very modern AMD Ryzen CPU that I have.

One thing I can tell is that I have multiple drives and multiple partitions in my system, maybe there's kind of a bug that it checks everything instead of the drive it actually needs to install on? (just a wild guess, I have no prove)
Because I also have a laptop with Windows 11 and MSYS2 installed, with only 1 SSD in it, and I haven't noticed the issue there yet. Edit: I have now confirmed that on my laptop it still takes quite long as well, but a bit less extreme and with less CPU usage. So it's a general problem, but it's impact is not equally large on every system.

Extra note:
I am also a big Arch Linux fan. I never have this issue with Pacman on any of my Arch Linux systems. So I believe the issue is really limited to Pacman on MSYS2 only.

[iSLC]

From what I could observe, the increase in CPU usage and duration might come from the Real-time protection feature of Windows. Which to this day I can't seem to be able to disable because it turns itself back on shortly after. If you look at the "Antimalware Service Executable" using task-manager during installation then you'll notice that it becomes quite CPU intensive (depending on how many cores you have, it may only show the % usage of one core from total cores i.e. if you have 4 then something like ~25%).

I just disable this whenever I run the update or install something. Actually whenever I need to do something file-intensive. And it should complete much faster.

[epigramx]

"Hot" take on pacman design (not really): they should never have had that option on by default for 2 very good reasons. 1) Most people have space 99.9% of their lives. 2) If you do get an error with space : handle it [programmatically as pacman devs] then in some way or another and if you can't handle it now: the general design might be faulty.

[Photosounder]

"Hot" take on pacman design (not really): they should never have had that option on by default for 2 very good reasons. 1) Most people have space 99.9% of their lives. 2) If you do get an error with space : handle it [programmatically as pacman devs] then in some way or another and if you can't handle it now: the general design might be faulty.

Or simpler yet, only check if there's less than 1 GB available on the drive. Even then it's hard to see why it would take this long, you can check the disk space once and that's done, so if the problem is checking the package size then maybe something quicker and less accurate could be used.

[epigramx]

The existence of the option itself is exposing faulty design by the pacman devs. If it can be turned off on choice why even accept it as a default when it creates more wasting time than not for most people? I'd just remove it from the code entirely and have ways to recover on error like other package managers appear to do.

[drankinatty]

I'll also add a few facts. I've used Arch since 2009. Never an issue with pacman update speed. MinGW64 has a huge problem that begins with "checking available disk space". I've used MinGW64 for several years now and I am extremely happy with it -- except for the pacman -Syu time.

It isn't a disk space issue either. I have 1.5T free on a 2T drive with Win10. I have watched this slow "checking available disk space" issue. I've correlated the Task Manager processes running at the same time to see if there was any distinguishable relationship with a windows process. At first I thought Windows Defender may be the issue, scanning the updates before allowing install. To check I disabled "Real Time Protection" to see if the slow install time improved. It did Not.

I wish I could be of further help, but I don't spend much time on Windows. The only thing I can say without any fear of being wrong is that pacman update in MinGW64 takes 10X (Ten Times) or more longer to complete than when running the same updates on Arch.

If it were just a small factor or two, I'd just chock it up to being windows and forget about, but it is a hell-of-a-problem if you need to do something in MinGW64 after starting an update. I just did an update of 116 packages and it took more than 27 minutes to complete. It would have taken 27 seconds on Arch.

I wish you the best in sorting this out, but it is rapidly becoming a usability issue. Think of it this way. When it becomes a big enough problem, that I'll search and locate the bug (this issue) and then take the time to author a comment with hopes that it helps -- it's become a big issue...

[iSLC]

I haven't tried this but I'm guessing that one could add the msys2 folder as an exception to windows defender via a command line:

Add-MpPreference -ExclusionPath "C:\msys64"

I guess I'll see if it works next time I have something to update.

[drankinatty]

I'll do the same and report back -- but I suspect disabling "Real Time Protection" (which to my understanding is the hook that calls windows defender on filesystem changes) should do the same thing. That said, I may be completely wrong there -- wouldn't be the first time on windows....

[jdpipe]

A bit update to my MSYS2 install required ~300 packages to be updates. The space takes more time than any other part of it... about half an hour or more... a bit ridiculous, and it won't be helping us to attract/retain users.

A suggestion here: For packages being downloaded, we also download checksums for the file integrity, as I understand it? Do we not, or can we not, also at that time, download a pre-calculated value for the installed size of each package? If we are trusting the server for checksums, we can also trust the server for installed sizes, surely?

Would there not be some sufficiently accurate way to quickly address this most of the time by simply using these pre-calculated installed sizes instead of, as I presume it is doing, parsing each downloaded .xz file to determine its inflated file size total? Seems like there must be a LOT of I/O going on here to calculate these totals, and it's very inefficient.

[jdpipe]

Indeed it appears from the code that there is no use of precalculated installed sizes:

MSYS2-pacman/lib/libalpm/diskspace.c

Line 281 in 72bebac

static int calculate_installed_size(alpm_handle_t *handle,

The calculation takes into account the block size of the device... but it would still not be too hard to calculate this size at least for the most common settings...?

[drankinatty]

Disabling "Real Time Protection" did not do much to improve the "checking available disk space" time. I can confirm that Windows Defender is triggered regardless of whether RTP is active. I watched the "Detailed" processes behavior in Task Manager and the virus scanner it triggered during the "checking available disk space" time.

I'm not sure what else a user could do to improve this issue, if anything. It looks like it will need the creative wizardry of the Msys2 devs to find a way to improve the time. To me it seems if it is a package size calculation, Windows already keeps track of the Available Disk Space and has that available. If looping pacman query package for the total install size is the problem, then maybe keeping a simple look-up table on the download mirror with the installed package sizes that could be queried by the local pacman instance to get the total installed size in a single Ajax or curl call could eliminate that part of the equation.

Just a left-field thought, I don't know enough about the issue to offer anything concrete, but would like to see this time cut down if possible. It is the only "cringe inducing" part of using Msys2 which is otherwise an absolute joy to use.

[Nyxeka]

On Windows 10, I found this is fixable by going to My Computer > in explorer, and taking a quick scroll through each of your HDD's (open each one up to browse contents) to make sure they're not in sleep mode. As soon as one of them jumped out of sleep mode, it completed instantly.

[fabiangreffrath]

Thank you for finding that workaround, but (1) a workaround shouldn't be necessary for something as trivial as checking disk space and (2) I have >10 network mounts in "My Computer" and I'm not sure if the concept of "device sleep mode" holds up here.

[drankinatty]

Yes, and the workaround doesn't explain slowness on desktop drives that do not enter sleep mode. I see this is perhaps another layer of impediment that may impact the disk space checking slowness.

So far, the best work-around I've found is to open windows defender and temporarily disable virus scanning during the update. Then re-enable when the update is through (it will go back and scan the new files, but doesn't impact the check-time or installation). Of course since this is windows, you will have to go though the UAC confirmations to disable checking.

The biggest question is how do other apps check disk space without suffering the major slowdown MSYS2 does? It appears to check space for every package. A fix may be to loop over the packages and simply sum the installed size (or delta from the currently installed package) and then check once for the total size required at the end.

[coz-eduardo-hernandez]

From what I could observe, the increase in CPU usage and duration might come from the Real-time protection feature of Windows. Which to this day I can't seem to be able to disable because it turns itself back on shortly after. If you look at the "Antimalware Service Executable" using task-manager during installation then you'll notice that it becomes quite CPU intensive (depending on how many cores you have, it may only show the % usage of one core from total cores i.e. if you have 4 then something like ~25%).

I just disable this whenever I run the update or install something. Actually whenever I need to do something file-intensive. And it should complete much faster.

This was my experience too. I disabled it just for a space check that had been going on for more than 5 minutes without even a third done, and it completed in seconds