Optimize the file management of GitHub Actions Windows builds (OSOE-156)

[Piedone]

The Windows builds of our GitHub Actions workflows are always a lot slower than the Ubuntu ones for every operation, even checkout (at least 2-3x slower, all the way up to 40x sometimes). Let’s try to make this better by tackling file management-related Windows issues.

• See this thread for a discussion about this with various tips. But this thread is now lost due to GitHub deleting the community forum, see: https://github.com/orgs/community/discussions/30387
• Check if we can configure Windows Defender and if it makes GHA builds faster: Check if under Windows we can configure Windows Defender to exclude devenv.exe, MSBuild.exe, and dotnet.exe from the checked processes, as well as the folder of the checkout directory, and if that makes builds faster.
• Related issues:
  • Windows Server 2022 builds are taking 4x longer than on Windows Server 2019 actions/runner-images#5166
  • extremely slow Network and Disk IO on Windows agent compared to Ubuntu/Mac actions/runner-images#3577 (comment)
  • extremely slow Network/Disk IO on Windows agent compared to Ubuntu/Mac actions/setup-dotnet#260
  • [Bug]: Slow Restores on Windows in GitHub Actions - Signature verification taking a long time NuGet/Home#11548
• Windows with a better filesystem

To be checked

See the attached spreadsheet for measurements.

• Disabling Windows Defender by @0liver
  • This doesn't seem to have any noticeable effect, and especially not a positive one.
• "While there is no ext4 driver for Windows there is WinBtrfs for Btrfs, another major Linux file system. Supposedly it’s better than NTFS though worse than ext4. None of the other major Linux file systems (xfs, f2fs, reiserfs, reiser4) have Windows drivers." by @DAud-IcI
  • Tried WinBtrfs out on this commit, see measurements in the spreadsheet. If anything, it's slower by about 10%. Corresponding GHA commit.
• NTFS settings:
  • "On NTFS playing around with fsutil might be helpful, though I assume the VM maintainers have already did this. If not, should open a PR on actions/virtual-environments and request it." by @DAud-IcI.
    • Needs reboot, so should rather be done in the image. Since rebooting a GitHub-hosted runner doesn’t work (see this run), none of these settings can be changed. There doesn't seem to be a way to run startup scripts or anything like that.
    • Shortnames are explicitly enabled in the GHA image.
    • See this script that I used to set NTFS settings.
  • Similar config recommendations.
• "As for git performance, looking for a different command line client based on libgit2 might help. (actions/checkout@v3 just front-ends normal git) Or perhaps the GitHub CLI which uses a different re-implementation called go-git." by @DAud-IcI
  • The checkout is slower than under Ubuntu (though lately it seems to be "just" at most 50% slower, and usually not too much slower) it's still just 30s-1m out of a 16-30m build. Even if we optimize this a lot, it won't really matter. So, I don't think we should put any effort into reinventing the git wheel, which seems like quite a lot of work anyway.
• Virtual file system with better "lot of small files" support
  • Solid File System was perhaps something like this, though it doesn't seem to be an active project. It seems the successor is CBFS Storage. It can create virtual drives too. It's not open-source though, so we can't include it in our open-source actions (a license is $1500).
  • WSL 2 can mount ext4 drives, including a VHDX file.
    • This is available under Windows 11 Insider Build but can also be installed from the Microsoft Store too. You still need Windows 11, though (the indicated minimum requirement of "Windows 10 version 22000 or higher" is actually Windows 11).
    • Here's a blog post about how to create a VHDX file to begin with and how to mount it. Here's an answer about the same thing with CLI samples.
    • setup-wsl action. Note that this is only WSL 1 because the currently used Azure VMs don't support nested virtualization.
    • Comment about manual setup but the WSL enable shouldn't be necessary anymore. PS-based installation is here.
    • Server installation for WSL.
    • See this script with basics of WSL drive mounting. However, this can be nicer by using New-VHD and Mount-VHD, which need Hyper-V enabled (which is necessary for WSL 2 anyway).
    • WSL 2 is not possible to use under GitHub-hosted runners.
• Other performance increase options?
  • Nobody seems to care much about .NET build perf on GHA.
  • We've tried NuGet and NPM caching before without benefits.
  • BuildJet seems interesting, but the thing they don't offer is Windows builds.
  • No other ideas...

Measurements.xlsx

Conclusion

Summary of findings:

• Disabling Windows Defender doesn't help.
• Using a pluggable virtual file system (WinBtrfs) is slower.
• Due to not being able to reboot, we can't change NTFS settings.
• Git performance is not a big issue.
• Perhaps ext4 would be better as a virtual file system, but it's impossible to use on GitHub-hosted runners.

Next steps:

• Tuning NTFS settings can work with self-hosted runners, using our own image. Mounting an ext4 virtual drive with WLS 2 too, for that additionally we need to use an Azure VM type that allows virtualization. I've written scripts for these (or the basics of these) that may be used during packing our own images.
• BuildJet sounds promising, and I'm testing it, but it's only for Ubuntu.

However, the best option, really, is to just use Ubuntu if possible. (It is possible for all of our Orchard Core projects, but for OSOCE, specifically, we need Windows builds to verify cross-platform behavior).

[0liver]

In a CI environment it should be feasible to disable Windows Defender altogether, no?

E.g. from an administrative PS shell (source):

Set-MpPreference -DisableRealtimeMonitoring $true

Here's a serverfault answer that does an additional step; might be helpful.

[Piedone]

Perhaps not completely disable, I'd err on the safe side by using only the above-mentioned configuration.

[Piedone]

Well, this is disappointing, also CC @DAud-IcI and @BenedekFarkas. See my findings and conclusions above. Do you have any other suggestions, perhaps? I'm out of ideas.

It seems that any Windows tuning option we have left necessitates using self-hosted runners.

[sarahelsaig]

Well, that's depressing. I don't have other ideas, sorry.

By the way the WSL angle isn't really clear to me. Isn't WSL just testing on Linux with extra steps? WSL1 is basically reverse Wine and WSL2 is a nice frontend for Hyper-V running a real Linux kernel. So it seems to me that going through that you wouldn't really test a Windows application and wouldn't detect any Windows-specific problems.

[Piedone]

It is :(. I'd have used WSL only to mount an ext4 virtual drive as a normal Windows drive (which doesn't seem to be possible from under Windows otherwise, but which is a fully supported scenario with WSL 2) and have the working directory there. So the build and test execution would all happen under Windows, just using ext4 as storage. (Hoping that it provides a performance gain over NTFS with our usage patterns.) I ultimately did that with WinBtrfs too. The results I got there hint that perhaps ext4 wouldn't offer advantages either. (My suspicion is that this is due to the old hardware being used for runners.)

[BenedekFarkas]

I don't have anything to add, unfortunately. Just that I remember that we had a problem with attached drives on Vultr too and build performance tanked due to slow I/O, so we had to switch larger VMs just for the sake of the larger NVMe storage, not the CPU/RAM...

[Piedone]

BTW I briefly thought about finding out which Azure DC the GitHub-hosted runners are located and trying to attach drives to them that we create in Azure. But that sounds hopelessly hackish (and until we have everything working and we can try it out, it's only hope that it'll actually provide any benefits) but at that point, I'd again just focus on creating self-hosted runners instead.

[Piedone]

I put some more thought and research into this whole question, but nothing, really.

[Piedone]

Nothing worked here, unfortunately. While I'm still testing BuildJet and larger GH-hosted runners, optimizing Windows builds didn't yield any results, so closing this.