Windows Server 2022 builds are taking 4x longer than on Windows Server 2019

[xt0rted]

Description

Over the last 2 weeks I've noticed runs on the windows-2022 image taking 4x longer, and sometimes more than that, than on the windows-2019 image. In all of these instances nothing's changed between runs aside from the version of Windows being used.

One of the examples I'm looking at from about 2 weeks ago took 7 minutes to complete, while a run from a day or two ago took 24 minutes. This increase in time is consistent across all new runs in our org.

In another repo I downgraded our build scripts & vm to Windows Server 2019 and the time went from 19 minutes down to about 5.5 minutes. I'm unable to permanently move back to 2019 though because new builds depend on VS 2022.

While testing something else I ran a simple checkout & build of an empty .net project and the build times for ubuntu-latest (25s) and windows-2019 (1m 12s) were about what I'd expect, while the windows-2022 image clocked in at 8m 14s.

All of this was originally reported to support in ticket 1521042 but I was told to open an issue here instead. That ticket has org/repo names and links to each run.

Virtual environments affected

• Ubuntu 18.04
• Ubuntu 20.04
• macOS 10.15
• macOS 11
• Windows Server 2016
• Windows Server 2019
• Windows Server 2022

Image version and build link

None of the repos are public but these are the Run Ids for each.

Runs in the image:

Run	VM	Version	Time
1860104886	windows-2019	20220207.1	6m 51s
1912616149	windows-2022	20220220.1	23m 59s

From another similar repo:

Run	VM	Version	Time
1912952429	windows-2019	20220223.1	5m 25s
1912601548	windows-2022	20220220.1	19m 8s

Test repo:

Run	VM	Version	Time
1912879771	ubuntu-latest	20220220.1	25s
1912879232	windows-2019	20220223.1	1m 12s
1912864061	windows-2022	20220220.1	8m 14s

Is it regression?

No response

Expected behavior

For run times on par with Windows 2019

Actual behavior

Run times up to 4-5x longer than Windows 2019

Repro steps

Run a .net full framework build on Windows 2019 and 2022, the 2022 runs should take significantly longer.

[xt0rted]

My build times just dropped to 5-10 minutes. All that changed was I downgraded the .net 6 sdk to .net 5. I doubt that's related, but I am seeing times on par with what I used to get.

[miketimofeev]

@xt0rted hi! Is it possible for you to provide the minimal steps to reproduce the issue? We can make some experiments if we have reproducible examples

[xt0rted]

@miketimofeev I can't reliably reproduce this. I see the issue in a couple of my private repos, but it seems to come and go now. It's only happening with the Windows 2022 workflows though, downgrading to 2019 where possible fixes it and the build times remain consistent. I'll keep seeing if there's a way to reproduce this though.

About 2 weeks ago I had a build on windows-latest which was taking anywhere from 6-10 minutes, switched to ubuntu and it dropped to 58 seconds. I know windows takes longer but it shouldn't be that much longer. I can dig up some details on that if you'd like, but the only change made at that point was switching the OS.

[mikhailkoliada]

@xt0rted Hello! Thanks, we will take a look!

[miketimofeev]

@xt0rted I'm afraid we can't proceed with the investigation without a reproducible example. We don't have access to the tickets, I'm afraid.

[xt0rted]

Not sure if it's related, but here's an instance where the Windows 2022 build took 3x longer than the Windows 2019 build https://github.com/xt0rted/dotnet-rimraf/actions/runs/2041805895.

I don't have a Windows 2019 build to compare to for this one, but you can see the Windows 2022 build is significantly longer than the macOS or Ubuntu builds https://github.com/xt0rted/dotnet-run-script/actions/runs/2041939525.

[rilysh]

Can tell that, I having the same issue as it is. Using Windows Server 2022, builds taking quite a longer time than 2019.

[lowlydba]

My Windows 2022 builds are so slow that they're repeatedly timing out and/or having issues provisioning resources. Works fine with 2019.

https://github.com/lowlydba/lowlydba.sqlserver/actions/runs/2458069276