Build performance in January 2021 vs December 2020

[RikkiGibson]

Tagging @dotnet/roslyn-infrastructure

It's been a week since #50446 has merged and we have a bit of data. A few things to note:

1. Helix build perf has been quite a bit better in mid-January than in early-mid December. This could be related to us moving back to regular "non-holiday" usage patterns, but it could also be that the data is skewed by some outlier runs. We should continue to track it and maybe come up with some more thorough stats on the times than just the averages over a date range.

2. Currently Helix is outperforming AzDo with ~17:30 runtime for Helix vs ~21:00 for AzDo on average.

3. Spanish seems to be taking an improvement from running in Release mode Run Spanish tests in release mode #50354 in January vs Debug mode in December.

Before (Build_Windows_Debug and Build_Windows_Release both use Helix):

After: (Build_Windows_Debug uses Helix but Build_Windows_Release uses AzDo agents):

[RikkiGibson]

At the beginning of this week we moved our test runs to Helix, so I've gathered a bit more data to try and help us compare the performance before and after the change.

Before:

After:

A few things to note:

1. The average is somewhat lower--53min this week, and 1hr 4min last week. However, Helix seems more variable in the duration of test runs. So it would be helpful to get some stats on the variability here.
2. We don't yet have the full breakdown of the start delay for Helix. What you see in red here is how long did AzDO take to prepare a build agent in order to queue the Helix job. Once the Helix job is queued there is another, currently unknown delay before the job actually starts. Sometimes this delay is rather long--depending on the queue. I will look into how to determine this delay and work it into the graph.
3. Many test runs are faster than before, but others are slower--particularly CoreCLR on Windows and Linux. CoreCLR on Mac is quite a bit faster, though, which is probably due to the high availability of the Mac Helix queue.
4. The Windows Debug and Release build are still split across AzDO and the Helix build queue. They seem to be within about the margin of error of each other this week. Changes are coming to the Helix build queue which will save a few minutes on those builds, so we will likely move back in order to take advantage of that.

[jaredpar]

What is our current partitioning scheme for Helix? Have we tried increasing the number of partitions to see if it would drive down test time?

[RikkiGibson]

Currently a Helix partition is ~500 tests.

I am honestly not sure if more work items is the way. In production I tend to see limited numbers of work items running concurrently. In the Spanish queue, I haven't seen more than 16 concurrent. I also don't know, for example, how many work items a test agent can run concurrently--is it one? is it more than that? etc.

I have been wondering if instead we should make sure we're fully utilizing the machine when the work item is running. I am assuming there is a non-negligible setup time for each work item. For example, I wonder if Helix copies the correlation payload per work item because it doesn't trust you to not modify it.

One way we could better utilize the machine is to try running multiple partitions for a single assembly within a work item. But the way that I'm curious to try is to enable parallel test collections wherever possible. I speculate that 90% of Roslyn tests can be made to do this without too much work and the rest can be exempted. Then, when we partition, we group them by whether the collections in the partition are parallelizable, and if so, we make the partition bigger to compensate.

To parallelize the emit tests, for example, I think we can just give each test collection its own temp folder and things will work, but I don't yet know for sure.

With this change I also have in mind the local developer who just wants to run tests on a single assembly. I feel like dotnet test src/Compilers/CSharp/Test/Semantic should just work for that if possible. I feel like nobody remembers to use RunTests for this in part because you have to keep track of the project path as well as the output path, and you have to remember to build separately before running the tests, etc.

[RikkiGibson]

All that said, though, I think my first priority is to try and measure queue delay (hopefully meaningfully) for Helix jobs. Then we will hopefully have a sense of proportion about what to try and address first.

[jaredpar]

I haven't seen more than 16 concurrent. I also don't know, for example, how many work items a test agent can run concurrently--is it one?

I'm unsure as well but it's possible for Helix to also break up the work amongst different agents. Back when I was testing this as a CLI tool I found that having partitions of size ~50 produced the best results. Basically I just kept toggling the number, queueing runs and seeing what worked. I admit I didn't dive as much into the mechanism of how it was all being scheduled though.

But the way that I'm curious to try is to enable parallel test collections wherever possible. I speculate that 90% of Roslyn tests can be made to do this without too much work and the rest can be exempted.

I've always been skeptical of this approach. Parallelizing test code is no different than parallelizing product code. It's really hard. Also while the compiler test can often be parallelized with some work (the underlying system is designed to support it), the IDE layer often cannot because of the thread affinity so much of the editor code maintains.

Helix should offer us the out card here. It should allow for efficient process level parallelization of our tests here.

[RikkiGibson]

Some changes:

1. Looks like the move from F: to D: in the Windows agents has significantly improved the build duration. The average start delay is somewhat longer this week, but this can maybe be chalked up to random variation.
2. We tried setting /p:Features=run-nullable-analysis=never in the Build_Windows_Release job, but it looks like any perf improvement from that is probably negligible.
3. I added standard error bars, more or less experimentally. Some of these bars are not really meaningful because the data doesn't follow a normal distribution--notably, the startDelays for the Helix test jobs, which are either very small or very large (because we hit our limit on parallel jobs, and have to wait for one to finish before the next job can start). To follow up on that here are some distribution charts:

This next chart is relevant to the Build_Windows_Debug and Build_Windows_Release jobs:

It seems there's a quite wide, pretty evenly distributed set of start delays for these jobs.

The next two charts are relevant to basically all the Windows Helix jobs. The second chart is only for the Test_Windows_CoreClr_Debug and Test_Windows_CoreClr_Release jobs. The similarity in the distribution and magnitude makes it seem these jobs are probably all dominated by queue time.

[RikkiGibson]

I also thought this was interesting. This graph of Windows build times shows a very clear delineation of "when restore was fast" and "when restore was slow".