Build performance in February 2022

[RikkiGibson]

Related to #50678

It's already been over a year since we took a measurement here. This graph shows the average run time for roslyn-ci jobs queued for main (i.e. jobs expected to succeed). We are about 7 minutes slower than our measurement last year.

Correctness_Build is definitely the long pole now, probably because we now use a bootstrap compiler built in debug mode to check if we hit any assertions. Note that the label on Correctness_Build is misleading. The runtime is actually 1hr03m33s, but it displays as 03:33.00. It's not clear if there's anything we can do to speed up the debug compiler.

The tests have relatively low queue times because they still use "Microsoft-hosted agents", while the builds use the 1ES pool. The 1ES pool appears to have slower queue times than the old dotnet-eng pools had a year ago. We could consider switching the builds to use the Microsoft-hosted agents again, but it doesn't matter right now because Correctness_Build is still slower.

Also, I recorded similar info for a similar date range for roslyn-integration-ci. It's slower than roslyn-ci, so perhaps if we were to try and improve anything about "how fast can we merge PRs", we would want to focus on the reliability and run time of integration-ci.

[jmarolf]

Correctness_Build does a lot right? it also runs analyzers? can we enumerate some of the work that Correctness_Build does an potentially split it out into a different queue?

[jmarolf]

Someday soon, a lot of testing that can only be done on integration machines could be moved to LSP tests where we just send/receive JSON data over a real workspace server process. This would be much faster that our integration tests and be parallelizable.

In theory, running analyzers is parallelizable as each compilation could run with analyzers on a separate machine.

I would still like a more granular work of what is most expensive in our correctness build though. Maybe build-boss is what is taking up all the time. Hard to optimize without that information

[CyrusNajmabadi]

All that said, integration tests are still the long pole here, and if we want to spend effort on reducing CI runtime, that's probably where we will want to start. Those tests are not parallelized, as far as I know.

Yes. Each integration test would likely be runnable on different machines. we'd then be as slow as hte single longest test.

[CyrusNajmabadi]

I'm not aware of the dotnet org doing distributed builds,

Note: wrt to distributed-builds. I'm not really saying that an individual build itself is spread over many machines. But rather a single machine can do the build, and then the built binaries are distributed to dependent machines (or, rather, the dependent machines pull those build results to them). That way we're not wasting time building the same things over and over again on dozens of machines.

[jmarolf]

This is already how our existing CI system works. @RikkiGibson did the work to build on one set of machines but then distribute tests across N number of machines using a system called helix.

We currently have Build_Unix_Debug that builds roslyn for unix and then Test_Linux_Debug, Test_Linux_Debug_Single_Machine, and Test_macOS_Debug that all consume those build artifacts and distribute it to N number of machines using helix.

Here is an overview of build machines and the test runs that consume their outputs. Unless the test run ends in _Single_Machine the test run is being split across N number of machines.

• Build_Unix_Debug
  • Test_Linux_Debug
  • Test_Linux_Debug_Single_Machine
  • Test_macOS_Debug
• Build_Windows_Debug
  • Test_Windows_Desktop_Debug_32
  • Test_Windows_Desktop_Debug_64
  • Test_Windows_CoreClr_Debug
  • Test_Windows_CoreClr_Debug_Single_Machine
  • Test_Windows_CoreClr_IOperation_Debug
• Build_Windows_Release
  • Test_Windows_Desktop_Release_32
  • Test_Windows_Desktop_Spanish_Release_64
  • Test_Windows_Desktop_Release_64
  • Test_Windows_CoreClr_Release

We do have integration runs that still do a full build and we could update these to instead consume artifacts from Build_Windows_Debug or Build_Windows_Release where appropriate.

• roslyn-integration-ci
  • VS_Integration_Debug_32
  • VS_Integration_Debug_64
  • VS_Integration_Release_32
  • VS_Integration_Release_64
• roslyn-integration-corehost
  • VS_Integration_CoreHost_Debug

This leaves the following bespoke runs

• Source-Build: verifies we can still build entirely from source (for linux distributions)
• Correctness_Determinism: special compiler build that ensures when building roslyn with roslyn the same inputs get the same outputs
• Correctness_Build: Does a lot of things
  • Builds roslyn with the bootstrap compiler
  • Runs all analyzers
  • Examines all of our build logs to ensure we aren't doing double read or double writes
• Correctness_Rebuild: Verifies that we can re-construct a build using embedded sourcelink info
• Correctness_TodoCheck: Quick check to Validate TODO/PROTOTYPE comments are not present in non-feature branches

All of these run except for Correctness_Build is verifying a particular thing and that thing is typically compilation itself under certain scenarios so I don't think it can be elided. Correctness_Build is the one thing left to me that "does a lot of stuff" that could be split out to other machines.

As with anything, splitting things out technically reduces test coverage because now we aren't doing as many things at once but I really want us to get out of the habit of test environment complexity being a virtue. Having a separate, analyzers only run seems like a good way to improve throughput.

[RikkiGibson]

The next thing I would want to try doing is something like:

1. make the "test artifact" available to integration tests. I think it would be best to just include all the unit tests and integration tests in a single artifact but would certainly be willing to experiment when we reach the point where download time is the limiting factor.
   • I'm also not sure whether it would be better to stick it all into 1 CI definition--this is what runtime does--or do further splitting. Most likely, it would be better to bring everything into 1 CI definition, then on the GitHub side we could consider being more granular about which jobs are required to merge--that's if we want the ability to temporarily make the integration tests optional when a severe reliability issue is introduced, for example. It's worth experimenting with whether AzDO build "stages" are represented in GitHub, which could make this simpler to maintain.
2. figure out how to partition and distribute the integration tests execution over Helix.