Concurrency bug fix - BuildManager instances acquire its own BuildTelemetry instance (#8444) #8561
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…t#8444) Fixes https://devdiv.visualstudio.com/DevDiv/_workitems/edit/1708215 Context In VS there are multiple instances of BuildManager called asynchronously. For DTB and normal build and maybe other which I have not identified yet. Changes Made BuildManager instances acquire its own BuildTelemetry instance as oppose to sharing single BuildTelemetry instance in non thread safe manner. Testing Locally # Conflicts: # src/Build/BackEnd/Client/MSBuildClient.cs - resolved with minimal and safe approach
Our CI builds fails because of bug NuGet/Home#12373. It is fixed in NuGet/NuGet.Client#5010. We are waiting for it to flow to CI machines. Meanwhile this PR applies a workaround. Note: This PR needs to be reverted once it happens.
rainersigwald
approved these changes
Mar 14, 2023
rokonec
added a commit
that referenced
this pull request
Mar 27, 2023
* Concurrency bug fix - BuildManager instances acquire its own BuildTelemetry instance (#8561) * BuildManager instances acquire its own BuildTelemetry instance (#8444) Fixes https://devdiv.visualstudio.com/DevDiv/_workitems/edit/1708215 Context In VS there are multiple instances of BuildManager called asynchronously. For DTB and normal build and maybe other which I have not identified yet. Changes Made BuildManager instances acquire its own BuildTelemetry instance as oppose to sharing single BuildTelemetry instance in non thread safe manner. Testing Locally # Conflicts: # src/Build/BackEnd/Client/MSBuildClient.cs - resolved with minimal and safe approach * Bumping version * Turn off static graph restore. (#8498) Our CI builds fails because of bug NuGet/Home#12373. It is fixed in NuGet/NuGet.Client#5010. We are waiting for it to flow to CI machines. Meanwhile this PR applies a workaround. Note: This PR needs to be reverted once it happens. --------- Co-authored-by: AR-May <67507805+AR-May@users.noreply.github.com> * Use AutoResetEvent as oppose to ManualResetEventSlim (#8575) Summary Customer, mainly internal like XStore, with huge repos, using msbuild /graph /bl on powerful development and build computers, might experience 15x plus regression in evaluation time. It has been identified as performance bug in our logging event pub/sub mechanism. When ingest queue reaches its bound, at .net472 ManualResetEventSlim causes way too many thread.Yields flooding the system with thread context switches. This hypothesis has been verified by PerfMon perfcounter System.ContextSwitches. Alhougt counterintuitive, AutoResetEvent , ManualResetEvent or even SpinLocking produced better behavior and with those the issue no longer reproduce. Customer Impact In case of XStore it was about 7 minutes in build time. Regression? Yes, introduced in VS 17.4. Testing Manual validation by @rokonec and automated tests. Using local repro to verify changes has fixed it. Risk Low Note It effect only VS MSBuild.exe. In dotnet build ManualResetEventSlim works better. --------- Co-authored-by: Roman Konecny <rokonecn@microsoft.com> Co-authored-by: AR-May <67507805+AR-May@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Original implementation did not expect multiple instances of BuildManager called concurrently. But in VS DTB and normal build are run concurrently.
This is backported from main PR #8444
Customer Impact
In rare cases Dictionary data structure is corrupted and can cause infinite loop. This affect only VS scenarios.
It is currently #7 ranked VS hang issue.
Regression?
Yes, introduced in VS 17.4.
Testing
Manual validation by @rokonec and automated tests. Additionally it has been in bleeding edge VS for about three weeks.
Risk
Low
Note
Has been already backported to 17.5
Contains fix for infra which disables nuget static graph restore (otherwise CI pipeline is failing)