dotnet build hang and running dotnet.exe processes cannot be killed

[NinjaCross]

The bug

This is similar to dotnet/sdk#9487, but i believe it's not exactly the same.
With NET 6.0.100, the build often hang and there is no way to kill the dotnet.exe left behind.
I also cannot inspect or create a dump of the hanging process (even with admin privileges), since I receive "access denied" error messages.

The real problem is that dotnet.exe lock files, which cannot be freed in any way.
Once this happens, the only solution is to reboot Windows.

Tried and failed workarounds

It's unclear to me the root cause, and I tried the following workarounds, but without success:

1. Disable any form of multi-process or parallel build (in both Rider and Visual Studio 2019)
2. Added <UseRazorBuildServer>false</UseRazorBuildServer> to all my projects using a global Directory.Build.props file, as suggested
3. Execute dotnet build-server shutdown (with admin privileges)
4. Killing the hanging processess. As you can see, even with admin processess, it doesn't work

C:\Windows\system32>taskkill /im dotnet.exe /F
ERROR: The process "dotnet.exe" with PID 19788 could not be terminated.
Reason: There is no running instance of the task.
ERROR: The process "dotnet.exe" with PID 17188 could not be terminated.
Reason: There is no running instance of the task.

To Reproduce

It seems somehow correlated to the presence of Razor components or libraries, but at this point this is more an intuition than a real hypothesis.
Since I don't see a clear pattern, and I cannot provide the sources due to NDA reasons, I can't provide more insights, sorry.

Further technical details

The problem happens with both VS 2019 and Rider.

This is the output of dotnet --info

.NET SDK (reflecting any global.json):
 Version:   6.0.100
 Commit:    9e8b04bbff

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.19042
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\6.0.100\

Host (useful for support):
  Version: 6.0.0
  Commit:  4822e3c3aa

.NET SDKs installed:
  2.2.207 [C:\Program Files\dotnet\sdk]
  3.1.408 [C:\Program Files\dotnet\sdk]
  5.0.202 [C:\Program Files\dotnet\sdk]
  5.0.403 [C:\Program Files\dotnet\sdk]
  6.0.100 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.All 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.14 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.20 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.21 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 6.0.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.30 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.14 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.20 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.21 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 6.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.11 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.14 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.21 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.5 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.12 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 6.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

These are the few infos I could get.

[NinjaCross]

Any news on this ?
I had to uninstall net. 6 in ALL the machines of our team in order to make them compile again :(

[marcpopMSFT]

@mkArtakMSFT can you have someone take a look given the comment that including Razor is leading to this repro?

[mkArtakMSFT]

@rainersigwald can you please loop into the right folks? This looks like MSBuild issue. We've reviewed this and it doesn't seem Razor related.

[rainersigwald]

@NinjaCross If you turn MSBuild file logging, does it give any clues about what is happening when the process hangs?

In general, I don't know of any problems that make processes undebuggable/unkillable; that is quite surprising. I don't know how much progress we'll be able to make without a repro or at least detailed logging.

[NinjaCross]

@NinjaCross If you turn MSBuild file logging, does it give any clues about what is happening when the process hangs?

In general, I don't know of any problems that make processes undebuggable/unkillable; that is quite surprising. I don't know how much progress we'll be able to make without a repro or at least detailed logging.

@rainersigwald i didn't try to enable the logging, sorry.
The problem seems to be solved after having:

• uninstalled net 6 SDK
• disabled multi-process build parallellism

Of course, this is just a workaround, and it bothers me the fact that I had to disable multi-process build parallellism, since this greatly increase the build time of my projects :(

[NinjaCross]

During the last 30 days or so, the problem never occurred again, so I assume it has been somehow solved (whatever was the cause). I'm closing the issue.