MSBuild server holds task/plugin assembly file handles for the process lifetime (no shadow-copy/unload)

[JanProvaznik]

Summary

The MSBuild server (OutOfProcServerNode, nodemode:8) is a persistent process that stays alive across builds. Today it releases some file handles between/after builds (the current directory and the runtime cache dir), but it never unloads task/plugin/logger/SDK-resolver assemblies, so any managed assembly loaded during a build stays loaded — and its file locked — for the entire lifetime of the server process. This is the classic "node reuse locks my task DLL" behavior, made more impactful by the server being longer-lived and shared across many builds/solutions.

This issue documents the current mechanism, the gap, how Roslyn's compiler server (VBCSCompiler) avoids the same problem, and options for improving it.

What the server does today

Two deliberate handle-release steps exist:

1. Current-directory handle. On Windows a process holds a handle to its CWD, which would block deleting/renaming the just-built folder. After every build and at shutdown the server resets the CWD off the user's folder back to the tools dir:

   • src/Build/BackEnd/Node/OutOfProcServerNode.cs:469 (after each build)
   • src/Build/BackEnd/Node/OutOfProcServerNode.cs:262 (at shutdown)
   • Worker nodes do the same: src/Build/BackEnd/Node/OutOfProcNode.cs:495,722,727

2. Runtime cache dir. Between reuse builds it deletes %TEMP%\MSBuild<pid>-<domain> via FileUtilities.ClearCacheDirectory() (OutOfProcServerNode.cs:129) to avoid stale/growing cache. It also disposes the redirected console writers and restores Console.Out/Console.Error each build (OutOfProcServerNode.cs:449-465).

Project files, imports, RAR reads, etc. are opened and closed, so those are not persistently held. The persistent-handle problem is specifically about loaded managed assemblies.

The gap: loaded assemblies are never released

Task / logger / SDK-resolver / BuildCheck / project-cache assemblies loaded during a build stay loaded and locked for the whole life of the server:

• On .NET: CoreClrAssemblyLoader loads them via MSBuildLoadContext, which extends AssemblyLoadContext but is non-collectible — the constructor never passes isCollectible: true (src/Framework/Loader/MSBuildLoadContext.cs:35-36). Loaded assemblies are cached in static dictionaries that are never cleared (src/Framework/Loader/CoreCLRAssemblyLoader.cs:25-26,113-124; src/Shared/TypeLoader.cs:30,56).
• On .NET Framework: tasks without [LoadInSeparateAppDomain] load into the default AppDomain and stay (src/Shared/TaskLoader.cs:128-134); only separate-AppDomain tasks get unloaded (TaskLoader.cs:168-178).

Net effect: the server has the same locking behavior as a reused worker node, just longer-lived and shared across more builds. The most common pain is rebuilding a solution whose own build output is consumed as a task/analyzer/generator DLL — the server keeps the previous copy locked.

How Roslyn's server (VBCSCompiler) avoids it

• Shadow-copy for analyzers/generators: ShadowCopyAnalyzerAssemblyLoader copies each analyzer DLL (+ dependencies/pdb) to a unique temp directory and loads from the copy, so the original build output is never locked and can be rebuilt while the server runs. On .NET Core it loads copies into a per-directory DirectoryLoadContext; on .NET Framework it's AppDomain + LoadFrom on the copy. Old shadow directories are cleaned up on startup.
• Reference assemblies / source files are read with delete-sharing and metadata is cached by timestamp, so no persistent exclusive locks are held.

Options to improve

Ordered roughly by behavioral risk (lowest first):

1. Shadow-copy task/plugin assemblies (Roslyn's model) — proven, lowest behavioral risk. Scope it to non-SDK / build-output assemblies; keep SDK / WellKnownAssemblyNames in the default ALC (the loader already special-cases those in MSBuildLoadContext.WellKnownAssemblyNames). Cost: copy overhead + care around ALC identity and dependency resolution. Natural hook point: CoreClrAssemblyLoader.LoadUsingPluginContext / MSBuildLoadContext.
2. Collectible ALC per build + unload — releases handles cleanly, but tasks routinely leak static state / event handlers, and any GC-reachable object pins the context, so unload is unreliable; reloading common SDK tasks every build is also a perf hit.
3. Do nothing / document — rely on the out-of-proc task host for tasks that must not lock, and accept parity with existing node-reuse behavior.

Suggested direction: mirror Roslyn — opt-in shadow-copy limited to task/plugin assemblies loaded from outside the SDK/tools dirs, gated behind a ChangeWave / escape hatch, plus a server-reuse test that rebuilds a locked task DLL.

Open questions

• Which concrete scenarios are hurting users most (rebuild-locking of output-as-task DLLs vs. others)?
• Should the fix also cover reused worker nodes (OutOfProcNode) and the task host, or is the server the priority?
• Interaction with existing escape hatches (UseSingleLoadContext, UseCustomLoadContextForDependenciesInToolsDirectory).

[AlesProkop]

MSBuild Grooming: Not a priority. A fix would be a shadow-copy. But for now, we may just add docs to call this out.