Skip to content

Enable MSBuild server by default in MSBuildForwardingApp#55231

Merged
baronfel merged 2 commits into
dotnet:mainfrom
JanProvaznik:janprovaznik/msbuild-server-default-on
Jul 14, 2026
Merged

Enable MSBuild server by default in MSBuildForwardingApp#55231
baronfel merged 2 commits into
dotnet:mainfrom
JanProvaznik:janprovaznik/msbuild-server-default-on

Conversation

@JanProvaznik

@JanProvaznik JanProvaznik commented Jul 10, 2026

Copy link
Copy Markdown
Member

Summary

  1. in MSBuild server dogfooding and release plan msbuild#14311
    Makes the MSBuild server enabled by default in MSBuildForwardingApp. When the user hasn't specified anything about the MSBuild server, it is now turned on.

Changes

  • DOTNET_CLI_USE_MSBUILD_SERVER now defaults to true (was false).
  • MSBUILDUSESERVER=1 is only forced on when the user hasn't already set MSBUILDUSESERVER themselves, so an explicit opt-out (MSBUILDUSESERVER=0) or DOTNET_CLI_USE_MSBUILD_SERVER=false is still respected.
  • Added ItEnablesMSBuildServerByDefault test verifying the default-on behavior.

Behavior change

⚠️ This is a user-visible behavior change: the MSBuild server is now on by default. Opt out via DOTNET_CLI_USE_MSBUILD_SERVER=false or MSBUILDUSESERVER=0.

Testing

Ran GivenMsbuildForwardingApp locally against a full Debug build; the new test passes and existing server-related tests pass.

@JanProvaznik

Copy link
Copy Markdown
Member Author

Test failure analysis: real bug, already fixed upstream, blocked on an MSBuild version lag

The CI test failures on this PR (e.g. Microsoft.TemplateEngine.Authoring.Tasks.IntegrationTests LocalizeTemplateTests / ValidateTemplatesTests, plus dotnet-watch.Tests) are real, not flaky, and are a direct consequence of enabling the MSBuild server by default in MSBuildForwardingAppWithoutLogging.

Root cause

With MSBUILDUSESERVER=1, TerminalLogger auto-detection runs in the persistent server node and inspects the node's stdout instead of the client's. When the client's stdout is redirected (as it is when tests capture dotnet build output), the server still emits terminal-logger ANSI output. So assertions like HaveStdOutContaining("Build FAILED.") fail because the output is the TL-formatted failed with 1 error(s) instead of the classic console-logger Build FAILED..

This is exactly dotnet/msbuild#14077 ("Fix TerminalLogger auto-detection under MSBuild Server", fixes dotnet/msbuild#13940), merged 2026-06-17, which explicitly calls out the Windows > file.txt (redirected) case.

Reproduction (pure CLI, no test harness)

Using the SDK built from this branch, a redirected dotnet build of a failing project:

Scenario Exit ANSI codes Build FAILED.
MSBUILDUSESERVER=0 (server off) 1 No Yes
server on (this PR's default) 1 Yes No

Deterministic, reproduces without any test framework. Verified end-to-end that the daily .NET 11 SDK (11.0.100-preview.7.26360.102, MSBuild 18.10.0-1.26360.102, post-#14077) makes server-on behave identically to server-off (plain output, no ANSI) — so the fix resolves it.

Why CI still fails (the blocker)

The SDK builds against / redistributes MSBuild via MicrosoftBuildVersion, which resolves from eng/Version.Details.props:

MicrosoftBuildPackageVersion = 18.9.0-preview-26311-113

That MSBuild is source-dated 2026-06-11 — six days before the fix. Note this is out of sync with eng/Version.Details.xml, which was VMR-source-updated to Microsoft.Build 18.9.0-preview-26325-102 (2026-06-25, which does contain #14077). Both files are identical on origin/main, so this is not a stale-branch issue on my side — main's redistributed MSBuild simply predates the fix.

What would unblock it

The SDK needs a post-#14077 MSBuild (>= 18.9.0-preview-26325-102; confirmed the fix is present in 26325 and daily 26360). Either wait for the dependency flow that bumps MicrosoftBuildPackageVersion in eng/Version.Details.props, or bump it manually as part of this PR.

Happy to do whichever the maintainers prefer.

@JanProvaznik

Copy link
Copy Markdown
Member Author

Update: MSBuild bump to 26325 fixed the Windows TL failures; residual failures are a separate Unix server bug

Pushed 31ecd76 bumping the redistributed MSBuild to 18.9.0-preview-26325-102 (which contains dotnet/msbuild#14077). Result on the next CI run (build 1503449): failed work items went 7 → 3, and all Windows legs are now green (TestBuild: windows 214/214, FullFramework: windows 152/152) — confirming the TerminalLogger-under-server fix resolves the Windows failures.

Remaining 3 failures (all Unix)

Test OS Symptom Verdict
TemplateEngine.Authoring.Tasks…CanRunValidateTask_OnInfo linux dotnet build prints Build succeeded. 0 Error(s) but returns exit code 1 real — Unix server bug
dotnet-watch…EvaluationTests.ProjectReferences_OneLevel macOS dotnet msbuild /t:GenerateWatchList restores successfully (no errors) but the process exits with code 1, so watch reports "Error(s) finding watch items" and targetsResult is null real — same Unix server bug
dotnet.Tests…ItReturnsNullWhenCommandNameDoesNotExistInProjectTools linux pure unit test (CommandResolver.Resolve), no MSBuild/server involved unrelated flake (0.06%)

The first two are the same root cause, and it is not the TerminalLogger bug: with the server enabled on Unix, an MSBuild invocation that succeeds (0 errors, Build succeeded.) intermittently returns exit code 1. It hit a different test each run (previously CanRunTask, now CanRunValidateTask_OnInfo), which points to an intermittent exit-code race in the server node on Unix rather than a deterministic failure. I could not find an existing MSBuild fix for it, and it reproduces only on Unix (I'm on Windows, so I can't repro/fix it locally).

Net

  • Windows: green with MSBuild ≥ 26325.
  • Unix: blocked on a separate, still-unfixed MSBuild-server "exit code 1 on successful build" issue that enabling the server by default surfaces.

This looks like it needs an upstream MSBuild fix (or confirmation that these are acceptable/known-flaky) before the server can be defaulted on for Unix. Flagging for maintainer guidance.

@JanProvaznik

Copy link
Copy Markdown
Member Author

Local repro investigation of the residual Unix failures (WSL Ubuntu 24.04)

To determine whether a newer MSBuild would fix the two Unix "exit code 1 on a successful build" failures, I tried to reproduce them locally with the exact SDK CI uses (11.0.100-preview.7.26325.102, MSBuild 18.9.0-preview-26325-102) and the daily (26360, MSBuild 18.10.0):

  • Sequential dotnet build of a message-logging succeeding project under the server: 0 failures.
  • Concurrent (8×6) dotnet build under the shared server: 0 failures.
  • Faithful replication of the dotnet-watch invocation (dotnet msbuild /restore /t:GenerateWatchList /p:DesignTimeBuild=true with the real DotNetWatch.targets + DotNetWatchTasks.dll, 2-project reference graph), sequential and concurrent, on 26325: 0 failures in 130+ invocations.

So the exit-code race is rare and timing/parallelism-dependent — it did not reproduce once locally, yet CI hits ~1 test per run (a different test each time). I therefore can't confirm locally whether bumping MSBuild further (e.g. to 18.10.0-1.26360.102) fixes it, and a 18.9→18.10 skew against the rest of the 26325 dependency set carries its own risk, so I did not make that change.

Summary for maintainers

  • The 26325 MSBuild bump fixes the deterministic Windows TerminalLogger failures (Windows now green).
  • The remaining Unix failures are a rare, intermittent MSBuild-server "exit code 1 on success" race (CanRunValidateTask_OnInfo, dotnet-watch ProjectReferences_OneLevel), plus one unrelated flaky unit test (ItReturnsNullWhenCommandNameDoesNotExistInProjectTools, no MSBuild involved).
  • These look like an upstream MSBuild-server correctness issue surfaced by defaulting the server on, rather than anything fixable in the SDK. Guidance welcome on whether to (a) pursue an upstream fix, (b) treat as known-flaky, or (c) gate the default differently.

@JanProvaznik
JanProvaznik force-pushed the janprovaznik/msbuild-server-default-on branch from 31ecd76 to d6d5ade Compare July 13, 2026 08:13
@JanProvaznik

Copy link
Copy Markdown
Member Author

Update: rebased on main + bumped MSBuild to 26360 (includes the exit-code fix #14251)

Rebased onto current main and confirmed main's MSBuild (18.10.0-preview-26359-117, VMR 2026-07-09) does not yet contain the exit-code race fix — dotnet/msbuild#14251 (fixes #14172) back-flowed into the VMR after that build.

Issue #14172 is an exact match for the two residual Unix failures: on Linux with the MSBuild Server, a successful build intermittently exits 1 with Build succeeded. dropped, and it only reproduces under CPU load (which is why my earlier idle WSL stress runs found nothing, and why it shows up on busy CI Helix agents). The issue even names SDK 26325.

Changes in this push

  • eng/Version.Details.props: bump Microsoft.Build / Microsoft.Build.Localization / Microsoft.Build.TaskAuthoring.Analyzer from 18.10.0-preview-26359-11718.10.0-1.26360.102 (VMR commit c98e79e9, 2026-07-10). I verified via source that this build contains [release/5.0.1xx] Update dependencies from dotnet/msbuild #14251 (ProcessPacketsUntilBuildFinished + DrainPacketQueue + the [master] Update dependencies from microsoft/vstest #14172 regression test hook).
  • Added IL2026 suppressions at the 6 CLI MSBuild-object-model call sites, because 18.10 adds [RequiresUnreferencedCode] to ProjectInstance.Build/BuildManager (matching the existing IL3050 suppression pattern).

Local Debug build is clean and the previously-failing Microsoft.TemplateEngine.Authoring.Tasks.IntegrationTests pass (6/6) with the server on. Pushed to see if CI (especially the Unix Helix legs) goes green.

Note: the 18.10.0-1.26360.102 pin is a manual bring-forward to get #14251 for testing; it should be reconciled with the normal VMR dependency flow (i.e. dropped once main flows a VMR build that already contains #14251).

@JanProvaznik

Copy link
Copy Markdown
Member Author

there seems to be no problem after this #55245

Change DOTNET_CLI_USE_MSBUILD_SERVER to default to true so the MSBuild server is enabled when the user hasn't specified anything about it. An explicit MSBUILDUSESERVER value (or DOTNET_CLI_USE_MSBUILD_SERVER=false) is still respected as an opt-out.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 499249b7-514f-4ecf-88bc-2466455069c5
@JanProvaznik
JanProvaznik force-pushed the janprovaznik/msbuild-server-default-on branch from 28de084 to 98217a4 Compare July 14, 2026 11:45
@JanProvaznik
JanProvaznik marked this pull request as ready for review July 14, 2026 12:02
Copilot AI review requested due to automatic review settings July 14, 2026 12:02
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
2 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Enables the MSBuild server by default for the CLI’s MSBuild forwarding path (MSBuildForwardingApp), while still respecting explicit user opt-outs via DOTNET_CLI_USE_MSBUILD_SERVER=false or MSBUILDUSESERVER=0.

Changes:

  • Default DOTNET_CLI_USE_MSBUILD_SERVER evaluation to true in MSBuildForwardingAppWithoutLogging.
  • Only force MSBUILDUSESERVER=1 when the user hasn’t already set MSBUILDUSESERVER.
  • Add a unit test asserting the default-on behavior for MSBuildForwardingApp.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
test/dotnet.Tests/CommandTests/MSBuild/GivenMsbuildForwardingApp.cs Adds coverage for the new default-on MSBuild server behavior.
src/Cli/Microsoft.DotNet.Cli.Utils/MSBuildForwardingAppWithoutLogging.cs Switches the default to enabling MSBuild server and refines env-var forcing logic to respect user overrides.

Comment thread test/dotnet.Tests/CommandTests/MSBuild/GivenMsbuildForwardingApp.cs
@JanProvaznik

Copy link
Copy Markdown
Member Author

codeflow this was dependending on to fix msbuild bugs merged so the tests pass

@baronfel baronfel left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's go!

@baronfel
baronfel merged commit 16e4424 into dotnet:main Jul 14, 2026
27 checks passed
@dotnet-milestone-bot dotnet-milestone-bot Bot added this to the 11.0-preview7 milestone Jul 15, 2026
@Nigusu-Allehu

Copy link
Copy Markdown
Member

I noticed a perf improvement in dotnet restore time.

diff where improvement occurred: dotnet/dotnet@cb8306a...e78e1a0

Any chance this PR could be the reason?

@Nigusu-Allehu

Copy link
Copy Markdown
Member

If you are interested here is a link to the dashboard : dashboard

@JanProvaznik

Copy link
Copy Markdown
Member Author

@Nigusu-Allehu agent inspected the traces and the scenario source and yes it's explainable by this change.
However I'm not sure if all of your scenario definitions are correct now. The improved ones are measuring warm restore (a case when MSBuild was already running it's daemon on the machine):
When the main "MSBuild server" is reused it saves startup, JIT and reconnect to "worker node" time 🎉.
But the "Cold" scenario is also valid and interesting. If you want to define that, there needs to be in the setup "dotnet build-server shutdown" call before the dotnet restore or all dotnet.exe/msbuild.exe processes need to be killed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants