x64 MSBuild is getting CLR quirks for .NET 4.0 behaviors

[mahaase]

Already reported by @ariccio here: #53
but a new ticket, because of 64-bit variant isn't solved.

As seen in Fix 260 character file name length limitation, there's quite a bit of support for better longer-than-MAX_PATH-filename handling.

Will Buik's response w/regard to fixing it was:

We understand that this can be a frustrating issue, however, fixing it requires a large and complicated architectural change across different products and features including Visual Studio, TFS, MSBuild and the .NET Framework. Dedicating resources to this work item would come at the expense of many other features and innovation.

I wondered, how big can those architectural changes be (for MSBuild)?

According to Naming Files, Paths, and Namespaces: Maximum Path Length Limitation, (which I have mirrored here ):

The Windows API has many functions that also have Unicode versions to permit an extended-length path for a maximum total path length of 32,767 characters. This type of path is composed of components separated by backslashes, each up to the value returned in the lpMaximumComponentLength parameter of the GetVolumeInformation function (this value is commonly 255 characters). To specify an extended-length path, use the "\\?\" prefix. For example, "\\?\D:\very long path".
[other stuff]

Because you cannot use the "\\?\" prefix with a relative path, relative paths are always limited to a total of MAX_PATH characters.

(emphasis mine)

...Which means that wherever MSBuild uses full paths, we can should be able to just prepend "\?" to ask for a long path.

Of course, we'd need to rework the existing path-length-workaround hack.

This won't fix the whole ecosystem, but it'll get us just one step closer.

I'd like to fix this myself (it seems simple enough), so in accordance with:

    Contributions must be discussed with the team first, or they will likely be declined. As our process matures and our experience grows, the team expects to take larger contributions.
    Only contributions referencing an approved Issue will be accepted.

...this is the suggested issue.

I'm a native developer at heart, not an experienced C# developer, but this shouldn't require anything crazy.

Direct references to MAX_PATH:

NativeMethods.cs#L15
NativeMethodsShared_Tests.cs#L53
NativeMethodsShared.cs#L318
NativeMethodsShared.cs#L540
NativeMethodsShared.cs#L750
FileState.cs#L293
ComReference.cs#L460
TargetsFile_Test.cs#L2041

[rainersigwald]

Can you provide some repro steps? what kind of project are you building? How exactly are you launching MSBuild?

[mahaase]

In my case, we use bob (build automation tool) which uses e.g. CMake to build windows binaries. CMake calls the msbuild by using the CMake VS2019 generators.
In some cases, the build automation tool creates paths, which resulting with the MAX_PATH issues.
It isn't possible to handle this paths, they will auto-generated by the tool.

refer: #53 (comment)

Has anyone ever seen/verified, that this fix still works?

I enabled long paths in gpedit.
I checked msbuild.exe resource stuff, still fine.
I use MSBuild v16.2, but the long path issues still exists.

"...maximale Pfadlimit des Betriebssystems. Der vollqualifizierte Dateiname muss weniger als 260 Zeichen umfassen"

The issue occurs inside a build-system, which handles the paths (directory-names a.s.o.) by itself, no way/easy way to change them.

more references:

refer: #53 (comment)

@rainersigwald was this expected to be fixed in both the 32bit and 64bit (amd64\msbuild.exe) msbuild binaries? I am getting interesting behavior wherein the 64 bit version throws this error:

"S:\Builds\GenerateInterop-60R39\WorkingDirectory\src\Dotnet\Source\Framework\Interop\Project\ComputersUnlimited.Interop.sln" (Restore target) (1) ->
(Restore target) ->
  C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\CommonExtensions\Microsoft\NuGet\NuGet.targets(121,5): error : The specified path, file name, or both are too long. The fully qualified file name must be less than 260 characters, and the directory name must be less than 248 characters. [S:\Builds\GenerateInterop-60R39\WorkingDirectory\src\Dotnet\Source\Framework\Interop\Project\ComputersUnlimited.Interop.sln]

Whereas using the 32bit version has no such qualms (restores and builds without issue).

Here's the version from the 64bit and 32bit versions (same):

Microsoft (R) Build Engine version 16.1.76+g14b0a930a7 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

I can open a new issue if need be but wanted to start here. If this is something I need to get off to the NuGet guys I can do that as well.

refer: #53 (comment)

@mahaase same problem here. 32bit is still working, 64bit does show the max_path error. Tested with newest visual studio build tools on newest windows 10. Thanks for the tip with 32bit. I was nearly giving up and taking myself to a knit.

But, generally I guess, we don't need any special use-case to reproduce - I guess there is no case, where the issue doesn't occurs if the paths just long enough.

[rainersigwald]

@benvillalobos, can you try to repro?

[mahaase]

@benvillalobos could you please 👍

[rainersigwald]

@mahaase (or anyone): Can you try patching your amd64\MSBuild.exe.config file like this:

diff --git a/src/MSBuild/app.amd64.config b/src/MSBuild/app.amd64.config
index f37149aca..c2023dc93 100644
--- a/src/MSBuild/app.amd64.config
+++ b/src/MSBuild/app.amd64.config
@@ -10,7 +10,7 @@
       <DisableFXClosureWalk enabled="true" />
       <DeferFXClosureWalk enabled="true" />
       <generatePublisherEvidence enabled="false" />
-      <AppContextSwitchOverrides value="Switch.System.Security.Cryptography.UseLegacyFipsThrow=false" />
+      <AppContextSwitchOverrides value="Switch.System.Security.Cryptography.UseLegacyFipsThrow=false;Switch.System.IO.UseLegacyPathHandling=false;Switch.System.IO.BlockLongPaths=false" />
       <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
         <dependentAssembly>
           <assemblyIdentity name="Microsoft.Build.Framework" culture="neutral" publicKeyToken="b03f5f7f11d50a3a" />

? That appears to resolve the problem for me.

[dannyvv]

@mahaase (or anyone): Can you try patching your amd64\MSBuild.exe.config file like this:

-      <AppContextSwitchOverrides value="Switch.System.Security.Cryptography.UseLegacyFipsThrow=false" />
+      <AppContextSwitchOverrides value="Switch.System.Security.Cryptography.UseLegacyFipsThrow=false;Switch.System.IO.UseLegacyPathHandling=false;Switch.System.IO.BlockLongPaths=false" />

? That appears to resolve the problem for me.

I can confirm that this makes the long path errors go away. If anyone needs a workaround for this in their CI system, the following step before building does the trick for our queue.

  - task: PowerShell@2
    displayName: Fix LongPath bug in MsBuild
    inputs:
      targetType: 'inline'
      script: |
        $msbExeConfigPath="C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Current\bin\amd64\msbuild.exe.config"
        [ xml ]$msbExeConfig = Get-Content -Path $msbExeConfigPath
        $msbExeConfig.configuration.runtime.AppContextSwitchOverrides.SetAttribute("value", "Switch.System.Security.Cryptography.UseLegacyFipsThrow=false;Switch.System.IO.UseLegacyPathHandling=false;Switch.System.IO.BlockLongPaths=false")
        $msbExeConfig.Save($msbExeConfigPath)

[dannyvv]

@rainersigwald: Thank you so much for a quick turnaround!

[Piedone]

What a time to be alive! :)

[rainersigwald]

I'm actually going to reactivate this as a more general problem; Something™ is going wrong that's causing .NET Framework to give us the .NET 4.0 behaviors for everything instead of .NET 4.7.2+ . . . only in our x64 MSBuild.exe. I'm investigating with some CLR folks.

[rainersigwald]

This turns out to be due to a bug in .NET Framework (internal link: https://dev.azure.com/devdiv/DevDiv/_workitems/edit/1148752).

If you load an assembly that needs a codeBase redirect to a different location, part of the CLR accidentally checks the target framework of the running application before it has actually been set. That causes .NET to provide compatibility shims as though the application targeted .NET 4.0, instead of whatever it actually targeted. That includes the System.IO.BlockLongPaths behavior (because that was the only behavior in .NET 4.0).

The Runtime folks are going to look the underlying bug. Fortunately, we don't really need the codeBase for x64--we still ship the assemblies in the bin\amd64 folder. #5471 drops the codeBase to work around the Framework bug.

[rainersigwald]

Reactivating: just removing the codeBase caused #5494.

The current state is that 16.7 gets all the 4.0 behavior (same as 16.6) except for long paths (thanks to #5460). We should try to come up with a better fix for 16.8 (and ship it in an earlier preview this time 😬).

[spanevin]

Any updates? long path is a real issue...

[odalet]

I thought this was fixed in 16.9, but I still had the issue. I've examined some of the commits here and it seems it was once part of the code base, but then removed. Any reason why? Now the target release is 16.10...

Btw, manually modifying amd64\MSBuild.exe.config did the trick, so I'm just wondering why this is still not official...

[spanevin]

up

[spanevin]

https://github.com/dotnet/msbuild/milestone/51

I don't see this issue here, where else I can check that target release is really 16.10 ?

[jasonycw]

Hi, I was brought here by https://developercommunity.visualstudio.com/t/msbuild-editorconfig-filename-length-issues/1462559

And I had updated my C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\amd64\MSBuild.exe.config to have

<AppContextSwitchOverrides value="...... Switch.System.Security.Cryptography.UseLegacyFipsThrow=false;Switch.System.IO.UseLegacyPathHandling=false;Switch.System.IO.BlockLongPaths=false" />

But my VS still not able to build properly and showing

Error MSB3491 Could not write lines to file “obj\Debug\netcoreapp3.1\MySuperDeepProject.UnitTests.GeneratedMSBuildEditorConfig.editorconfig”. Path: obj\Debug\netcoreapp3.1\MySuperDeepProject.UnitTests.GeneratedMSBuildEditorConfig.editorconfig exceeds the OS max path limit. The fully qualified file name must be less than 260 characters.

The error point to this line in C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\Roslyn\Microsoft.Managed.Core.targets which is not my project

Tried running dotnet build directly in the root folder and it can build properly.

Running Microsoft Visual Studio Community 2019 Version 16.11.1

[KrisThielemans]

I have this problem with Visual Studio Community 2022

2>MSBuild version 17.3.1+2badb37d1 for .NET Framework
....
6>C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(383,5): error MSB3491: Could not write lines to file "test_PoissonLogLikelihoodWithLinearModelForMeanAndProjData.dir\RelWithDebInfo\test_Poi.AB258268.tlog\test_PoissonLogLikelihoodWithLinearModelForMeanAndProjData.lastbuildstate". Path: test_PoissonLogLikelihoodWithLinearModelForMeanAndProjData.dir\RelWithDebInfo\test_Poi.AB258268.tlog\test_PoissonLogLikelihoodWithLinearModelForMeanAndProjData.lastbuildstate exceeds the OS max path limit. The fully qualified file name must be less than 260 characters. [C:\Users\krisf\Documents\devel\buildVC22\SIRF-SuperBuild\builds\STIR\build\src\recon_test\test_PoissonLogLikelihoodWithLinearModelForMeanAndProjData.vcxproj]

I've tried patching the MSBuild.exe.config but that didn't help.

[KrisThielemans]

I have this problem with Visual Studio Community 2022

apologies. This wasn't msbuild, it was actually the OS path limit, as it said...

(Setting the registry key via Set-ItemProperty 'HKLM:\System\CurrentControlSet\Control\FileSystem' -Name 'LongPathsEnabled' -value 1 wasn't enough for me (Windows 10 Pro). I had to use the Group Policy editor as well.)

Apologies for the noise.

[neominky]

This bug still exists. (17.9.6, x64, .NET8) Why isn't it fixed?
In my case, an error occurs even with a folder length of 122 characters. Is it because it is a non-ascii country?
Set-ItemProperty 'HKLM:\System\CurrentControlSet\Control\FileSystem' -Name 'LongPathsEnabled' -value 1 is not works

patching "MSBuild.exe.config" is not works

[rainersigwald]

@neominky please file a new issue describing your problem, including details about how you are launching the build.