New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MSBuild should handle paths longer than MAX_PATH #53

Open
ariccio opened this Issue Mar 30, 2015 · 49 comments

Comments

Projects
None yet
@ariccio

ariccio commented Mar 30, 2015

As seen in Fix 260 character file name length limitation, there's quite a bit of support for better longer-than-MAX_PATH-filename handling.

Will Buik's response w/regard to fixing it was:

We understand that this can be a frustrating issue, however, fixing it requires a large and complicated architectural change across different products and features including Visual Studio, TFS, MSBuild and the .NET Framework. Dedicating resources to this work item would come at the expense of many other features and innovation.

I wondered, how big can those architectural changes be (for MSBuild)?

According to Naming Files, Paths, and Namespaces: Maximum Path Length Limitation, (which I have mirrored here ):

The Windows API has many functions that also have Unicode versions to permit an extended-length path for a maximum total path length of 32,767 characters. This type of path is composed of components separated by backslashes, each up to the value returned in the lpMaximumComponentLength parameter of the GetVolumeInformation function (this value is commonly 255 characters). To specify an extended-length path, use the "\\?\" prefix. For example, "\\?\D:\very long path".
[other stuff]

Because you cannot use the "\\?\" prefix with a relative path, relative paths are always limited to a total of MAX_PATH characters.

(emphasis mine)

...Which means that wherever MSBuild uses full paths, we can should be able to just prepend "\\?\" to ask for a long path.

Of course, we'd need to rework the existing path-length-workaround hack.

This won't fix the whole ecosystem, but it'll get us just one step closer.

I'd like to fix this myself (it seems simple enough), so in accordance with:

  • Contributions must be discussed with the team first, or they will likely be declined. As our process matures and our experience grows, the team expects to take larger contributions.
  • Only contributions referencing an approved Issue will be accepted.

...this is the suggested issue.

I'm a native developer at heart, not an experienced C# developer, but this shouldn't require anything crazy.

Direct references to MAX_PATH:

@AndyGerlicher

This comment has been minimized.

Show comment
Hide comment
@AndyGerlicher

AndyGerlicher Mar 31, 2015

Member

MSBuild team triage: This isn't a change we're looking to take anytime soon as it would be fairly substantial with implications on dependent tasks and tools.

Member

AndyGerlicher commented Mar 31, 2015

MSBuild team triage: This isn't a change we're looking to take anytime soon as it would be fairly substantial with implications on dependent tasks and tools.

@ariccio

This comment has been minimized.

Show comment
Hide comment
@ariccio

ariccio Apr 1, 2015

[...]it would be fairly substantial with implications on dependent tasks and tools.

Could you be a bit more specific?

ariccio commented Apr 1, 2015

[...]it would be fairly substantial with implications on dependent tasks and tools.

Could you be a bit more specific?

@ariccio

This comment has been minimized.

Show comment
Hide comment
@ariccio

ariccio Apr 8, 2015

I'm still curious as to what the substantial implications of an incremental fix for MSBuild in isolation are, exactly.

ariccio commented Apr 8, 2015

I'm still curious as to what the substantial implications of an incremental fix for MSBuild in isolation are, exactly.

@MattWhilden

This comment has been minimized.

Show comment
Hide comment
@MattWhilden

MattWhilden Apr 9, 2015

Member

In general, you'd have to have all of the other things you're using understand long paths. There's been a number of discussions internally and externally about the MAX_PATH issue and they always eventually boil down to: "If we could rewrite everything we'd be good but we can't so there's an infinite sea of compat risk here :-(".

So... maybe some day?

Member

MattWhilden commented Apr 9, 2015

In general, you'd have to have all of the other things you're using understand long paths. There's been a number of discussions internally and externally about the MAX_PATH issue and they always eventually boil down to: "If we could rewrite everything we'd be good but we can't so there's an infinite sea of compat risk here :-(".

So... maybe some day?

@gwojan

This comment has been minimized.

Show comment
Hide comment
@gwojan

gwojan Apr 9, 2015

Have you looked at AlphaFS?

gwojan commented Apr 9, 2015

Have you looked at AlphaFS?

@MattWhilden

This comment has been minimized.

Show comment
Hide comment
@MattWhilden

MattWhilden Apr 9, 2015

Member

The key issue here is that it doesn't really matter if MSBUILD is updated. It's not an island. it passes paths to lots of other things (OS apis, tools etc) and updating yourself in a way that doesn't break them is a Hard Problem (tm). Almost certainly not impossible but there's lots of other work with better payoffs.

Member

MattWhilden commented Apr 9, 2015

The key issue here is that it doesn't really matter if MSBUILD is updated. It's not an island. it passes paths to lots of other things (OS apis, tools etc) and updating yourself in a way that doesn't break them is a Hard Problem (tm). Almost certainly not impossible but there's lots of other work with better payoffs.

@ariccio

This comment has been minimized.

Show comment
Hide comment
@ariccio

ariccio Apr 10, 2015

Hmm. The way I see it, is that the entire toolchain is not compatible with paths longer than MAX_PATH, and thus by (incrementally) fixing a single component will not lead to toolchain-wide regression. It'll just be broken in a different place - the entire toolchain, when used as a system, will still be broken, but in a slightly smaller way.

It's even worse, by the way, as MAX_PATH was only the "maximum total file path length" for FAT16, and pre-Unicode/pre-NT APIs. Any documentation that says MAX_PATH is the longest possible file path is horribly wrong! MAX_PATH should really have been named MAX_FAT_PATH or MAX_PATH_COMPONENT; any program that was built after Windows NT 3.1 (yes, July 27, 1993, the earliest default use of NTFS that I know of) to use Unicode APIs with the assumption that MAX_PATH is the longest possible file path is, and always was, terminally broken. GCC 4.9-style broken.

I can even find discussions dating more than ten years back, of developers realizing that assuming MAX_PATH is the maximum length of a path is an outdated (ridiculous) assumption.

Any components that have been retroactively "fixed" to ensure compatibility with only paths that are MAX_PATH or shorter - well, that's just crazy.

Also, I can't imagine that we can break the OS APIs themselves by using them in a documented manner.

So yeah, it's a Hard Problem™, and I know that we don't yet have proper time travel, but if we attack it incrementally, we either make small (and effectively non-functional) improvements, or expose bugs/unforeseen design issues in other components.

I say effectively non-functional, because those components that can't handle MAX_PATH are already broken for paths longer than MAX_PATH. I can't imagine that those applications rely internally on the exact full path name that's given to them.

Furthermore, I imagine that OTHER toolchains use MSBuild in ways that are crippled by the MAX_PATH limitations.

...and while new features/better compliance are nice, I think they're counterproductive/add silly complexity if the core components that they're built on are shaky and fail in unexpected ways (e.g. MAX_PATH limitations). Putting it off into the future only makes it harder to fix.

Lastly, let me remind you of the nearly 3000 users who voted on "Fix 260 character file name length limitation" before it was declined. There are still users commenting on the issue with frustration.

Here's an idea

How about you/we go ahead and make the changes to MSBuild, make it available as a "preview" or "technology demonstration", and see what (if anything) breaks.

ariccio commented Apr 10, 2015

Hmm. The way I see it, is that the entire toolchain is not compatible with paths longer than MAX_PATH, and thus by (incrementally) fixing a single component will not lead to toolchain-wide regression. It'll just be broken in a different place - the entire toolchain, when used as a system, will still be broken, but in a slightly smaller way.

It's even worse, by the way, as MAX_PATH was only the "maximum total file path length" for FAT16, and pre-Unicode/pre-NT APIs. Any documentation that says MAX_PATH is the longest possible file path is horribly wrong! MAX_PATH should really have been named MAX_FAT_PATH or MAX_PATH_COMPONENT; any program that was built after Windows NT 3.1 (yes, July 27, 1993, the earliest default use of NTFS that I know of) to use Unicode APIs with the assumption that MAX_PATH is the longest possible file path is, and always was, terminally broken. GCC 4.9-style broken.

I can even find discussions dating more than ten years back, of developers realizing that assuming MAX_PATH is the maximum length of a path is an outdated (ridiculous) assumption.

Any components that have been retroactively "fixed" to ensure compatibility with only paths that are MAX_PATH or shorter - well, that's just crazy.

Also, I can't imagine that we can break the OS APIs themselves by using them in a documented manner.

So yeah, it's a Hard Problem™, and I know that we don't yet have proper time travel, but if we attack it incrementally, we either make small (and effectively non-functional) improvements, or expose bugs/unforeseen design issues in other components.

I say effectively non-functional, because those components that can't handle MAX_PATH are already broken for paths longer than MAX_PATH. I can't imagine that those applications rely internally on the exact full path name that's given to them.

Furthermore, I imagine that OTHER toolchains use MSBuild in ways that are crippled by the MAX_PATH limitations.

...and while new features/better compliance are nice, I think they're counterproductive/add silly complexity if the core components that they're built on are shaky and fail in unexpected ways (e.g. MAX_PATH limitations). Putting it off into the future only makes it harder to fix.

Lastly, let me remind you of the nearly 3000 users who voted on "Fix 260 character file name length limitation" before it was declined. There are still users commenting on the issue with frustration.

Here's an idea

How about you/we go ahead and make the changes to MSBuild, make it available as a "preview" or "technology demonstration", and see what (if anything) breaks.

@MattWhilden

This comment has been minimized.

Show comment
Hide comment
@MattWhilden

MattWhilden Apr 10, 2015

Member

You seem to understand the issue pretty well. I don't disagree that isn't a frustrating experience some times but they have finite resources and can only take so much risk. I'd be more interested in them making sure cross plat is brought up cleanly but it's up to them. I'm sure they appreciate your enthusiasm and feedback. I'll bow out of this issue for now.. lots of other bugs to track down.

Member

MattWhilden commented Apr 10, 2015

You seem to understand the issue pretty well. I don't disagree that isn't a frustrating experience some times but they have finite resources and can only take so much risk. I'd be more interested in them making sure cross plat is brought up cleanly but it's up to them. I'm sure they appreciate your enthusiasm and feedback. I'll bow out of this issue for now.. lots of other bugs to track down.

@ariccio

This comment has been minimized.

Show comment
Hide comment
@ariccio

ariccio Apr 10, 2015

Fair enough.

ariccio commented Apr 10, 2015

Fair enough.

@noamkfir

This comment has been minimized.

Show comment
Hide comment
@noamkfir

noamkfir Apr 10, 2015

@ariccio If you have the inclination, I say go for it. Everybody will eventually benefit, especially if more pieces of the toolchain go open source. Saying that it can't be fixed because everything else in every possible toolchain would also have to be fixed immediately is, quite simply, a non-starter and completely unrealistic. The whole point of going open source is to let people scratch their itches, especially when Microsoft doesn't have the resources to do it themselves...

One way to minimize the risk associated with backwards and toolchain compatibility that Microsoft keeps talking about is to ensure that the current limitation remains the default behavior while making it possible to opt-in to longer paths. The opt-in mechanism can be via any relevant configuration mechanism, including a new command line option, a build file setting, a registry setting and/or some other mechanism.

noamkfir commented Apr 10, 2015

@ariccio If you have the inclination, I say go for it. Everybody will eventually benefit, especially if more pieces of the toolchain go open source. Saying that it can't be fixed because everything else in every possible toolchain would also have to be fixed immediately is, quite simply, a non-starter and completely unrealistic. The whole point of going open source is to let people scratch their itches, especially when Microsoft doesn't have the resources to do it themselves...

One way to minimize the risk associated with backwards and toolchain compatibility that Microsoft keeps talking about is to ensure that the current limitation remains the default behavior while making it possible to opt-in to longer paths. The opt-in mechanism can be via any relevant configuration mechanism, including a new command line option, a build file setting, a registry setting and/or some other mechanism.

@gingters

This comment has been minimized.

Show comment
Hide comment
@gingters

gingters May 28, 2015

I'm also in.

Especially down the road to xplat, where *NIX operating systems do not care about a 260 char path length at all. IF set at all (this may well be -1 to indicate that there is no limit) it for example defaults to 4096 on x86 Linux. On OS X's HFSP the max path length is unlimited (while the file name is limited to 255).

So, in fact, this issue here is a blocker for xplat.

Yes, there may be other places in VS, TFS and other tools that have problems with this, but as long as we want to simply 'MSBuild mysolution' on commandline or on an CI system like AppVoyer or Travis, and happen to have paths longer than 260 chars there, it should not artificially prevent us from doing so.

I also don't see how removing this limitation would break other stuff.
If other stuff can't handle paths longer than 260 chars, it already IS broken. It won't break more. I would just break elsewhere.

And: Yes. Especially because MSBuild is not an island, but a part in a chain, someone has to start. And MSBuild is the perfect place because it can be used more or less standalone on new DNX projects (because, honestly, Sake is the totally wrong thing for building stuff), and it could be the door-opener to fix the stuff in other places too. Best thing is, that others like the VS or TFS team won't have excuses like 'MSBuild doesn't support that, so why should we change that?' anymore.

When talking about where to spent time, it's mostly the time of the community that will be spend here. So why not simply start here, by removing that artifical limitation, pass longer paths around where necessary and, who knows, maybe much less is broken that is currently being thought off and a lot of stuff simply starts to work out of the box?

gingters commented May 28, 2015

I'm also in.

Especially down the road to xplat, where *NIX operating systems do not care about a 260 char path length at all. IF set at all (this may well be -1 to indicate that there is no limit) it for example defaults to 4096 on x86 Linux. On OS X's HFSP the max path length is unlimited (while the file name is limited to 255).

So, in fact, this issue here is a blocker for xplat.

Yes, there may be other places in VS, TFS and other tools that have problems with this, but as long as we want to simply 'MSBuild mysolution' on commandline or on an CI system like AppVoyer or Travis, and happen to have paths longer than 260 chars there, it should not artificially prevent us from doing so.

I also don't see how removing this limitation would break other stuff.
If other stuff can't handle paths longer than 260 chars, it already IS broken. It won't break more. I would just break elsewhere.

And: Yes. Especially because MSBuild is not an island, but a part in a chain, someone has to start. And MSBuild is the perfect place because it can be used more or less standalone on new DNX projects (because, honestly, Sake is the totally wrong thing for building stuff), and it could be the door-opener to fix the stuff in other places too. Best thing is, that others like the VS or TFS team won't have excuses like 'MSBuild doesn't support that, so why should we change that?' anymore.

When talking about where to spent time, it's mostly the time of the community that will be spend here. So why not simply start here, by removing that artifical limitation, pass longer paths around where necessary and, who knows, maybe much less is broken that is currently being thought off and a lot of stuff simply starts to work out of the box?

@Piedone

This comment has been minimized.

Show comment
Hide comment
@Piedone

Piedone May 28, 2015

👍

Mediaeval issues like this still prevent us from using proper folder structures and proper file/folder names still. And it's not a corner case: for us all of our builds would fail if not copied to the drive root of the build server!

Piedone commented May 28, 2015

👍

Mediaeval issues like this still prevent us from using proper folder structures and proper file/folder names still. And it's not a corner case: for us all of our builds would fail if not copied to the drive root of the build server!

@hacst

This comment has been minimized.

Show comment
Hide comment
@hacst

hacst Sep 4, 2015

👍 This is something everyone with a complex build in a build system encounters sooner or later and it is quite annoying to work around. Fixing it has to start somewhere.

hacst commented Sep 4, 2015

👍 This is something everyone with a complex build in a build system encounters sooner or later and it is quite annoying to work around. Fixing it has to start somewhere.

@ariccio

This comment has been minimized.

Show comment
Hide comment
@ariccio

ariccio Sep 4, 2015

Glad to see that people are still voicing their support. Fixing this arcane issue is certainly not a glamorous job - hence the foot-dragging - but it's a desperately important one.

ariccio commented Sep 4, 2015

Glad to see that people are still voicing their support. Fixing this arcane issue is certainly not a glamorous job - hence the foot-dragging - but it's a desperately important one.

@jogibear9988

This comment has been minimized.

Show comment
Hide comment
@jogibear9988

jogibear9988 Sep 6, 2015

I'd also see this open 10 years from now if noone does a start!

But we also need a Api for full length paths directly in dotnet core!

jogibear9988 commented Sep 6, 2015

I'd also see this open 10 years from now if noone does a start!

But we also need a Api for full length paths directly in dotnet core!

@dsplaisted

This comment has been minimized.

Show comment
Hide comment
@dsplaisted

dsplaisted Sep 6, 2015

Member

Work is now in progress to enable long path support for .NET Core. I think the goal is to also add this to the full .NET Framework (@terrajobst can probably confirm whether this is the case).

Once this is done it should be a lot easier to add support for long paths to MSBuild- long paths should "just work" when they are passed to .NET APIs, without having to add the \\?\ prefix.

@AndyGerlicher Should we consider re-opening this issue?

Member

dsplaisted commented Sep 6, 2015

Work is now in progress to enable long path support for .NET Core. I think the goal is to also add this to the full .NET Framework (@terrajobst can probably confirm whether this is the case).

Once this is done it should be a lot easier to add support for long paths to MSBuild- long paths should "just work" when they are passed to .NET APIs, without having to add the \\?\ prefix.

@AndyGerlicher Should we consider re-opening this issue?

@ariccio

This comment has been minimized.

Show comment
Hide comment
@ariccio

ariccio Sep 7, 2015

Should we consider re-opening this issue?

Technically, it is still open.

But, if long path support is on it's way to the .NET Core, then I'd be totally cool treating this as a tracking issue.

ariccio commented Sep 7, 2015

Should we consider re-opening this issue?

Technically, it is still open.

But, if long path support is on it's way to the .NET Core, then I'd be totally cool treating this as a tracking issue.

@nalinjay

This comment has been minimized.

Show comment
Hide comment
@nalinjay

nalinjay Oct 14, 2015

Also running in to this same issue (with ASP.NET 5 DNX Beta 7 website publish operation):
System.ArgumentException: The name can be no more than 260 characters in length.

Do you have an update on when this will be resolved?

Thanks!

nalinjay commented Oct 14, 2015

Also running in to this same issue (with ASP.NET 5 DNX Beta 7 website publish operation):
System.ArgumentException: The name can be no more than 260 characters in length.

Do you have an update on when this will be resolved?

Thanks!

@AndyGerlicher

This comment has been minimized.

Show comment
Hide comment
@AndyGerlicher

AndyGerlicher Oct 14, 2015

Member

I know there's been some progress getting long path support in the core clr, but I think it will be a while before it's ported back to the desktop framework. Until that happens, this just isn't really high on our priority list. It's something we would probably do in xplat first since we'll get the core clr updates sooner, but it's not on our radar right now.

Member

AndyGerlicher commented Oct 14, 2015

I know there's been some progress getting long path support in the core clr, but I think it will be a while before it's ported back to the desktop framework. Until that happens, this just isn't really high on our priority list. It's something we would probably do in xplat first since we'll get the core clr updates sooner, but it's not on our radar right now.

@terrajobst

This comment has been minimized.

Show comment
Hide comment
@terrajobst

terrajobst Oct 14, 2015

Member

Given how far we are with making progress on solving it in .NET Core, I wouldn't invest in handling it in MSBuild. At least for the .NET Core based version it would get a free ride by doing nothing. .NET Framework is a bit more out, but again, our goal is to port those changes back and make it transparent which would mean that MSBuild would get a free ride.

Either way, I don't think doing a duct tape fix in MSBuild is currently worth it.

Member

terrajobst commented Oct 14, 2015

Given how far we are with making progress on solving it in .NET Core, I wouldn't invest in handling it in MSBuild. At least for the .NET Core based version it would get a free ride by doing nothing. .NET Framework is a bit more out, but again, our goal is to port those changes back and make it transparent which would mean that MSBuild would get a free ride.

Either way, I don't think doing a duct tape fix in MSBuild is currently worth it.

@AndyGerlicher

This comment has been minimized.

Show comment
Hide comment
@AndyGerlicher

AndyGerlicher Oct 14, 2015

Member

Completely agree. However the one caveat is we have several places in our code where we check if the path is over some const defined in our code. So we won't quite get a free ride.

Member

AndyGerlicher commented Oct 14, 2015

Completely agree. However the one caveat is we have several places in our code where we check if the path is over some const defined in our code. So we won't quite get a free ride.

@terrajobst

This comment has been minimized.

Show comment
Hide comment
@terrajobst

terrajobst Oct 14, 2015

Member

If that's the case, I'd encourage you to remove those. This isn't even a long path thing; for cross-platform alone you don't want to restrict paths on Mac/Linux to a Windows specific limit. In other words, removing those checks immediately benefits some customers.

(Also, on Windows you don't really lose anything. You might trade errors against different errors or run for longer before detecting it but that seems like the correct behavior anyways moving forward.)

Member

terrajobst commented Oct 14, 2015

If that's the case, I'd encourage you to remove those. This isn't even a long path thing; for cross-platform alone you don't want to restrict paths on Mac/Linux to a Windows specific limit. In other words, removing those checks immediately benefits some customers.

(Also, on Windows you don't really lose anything. You might trade errors against different errors or run for longer before detecting it but that seems like the correct behavior anyways moving forward.)

@ariccio

This comment has been minimized.

Show comment
Hide comment
@ariccio

ariccio Oct 15, 2015

Given how far we are with making progress on solving it in .NET Core, I wouldn't invest in handling it in MSBuild.

I'm glad to hear that that project is progressing nicely! Any idea on how long "a bit more out" is?

Of course, it's software, so I wont hold it against you as a "deadline" of some sort :)

ariccio commented Oct 15, 2015

Given how far we are with making progress on solving it in .NET Core, I wouldn't invest in handling it in MSBuild.

I'm glad to hear that that project is progressing nicely! Any idea on how long "a bit more out" is?

Of course, it's software, so I wont hold it against you as a "deadline" of some sort :)

@terrajobst

This comment has been minimized.

Show comment
Hide comment
@terrajobst

terrajobst Oct 19, 2015

Member

We don't have a timeline but I'll keep you posted via our design reviews, Twitter and so on. Hopefully the dev driving it can give us an update in a couple of weeks.

Member

terrajobst commented Oct 19, 2015

We don't have a timeline but I'll keep you posted via our design reviews, Twitter and so on. Hopefully the dev driving it can give us an update in a couple of weeks.

@ishepherd

This comment has been minimized.

Show comment
Hide comment
@ishepherd

ishepherd Aug 14, 2017

Any progress? Please shift this way up the backlog and get that significant cross-team collaboration going. I appreciate it may not feature on any marketing checklists but it will be very widely appreciated.

I curse every time I have to workaround this last-century limitation.

Edit: 'Sarabeth-Jaffe-Microsoft was unassigned by ishepherd', um no I didn't 😕

ishepherd commented Aug 14, 2017

Any progress? Please shift this way up the backlog and get that significant cross-team collaboration going. I appreciate it may not feature on any marketing checklists but it will be very widely appreciated.

I curse every time I have to workaround this last-century limitation.

Edit: 'Sarabeth-Jaffe-Microsoft was unassigned by ishepherd', um no I didn't 😕

jonpryor referenced this issue in xamarin/xamarin-android Sep 6, 2017

[Xamarin.Android.Build.Tasks] Path is too long (#700)
Fixes: https://bugzilla.xamarin.com/show_bug.cgi?id=30147

Windows (still) has a max path limit of 260 characters..
This can cause us a problem if the user creates a project in
a directory which already takes up most of that limit.. Like on
their Desktop!

This is because of the intermediate structure that was introduced
to handle embedding resources into the assemblies. The current
intermediate structure is as follows

	$(IntermediateOutputPath)\__library_project_imports__\$(AssemblyName)\library_project_imports\
	$(IntermediateOutputPath)\__library_project_imports__\$(AssemblyName)\native_library_imports\

Now consider that `$(AssemblyName)` can sometimes end up with something lile
"Xamarin.Android.Support.v7.AppCompat.21.0.3.0" you can easily see how we
start getting into trouble.

There was an attempt to fix this up a while ago (722dcc05) by introducing the
`$(UseShortFileNames)` property. However while this did introduce a
directory structure which was shorter is broke backwards compatability
with older versions of xamarin-android.

This is because of the structore of the zip files we that are embedded
into the assemblies. The current system just extracts the zip into
the `$(IntermediateOutputPath)\__library_project_imports__\$(AssemblyName)\`
directory and assumes that it contains a `library_project_imports` directory.

All the build tasks also assumed that as well.

So the fix needs to be done on a number of fronts. Firstly we need to
update the `$(_LibraryProjectImportsDirectoryName)` and
`$(_NativeLibraryImportsDirectoryName)` to be something shorter.
Next up we need to shorten "__library_project_imports__" to something
else verbose as well. This should cut down on the amount of the MAX_PATH
we chew up.

The real key to this is to NOT change the structure of the zip files in the
assemblies! So when we generate a zip from "jlibs" we make sure that the
folder in the zip is called "library_project_imports". And when we extract
the zip file we makle sure that "library_project_imports" is replaced by
the shorter name.
This will ensure that we are backward compatbile with older versions BUT
more importantly we get to use the shorter directory structure.
The files for native librarys are not extracted to disk but are extracted
from memory so as long as the structure remains the same i.e "native_library_imports"
that code does not need to change.

The other thing we need to do is to update ResolveLibraryProjectImports Task to
upgrade the system when it runs. So if we already have a "libraryprojectimports.cache"
in place we just use that as is. But if we re-run the ResolveLibraryProjectImports
task (due to a change or a clean build) we detect if we have the older structure in place
and just remove it.. Since we are going to regenerate the entire cache again anyway
we might as well start from scratch.

With this in place it becomes possible we can now enable `$(UseShortFileNames)`
by default!

So the new structure that is created is as follows

	$(IntermediateOutputPath)\lp\<id>\jl
	$(IntermediateOutputPath)\lp\<id>\nl

The <id> will be a single integer value which can be mapped to the source
assembly via a new map.cache file. This .cache file will contain a list of
assemblys. We use the `Index` of the assembly in the list to figure out
which cache directory to use.

	UnnamedProject
	Library1

In this case `Library1` would have an index of `1` and `UnnamedProject` will
be `0`. The old behaviour can be enabled by setting
`$(UseShortFileNames)` to `False`.

Note in order to keep existing bindings generator behaviour consistent,
the BindingsGenerator task will not use the `$(UseShortFileNames)`
property to control how it generates its .cs files. Instead a new
property

	$(UseShortGeneratorFileNames)

which can be used to control if the generator produces short names
(e.g 1.cs, 2.cs). This will be `False` by default.
@daghb

This comment has been minimized.

Show comment
Hide comment
@daghb

daghb Oct 2, 2017

With .NET Framework 4.6.2 partly supporting > MAX_PATH now, this is perhaps an easier task than ever?

daghb commented Oct 2, 2017

With .NET Framework 4.6.2 partly supporting > MAX_PATH now, this is perhaps an easier task than ever?

@mdealer

This comment has been minimized.

Show comment
Hide comment
@mdealer

mdealer Jan 18, 2018

Every build system on Windows that I encounter requires ugly workarounds due to this old problem. Any progress? I really don't care if MSBuild suddenly needs 10MB more RAM, just make it work. Our build server got a 256GB upgrade on the cheap, which you should try too.

mdealer commented Jan 18, 2018

Every build system on Windows that I encounter requires ugly workarounds due to this old problem. Any progress? I really don't care if MSBuild suddenly needs 10MB more RAM, just make it work. Our build server got a 256GB upgrade on the cheap, which you should try too.

@rainersigwald rainersigwald modified the milestones: After 15, MSBuild 15.8 Mar 27, 2018

radical added a commit to radical/msbuild that referenced this issue May 3, 2018

Merge pull request Microsoft#53 from radical/update-script
[mono] Add support for fetching libhostfxr* when updating SDKs
@michaelheilmann

This comment has been minimized.

Show comment
Hide comment
@michaelheilmann

michaelheilmann May 19, 2018

Let me summarize the thread so far because it is quite lengthy: Basically Microsoft teams are telling us that they are not able to provide a working build tool in the year 2018 because of reasons ...

As a side note: No problems with path lengths under Cygwin ... that suggests that this issue is very msbuild specific.

michaelheilmann commented May 19, 2018

Let me summarize the thread so far because it is quite lengthy: Basically Microsoft teams are telling us that they are not able to provide a working build tool in the year 2018 because of reasons ...

As a side note: No problems with path lengths under Cygwin ... that suggests that this issue is very msbuild specific.

@skiryazov

This comment has been minimized.

Show comment
Hide comment
@skiryazov

skiryazov Jun 18, 2018

1996 called, they want their short paths back.

Seriously though, I'm working on a workaround for this and I'm desperately trying to avoid my Java and PHP colleagues figuring out what I'm working on as my team will be mercilessly mocked. By PHP developers! They have like 10 years of mocking to pay back.

Please guys, fix this before somebody on the other camps finds out.

skiryazov commented Jun 18, 2018

1996 called, they want their short paths back.

Seriously though, I'm working on a workaround for this and I'm desperately trying to avoid my Java and PHP colleagues figuring out what I'm working on as my team will be mercilessly mocked. By PHP developers! They have like 10 years of mocking to pay back.

Please guys, fix this before somebody on the other camps finds out.

ccastanedaucf added a commit to ccastanedaucf/msbuild that referenced this issue Jun 19, 2018

Preserve long path behavior on platforms affected by max path limitat…
…ions (Microsoft#53)

On systems limited by max path, checks were added to ensure that they handle long paths as before and throw exceptions when a path can't be shortened.

In File\Directory exists methods, the native calls to FileExists were removed. The native GetFileAttributesEx seems to have issues with long relative paths. Tests showed that there's no performance difference on net46 and netcore compared to the managed call, and FileOrDirectoryExistsNoThrow() now accesses the disk just once on all platforms.
@KirillOsenkov

This comment has been minimized.

Show comment
Hide comment
@KirillOsenkov
Member

KirillOsenkov commented Jul 20, 2018

See also: dotnet/sdk#2132

@natemcmaster

This comment has been minimized.

Show comment
Hide comment
@natemcmaster

natemcmaster Aug 23, 2018

Member

I ran into this again today. Apparently, globbing will produce items for files that exceed MAX_PATH, but when those items are passed to built-in tasks such as <Copy>, MSBuild chokes.

error MSB3030: Could not copy the file "C:\src\aspnet\Coherence-Signed\obj\UnsignedPackages\Microsoft.AspNetCore.AzureAppServices.SiteExtension.2.2.0-preview1-35029\content\store\x64\netcoreapp2.2\microsoft.aspnetcore.azureappservices.hostingstartup\2.2.0-preview1-35029\lib\netcoreapp2.1\Microsoft.AspNetCore.AzureAppServices.HostingStartup.dll" because it was not found.

My workaround: https://gist.github.com/natemcmaster/1622db9d42156a306d16da8b1dafc795

Member

natemcmaster commented Aug 23, 2018

I ran into this again today. Apparently, globbing will produce items for files that exceed MAX_PATH, but when those items are passed to built-in tasks such as <Copy>, MSBuild chokes.

error MSB3030: Could not copy the file "C:\src\aspnet\Coherence-Signed\obj\UnsignedPackages\Microsoft.AspNetCore.AzureAppServices.SiteExtension.2.2.0-preview1-35029\content\store\x64\netcoreapp2.2\microsoft.aspnetcore.azureappservices.hostingstartup\2.2.0-preview1-35029\lib\netcoreapp2.1\Microsoft.AspNetCore.AzureAppServices.HostingStartup.dll" because it was not found.

My workaround: https://gist.github.com/natemcmaster/1622db9d42156a306d16da8b1dafc795

@AndyGerlicher

This comment has been minimized.

Show comment
Hide comment
@AndyGerlicher

AndyGerlicher Aug 23, 2018

Member

@ccastanedaucf
This may be addressed by #3503? Either way it's still a work in progress. We're getting closer, but there are still some fundamental "this string probably isn't a file because it's so long" logic to weed out in MSBuild.

Member

AndyGerlicher commented Aug 23, 2018

@ccastanedaucf
This may be addressed by #3503? Either way it's still a work in progress. We're getting closer, but there are still some fundamental "this string probably isn't a file because it's so long" logic to weed out in MSBuild.

@mdealer

This comment has been minimized.

Show comment
Hide comment
@mdealer

mdealer Aug 27, 2018

"this string probably isn't a file because it's so long" logic to weed out in MSBuild.
Wow, thanks for sharing a pointer to the root cause of all our Windows suffering. There's not only logic to weed out, but also the related developers and managers which find this acceptable to get shipped to millions of people for tens of years to come!

mdealer commented Aug 27, 2018

"this string probably isn't a file because it's so long" logic to weed out in MSBuild.
Wow, thanks for sharing a pointer to the root cause of all our Windows suffering. There's not only logic to weed out, but also the related developers and managers which find this acceptable to get shipped to millions of people for tens of years to come!

@AndyGerlicher

This comment has been minimized.

Show comment
Hide comment
@AndyGerlicher

AndyGerlicher Aug 27, 2018

Member

What I was referring to was pretty specific to MSBuild and not necessarily a root cause for the larger ecosystem. And I get the frustration, we are affected by this here as well. But there are a lot of barriers for this particular problem for it to be fully solved. For example:
https://github.com/search?q=MAX_PATH&type=Code

That search alone has over 1M code hits. Through a modern lens I don't think you'll get an argument from anyone that this was a good idea, but we are trying to address it. Our hope is to have all of these solved for MSBuild in our codebase for the next major release. Unfortunately, even that may not fully help you if one of your dependencies still has path limits.

Member

AndyGerlicher commented Aug 27, 2018

What I was referring to was pretty specific to MSBuild and not necessarily a root cause for the larger ecosystem. And I get the frustration, we are affected by this here as well. But there are a lot of barriers for this particular problem for it to be fully solved. For example:
https://github.com/search?q=MAX_PATH&type=Code

That search alone has over 1M code hits. Through a modern lens I don't think you'll get an argument from anyone that this was a good idea, but we are trying to address it. Our hope is to have all of these solved for MSBuild in our codebase for the next major release. Unfortunately, even that may not fully help you if one of your dependencies still has path limits.

@rkeithhill

This comment has been minimized.

Show comment
Hide comment
@rkeithhill

rkeithhill Sep 28, 2018

I get that MSBuild is just part of the problem but we'll never find out what the rest of the problems are until MSBuild gets out of the way first by supporting long paths.

rkeithhill commented Sep 28, 2018

I get that MSBuild is just part of the problem but we'll never find out what the rest of the problems are until MSBuild gets out of the way first by supporting long paths.

@sanchitabrol

This comment has been minimized.

Show comment
Hide comment
@sanchitabrol

sanchitabrol Oct 1, 2018

Member

I encountered this problem while using the <Copy> task which fails for paths with length > MAX_PATH. I ended up using a Nuget package MSBuildTasks which had a <RoboCopy> task which can handle MAX_PATH. Would be good to have this support natively in <Copy>.

Member

sanchitabrol commented Oct 1, 2018

I encountered this problem while using the <Copy> task which fails for paths with length > MAX_PATH. I ended up using a Nuget package MSBuildTasks which had a <RoboCopy> task which can handle MAX_PATH. Would be good to have this support natively in <Copy>.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment