Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

os: some functions fail if full paths are long (more than about 250 chars) on windows #3358

Closed
alexbrainman opened this issue Mar 20, 2012 · 42 comments
Assignees
Milestone

Comments

@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented Mar 20, 2012

This program creates directory, but fails to create file inside (CAREFUL IF CHANGING
THIS PROGRAM - YOU MIGHT END UP WITH FILES/DIRECTORIES YOU WILL NOT BE ABLE TO DELETE
WITH STANDARD SHELL TOOLS):

package main

import (
    "fmt"
    "log"
    "os"
    "strings"
)

func main() {
//  root := `\\?\c:\`
    root := `c:\`
    dir := root + strings.Repeat("d", 200)
    file := dir + `\` + strings.Repeat("f", 200)
    err := os.Mkdir(dir, 0777)
    if err != nil {
        log.Fatalf("Mkdir(%s): %v", dir, err)
    }
    f, err := os.Create(file)
    if err != nil {
        log.Fatalf("Create(%s): %v", file, err)
    }
    _, err = fmt.Fprintf(f, "Hello world")
    if err != nil {
        log.Fatalf("Fprintf(%s): %v", file, err)
    }
    err = f.Close()
    if err != nil {
        log.Fatalf("Close(%s): %v", file, err)
    }
    err = os.Remove(file)
    if err != nil {
        log.Fatalf("Remove(%s): %v", file, err)
    }
    err = os.Remove(dir)
    if err != nil {
        log.Fatalf("Remove(%s): %v", dir, err)
    }
}

This can be handled properly if we use \\?\ path prefix (uncomment line in the code).
But \\?\ prefixed paths needs to be "normalized": they have to be absolute, no
. or .. allowed, / is not allowed, UNC paths needs to be handled properly. See
http://goo.gl/th2fJ for details, especially "Maximum Path Length Limitation"
section.

This issue has come up on our own builder: http://goo.gl/FbblU. Perhaps our own paths do
not need to be that long. I did not look into that yet.

Alex
@dsymonds

This comment has been minimized.

Copy link
Member

@dsymonds dsymonds commented Mar 20, 2012

Comment 1:

Labels changed: added priority-go1, os-windows, removed priority-triage.

@minux

This comment has been minimized.

Copy link
Member

@minux minux commented Mar 22, 2012

Comment 2:

Should this be fixed in pkg syscall?
@alexbrainman

This comment has been minimized.

Copy link
Member Author

@alexbrainman alexbrainman commented Mar 23, 2012

Comment 3:

I am not sure how this should be fixed. It might not be easy. I never dealt with these
in the past. But after some investigation, here is what I see is happening
(http://goo.gl/th2fJ for details).
All Windows apis, that take file name, accept it in a form that needs to be processed
before it gets passed to correspondent file system:
- it can contains relative paths, then "file name processing" code tucks in current
directory in front;
- all / are converted into \;
- it deals with . and ..;
- ... and so on ...
It appears that this "file name processor" can only deal with files with absolute path
less then ~ 260 characters long. So even if we pass file name like "alex.txt" to it, it
might fail, because current directory full path might be too long already.
There is one way to overcome ~ 260 length limitation - it is to skip "file name
processor", and pass file name directly to file system (the file length limit there is
~32000). The way to do it is to prefix file name with \\?\, so c:\tmp\alex.txt would
become \\?\c:\tmp\alex.txt and so on. The down side to this solution is that we loose
all "file name processing" along the way, so we need to do it all manually. All file
names have to be full path (absolute), we must reject file names like . and .. by hand
and so on.
I do not think we should "convert" all our file names into \\?\... in syscall by
default. It could be quite dangerous, because it is possible to create files with weird
names, like . and .. and such. Normal shell utilities and GUI wouldn't be able to deal
with those. We might discover that, while perfectly accepted by Windows api, these might
be rejected by other programs / apis. It could be confusing for some users to find that
they pass "alex.txt" into Go and to see that Windows api gets \\?\c:\tmp\alex.txt.
We might let things be the way they are, but monitor for particular Windows api errors
and call api second time with \\?\... form of the file name. But again, it feels too
magical to me.
On the other hand, nothing stops our users from doing os.Create(`\\?\c:\tmp\alex.txt`),
if they want to. So, I think, the issue here are not long file names as such, but more
the fact that our Go builder hit this problem http://goo.gl/FbblU. So, perhaps we can
change go command in such way that it is unlikely to happen. Looking at particular error:
# ..\test\bench\go1
#
_/C_/Users/ADMINI~1/AppData/Local/Temp/2/gobuilder/windows-amd64-0449267813c1/go/test/bench/go1
mkdir
C:\Users\ADMINI~1\AppData\Local\Temp\2\go-build317224146\_\C_\Users\ADMINI~1\AppData\Local\Temp\2\gobuilder\windows-amd64-0449267813c1\go\test\bench\go1\_test\_\C_\Users\ADMINI~1\AppData\Local\Temp\2\gobuilder\windows-amd64-0449267813c1\go\test\bench\:
The filename or extension is too long.
it happens during
go test ..\test\bench\go1
If we break long path, we could see what is happening. It starts with TEMP directory
C:\Users\ADMINI~1\AppData\Local\Temp\2\
we created a temp directory there
go-build317224146\
then we converted full path of ..\test\bench\go1 into import path (because it is not in
GOPATH)
_\C_\Users\ADMINI~1\AppData\Local\Temp\2\gobuilder\windows-amd64-0449267813c1\go\test\bench\go1\
(watch full TEMP path here again that was created by gobuilder),
then we get _test + import-path again directory that holds transient test files
_\C_\Users\ADMINI~1\AppData\Local\Temp\2\gobuilder\windows-amd64-0449267813c1\go\test\bench\
Perhaps, some of these rules can be changed to shorten the path.
Maybe
go test ..\test\bench\go1
on gobuilder is not important, but I am concerned with what happens if people start
creating deep directories inside their GOPATH. Especially encouraged by "go get ..."
conventions. Something like that:
C:\a\code\src\code.google.com\p\gowingui\code.google.com\p\codesearch
Alex
@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Mar 26, 2012

Comment 4:

This is a big change.  Let's leave it for after Go 1.

Labels changed: added priority-later, removed priority-go1.

@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Sep 12, 2012

Comment 5:

Labels changed: added go1.1.

@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Dec 10, 2012

Comment 6:

Labels changed: added size-l.

@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Dec 10, 2012

Comment 7:

Labels changed: added suggested.

@kardianos

This comment has been minimized.

Copy link
Contributor

@kardianos kardianos commented Mar 7, 2013

Comment 8:

rsc: What do you consider the issue here? Is it that the package "os" can't operate on
long paths in windows or the go tool can't?
@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Mar 11, 2013

Comment 9:

The real issue is that the Windows API needs to allow long paths. But I agree with Alex
that it is far too invasive a change to be considering so close to Go 1.1.
We can tweak the go command to shorten some of the temporary paths, so that it is at
least not doubled. I created issue #5027 for that.

Labels changed: added go1.2, removed go1.1, suggested.

@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Jul 30, 2013

Comment 10:

Labels changed: added feature.

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Jul 31, 2013

Comment 11 by diogomfranco:

> On the other hand, nothing stops our users from doing
os.Create(`\\?\c:\tmp\alex.txt`), if they want to.
That'd mean making stdlib users write OS-dependent code even without using syscall
(which doesn't sound desirable); moreover, *not* writing that OS-dependent code
would only be noticed on corner cases and would imply lots of boilerplate around
every single call that deals with the filesystem, which is specially harsh when dealing
with relative paths.
If making os handling such conversion automatically looks overkill, then maybe
adding a function to path/filepath that can create such a "super-absolute" path,
but returns a regular absolute path on non-windows? It would also need documentation
that it's needed to handle long paths (or long CWDs) on windows, and possibly a
reference on the os package's filesystem functions.
As an additional note, "native Cygwin" apps are not hit by this restriction and can
access the full 32k character paths with stdio by internally always doing the path
conversion to "super-absolute" paths. I was hit by this when running a Go program
against files that were created by a cygwin program.
@alexbrainman

This comment has been minimized.

Copy link
Member Author

@alexbrainman alexbrainman commented Jul 31, 2013

Comment 12:

We've found a *work-around* for our builder. We didn't have anyone else complain about
this functionality. Given how complicated it might get, lets wait until we see more
complains.
Alex
@kardianos

This comment has been minimized.

Copy link
Contributor

@kardianos kardianos commented Jul 31, 2013

Comment 13:

As far as I can tell, resolving this would require mangling the paths, while recognizing
device and UNC patterns as well. So long as this was done at the same time as the UTF-8
to UTF-16 conversion, this shouldn't introduce any additional mallocs I think.
* If this was implemented, would reviewers want to see an environment variable or an
easy switch to turn it off?
@alexbrainman

This comment has been minimized.

Copy link
Member Author

@alexbrainman alexbrainman commented Jul 31, 2013

Comment 14:

I think it is going to be unnecessary complication. I don't want to see this change.
Only if you have a real life problem.
Alex
@robpike

This comment has been minimized.

Copy link
Contributor

@robpike robpike commented Aug 30, 2013

Comment 15:

Not for 1.2.

Labels changed: removed go1.2.

@robpike

This comment has been minimized.

Copy link
Contributor

@robpike robpike commented Sep 1, 2013

Comment 16:

Issue #6300 has been merged into this issue.

@alexbrainman

This comment has been minimized.

Copy link
Member Author

@alexbrainman alexbrainman commented Sep 2, 2013

Comment 17:

Issue #6300 has been merged into this issue.

@alexbrainman

This comment has been minimized.

Copy link
Member Author

@alexbrainman alexbrainman commented Sep 3, 2013

Comment 18:

Issue #6300 has been merged into this issue.

@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Nov 27, 2013

Comment 19:

Labels changed: added go1.3maybe.

@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Nov 27, 2013

Comment 20:

Labels changed: removed feature.

@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Dec 4, 2013

Comment 21:

Labels changed: added release-none, removed go1.3maybe.

@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Dec 4, 2013

Comment 22:

Labels changed: added repo-main.

@gmlewis

This comment has been minimized.

Copy link
Contributor

@gmlewis gmlewis commented Jan 23, 2014

Comment 23:

I have run into this problem as well on Windows, and believe that I might have a
relatively simple solution.
On this line:
https://code.google.com/p/go/codesearch#go/src/cmd/go/test.go&l=579
testDir is sometimes creating very long directory names for a test.
This becomes a problem when p.ImportPath has been created by this function:
https://code.google.com/p/go/codesearch#go/src/cmd/go/pkg.go&l=212
and could appear like in the example given in the code:
_/c_/home/gopher/my/pkg
My proposed fix would be this:
If p.ImportPath already starts with an underscore, then simply leave it out of the
creation of "testDir" altogether.
So this: https://code.google.com/p/go/codesearch#go/src/cmd/go/test.go&l=579
could look something like this:
    addPath := "_test"
    if !strings.HasPrefix(p.ImportPath, "_") {
        addPath = filepath.FromSlash(p.ImportPath+"/_test")
    }
    testDir := filepath.Join(b.work, addPath)
@alexbrainman

This comment has been minimized.

Copy link
Member Author

@alexbrainman alexbrainman commented Jan 23, 2014

Comment 24:

@gmlewis, please, try and test your suggestion (run all.bat). If it works, send it out
for code review. I tried it long time ago, but failed to come up with a simple solution.
Alex
@iafan

This comment has been minimized.

Copy link

@iafan iafan commented May 9, 2016

Hi there. Just ran across the same issue in #15615 (thanks to @bradfitz for pointing me to this thread).

I don't think it makes sense to "magically" fix the file paths on every syscall, tbh. My concern is, obviously, the speed. I traverse a huge directory tree, and I don't want Go internals to be analyzing and fixing the path deep in the loop, when the proper solution in my case would be to provide a \\?\-prefixed canonical path just once in the call to filepath.Walk().

What makes sense to me is to have some helper function, e.g. filepath.Canonicalize() that, under Windows, would replace all / with \ and prepend \\?\ (or return an error if the provided path is not absolute). This way everyone can use such a function as early as possible (when processing user input or external data), and forget about "if windows then ... else ..." kind of code. The very existence of such a function, along with its documentation, will make other developers aware of OS-specific caveats. Other corresponding FS-related functions might then provide a link in their documentation to that function.

Internal functions that return the file path, like os.Getwd(), os.TempDir() or filepath.Abs() could also benefit from returning a canonicalized path that other functions can safely consume.

@bradfitz

This comment has been minimized.

Copy link
Contributor

@bradfitz bradfitz commented May 9, 2016

@iafan, what matter is being correct and consistent, and providing a simple interface. The speed is a detail which can be fixed. We shouldn't make an ugly API just for speed reasons.

@minux

This comment has been minimized.

Copy link
Member

@minux minux commented May 10, 2016

A compromise solution:

We make all the std library function (exclude syscall) that manipulate
paths always generate and use Cleaned paths so that when necessary, the
user can pass a ?\ style path and all the functions still work as
expected.

We can even document this.

@wpostma

This comment has been minimized.

Copy link

@wpostma wpostma commented Oct 19, 2016

Further activity by Microsoft on Windows 10 updates has rendered some of the underlying MAX_PATH/250-char limit stuff less troublesome.

For your GO app to take advantage of the "I am large path aware" feature in Windows, a manifest file must be attached to your app. It would be nice if Go made adding that manifest easy on Windows.

Manifests are also useful for other purposes like specifying that you require a particular version of some system library, and may be useful to some very small amount of users to have a custom manifest, but a default manifest that is at least marking the app as "large path aware" might be a useful thing to think of doing.

@harshavardhana

This comment has been minimized.

Copy link
Contributor

@harshavardhana harshavardhana commented Oct 19, 2016

FWIW - We have been successfully using a nice wrapper on windows for Minio, same code path works transparently for both Unixes and Windows.

Relevant windows translation code.

// UNCPath converts a absolute windows path to a UNC long path.
func UNCPath(path string) string {
        // Clean the path for any trailing "/".
        path = filepath.Clean(path)

        // UNC can NOT use "/", so convert all to "\".
        path = filepath.FromSlash(path)

        // If prefix is "\\", we already have a UNC path or server.
        if strings.HasPrefix(path, `\\`) {

                // If already long path, just keep it
                if strings.HasPrefix(path, `\\?\`) {
                        return path
                }

                // Trim "\\" from path and add UNC prefix.
                return `\\?\UNC\` + strings.TrimPrefix(path, `\\`)
        }
        path = `\\?\` + path
        return path
}
@alexbrainman

This comment has been minimized.

Copy link
Member Author

@alexbrainman alexbrainman commented Oct 21, 2016

It would be nice if Go made adding that manifest easy on Windows.

This https://github.com/lxn/walk/blob/master/README.mdown#create-manifest-testmanifest is one one way to do it.

Alex

@rsc

This comment has been minimized.

Copy link
Contributor

@rsc rsc commented Oct 24, 2016

Even though Windows 10 will "fix" this, not everyone has it, and not everyone will have it for quite a while.

It seems like we should probably fix it too, by transparently flipping to the ?\ syntax for absolute paths passed to the os calls. This is worth prototyping at least.

@rsc rsc modified the milestones: Go1.8Maybe, Unplanned Oct 24, 2016
@harshavardhana

This comment has been minimized.

Copy link
Contributor

@harshavardhana harshavardhana commented Oct 24, 2016

Even though Windows 10 will "fix" this, not everyone has it, and not everyone will have it for quite a while.

It seems like we should probably fix it too, by transparently flipping to the ?\ syntax for absolute paths passed to the os calls. This is worth prototyping at least

All operations work transparently other than helpers like MkdirAll and RemoveAll which needs a custom wrapper. But it is neatly doable

Should i make a PR?

@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Oct 31, 2016

CL https://golang.org/cl/32451 mentions this issue.

@rsc rsc modified the milestones: Go1.8, Go1.8Maybe Nov 2, 2016
@gopherbot gopherbot closed this in 231aa9d Nov 7, 2016
@gopherbot

This comment has been minimized.

Copy link

@gopherbot gopherbot commented Nov 8, 2016

CL https://golang.org/cl/32886 mentions this issue.

gopherbot pushed a commit that referenced this issue Nov 8, 2016
Notably, don't allocate.

Follow-up to https://golang.org/cl/32451 which added long path
cleaning.

Updates #3358

Change-Id: I89c59cbd660d0a030f31b6acd070fa9f3250683b
Reviewed-on: https://go-review.googlesource.com/32886
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
@cretz

This comment has been minimized.

Copy link

@cretz cretz commented Dec 29, 2016

I have tested this on 1.8 beta 2 and still have issues with large go build calls:

go build github.com/cretz/myproj: C:\dls\go1.8\go\pkg\tool\windows_amd64\compile.exe: fork/exec C:\dls\go1.8\go\pkg\tool\windows_amd64\compile.exe: The filename or extension is too long.

I can open this as a separate bug if that works better.

@bradfitz

This comment has been minimized.

Copy link
Contributor

@bradfitz bradfitz commented Dec 29, 2016

@cretz, please file a separate bug. My guess is that problem is separate, since the magic we added here to the os package doesn't help the syscall package, since syscall (where the exec happens) doesn't use os.

@golang golang locked and limited conversation to collaborators Dec 29, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.