Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

path/filepath: Walk on very long paths fails on Windows #21782

Open
sandreas opened this issue Sep 6, 2017 · 18 comments
Open

path/filepath: Walk on very long paths fails on Windows #21782

sandreas opened this issue Sep 6, 2017 · 18 comments
Labels
help wanted NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Windows
Milestone

Comments

@sandreas
Copy link

sandreas commented Sep 6, 2017

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

1.9

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

set GOARCH=amd64
set GOBIN=
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=C:\Users\mediacenter\go
set GORACE=
set GOROOT=C:\Go
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0
set CXX=g++
set CGO_ENABLED=1
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config

What did you do?

Executing the sample code on a current windows 10 machine fails with an error - seems that running filepath.Walk on very long paths fails on windows in general.

Sample code, which works on play: https://play.golang.org/p/wMS5gti4SD
But it DOES NOT WORK on windows systems!

What did you expect to see?

correctly walk over..../package.json

What did you see instead?

ERROR ON PATH node_modules\babel-preset-es2015\node_modules\babel-plugin-transform-es2015-block-scoping\node_modules\babel-traverse\node_modules\babel-code-frame\node_modules\chalk\node_modules\strip-ansi\node_modules\ansi-regex\package.json: GetFileAttributesEx node_modules\babel-preset-es2015\node_modules\babel-plugin-transform-es2015-block-scoping\node_modules\babel-traverse\node_modules\babel-code-frame\node_modules\chalk\node_modules\strip-ansi\node_modules\ansi-regex\package.json: Das System kann den angegebenen Pfad nicht finden.

Can anyone confirm this behaviour?

@mvdan
Copy link
Member

mvdan commented Sep 6, 2017

This seems similar to #20829. Perhaps there's another part of the standard library that needs this fix for long paths.

@mvdan mvdan added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Windows labels Sep 6, 2017
@odeke-em odeke-em changed the title Running filepath.Walk on very long paths fails on windows path/filepath: Walk on very long paths fails on Windows Sep 6, 2017
@odeke-em
Copy link
Member

odeke-em commented Sep 7, 2017

/cc @alexbrainman @hirochachacha

@alexbrainman
Copy link
Member

This fails in os.Lstat(path), and path is relative path here. The os.Lstat is using os.fixLongPath, which has been created just for such purpose. But os.fixLongPath does not handle relative paths, and for a good reason - relative paths cannot be longer than 255 characters (see https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#maxpath for details). I do not know how to fix this. Perhaps @quentinmit has some suggestions - he wrote os.fixLongPath.

Alex

@sandreas
Copy link
Author

sandreas commented Sep 7, 2017

I'm not sure, if this is possible, but i think os.Lstat returns the same value for absolute and relative paths. Is that correct?

If so, why not convert all paths to absolute paths (e.g. with filepath.Abs) in os.Lstat on windows before using the syscall?

@alexbrainman
Copy link
Member

why not convert all paths to absolute paths (e.g. with filepath.Abs) in os.Lstat on windows before using the syscall?

Maybe we'll do that. I am not sure yet.

Alex

@sandreas
Copy link
Author

sandreas commented Sep 8, 2017

As a short feedback: I implemented a simple filesystem abstraction based on afero (https://github.com/spf13/afero) and the use filepath.Abs did the job for me.

I used following logic:
If the path is not absolute and it is longer than 200 chars and it is a windows system, apply filepath.Abs before any other operation.

Why 200 chars, not 255?
Because a relative path containing 227 chars led to an error, while less than 200 did not. I don't know the reason but 255 did not work. To exclude unicode errors, i tried to use special chars in filename (äöü€), but the 227 bound was still valid. The slash-count is 14, 255-14 = 241, so diff is also not the amount of slashes of the relative path.

The problem is, that in later testing i noticed, that there errors occurred with some paths lower than 227 chars, so i corrected it down to 200. Would be nice to know the exact bound, but i didn't find out.

The path with less than 255 chars, which caused the first error was:
relative:
node_modules\babel-preset-es2015\node_modules\babel-plugin-transform-es2015-block-scoping\node_modules\babel-traverse\node_modules\babel-code-frame\node_modules\chalk\node_modules\strip-ansi\node_modules\ansi-regex\package.json

absolute:
C:\inetpub\wwwroot\irgendwas_4.0\node_modules\babel-preset-es2015\node_modules\babel-plugin-transform-es2015-block-scoping\node_modules\babel-traverse\node_modules\babel-code-frame\node_modules\chalk\node_modules\strip-ansi\node_modules\ansi-regex\package.json

@quentinmit
Copy link
Contributor

why not convert all paths to absolute paths (e.g. with filepath.Abs) in os.Lstat on windows before using the syscall?

Unfortunately, it's very nontrivial to convert relative paths to absolute paths on Windows. I already went down that road and gave up on trying to make it work. syscall.FullPath, which is what filepath.Abs calls, is also only supported on short paths. What makes this hard is that Windows has a separate working directory on each drive, but getcwd and friends only return one of them. e.g. "d:foo.txt" is a relative path that's relative to whatever the current directory is on the D drive.

If you need to Walk on long paths, you should just call Walk with an absolute path. I believe everything inside Walk should work correctly in that case.

@sandreas
Copy link
Author

sandreas commented Sep 15, 2017

I already fixed it and for me it is working prepending a \\?\. You said that filepath.Abs only will work for short paths. In my case it does work perfectly with long paths. I'm not saying it is working in every case, but for now i did not find a path (including some node modules directories), that did not work. Code sample (taken from my project https://github.com/sandreas/graft/blob/master/filesystem/osfs_windows.go):

func makeAbsolute(name string) string {
	absolutePath, err := filepath.Abs(name)
	if err == nil {
		if strings.HasPrefix(absolutePath, `\\?\UNC\`) || strings.HasPrefix(absolutePath, `\\?\`) {
			return absolutePath
		}

		if strings.HasPrefix(absolutePath, `\\`) {
			return strings.Replace(absolutePath, `\\`, `\\?\UNC\`, 1)
		}

		return `\\?\` + absolutePath
	}
	return name
}

@alexbrainman
Copy link
Member

I already went down that road and gave up on trying to make it work. syscall.FullPath, which is what filepath.Abs calls, is also only supported on short paths.

@quentinmit I just tried to test syscall.FullPath:

c:\Users\Alex>type main.go
package main

import (
        "fmt"
        "log"
        "os"
        "path/filepath"
        "syscall"
)

func makeLongPath() string {
        const file = "abcdefghij1234567890"
        var s string
        for i := 0; i < 13; i++ {
                s = filepath.Join(s, file)
        }
        return s
}

func testPath(path string) {
        fullpath, err := syscall.FullPath(path)
        if err != nil {
                log.Fatal(err)
        }
        fmt.Printf("\nFullPath(%q) is %v\n", path, fullpath)
}

func main() {
        err := os.Chdir(`U:\`)
        if err != nil {
                log.Fatal(err)
        }
        longpath := makeLongPath()
        testPath(`C:` + longpath)
        testPath(`C:\` + longpath)
        testPath(`U:` + longpath)
        testPath(`U:\` + longpath)
        testPath(longpath)
}

c:\Users\Alex>go run main.go

FullPath("C:abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890") is C:\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890

FullPath("C:\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890") is C:\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890

FullPath("U:abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890") is U:\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890

FullPath("U:\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890") is U:\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890

FullPath("abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890\\abcdefghij1234567890") is U:\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890\abcdefghij1234567890

c:\Users\Alex>

and it seems to be working. It works on both Windows 10 and Windows XP.

Alex

@networkimprov
Copy link

@gopherbot add "help wanted"

@networkimprov
Copy link

networkimprov commented Jan 27, 2020

@quentinmit what do you think of making os.fixLongPath() return an error for a path over the max length that doesn't start with \\?\ ?

EDIT: this says we can prepend \\?\ to the result of GetFullPathNameW() to create a valid long path: https://stackoverflow.com/questions/38036943/getfullpathnamew-and-long-windows-file-paths

cc @iwdgo

@sandreas
Copy link
Author

sandreas commented Jan 27, 2020

@networkimprov
See https://github.com/sandreas/afero/blob/master/os_windows.go for an implementation that works for me.

@networkimprov
Copy link

networkimprov commented Jan 27, 2020

@sandreas, according to #21782 (comment) syscall.FullPath doesn't prefix paths on Windows with \\?\ (required for long paths). Has that changed? If not, your code only fixes UNC paths.

Your file is missing import btw.
EDIT: And it's not what you posted in #21782 (comment)

@sandreas
Copy link
Author

@networkimprov

according to #21782 (comment) syscall.FullPath doesn't prefix paths on Windows with \?\ (required for long paths). Has that changed? If not, your code only fixes UNC paths.
Your file is missing import btw.
EDIT: And it's not what you posted in #21782 (comment)

You're absolutely right... The code i posted has (accidentally) not been tested on a windows system, where it would belong to - so it is incomplete. I'm sorry that this did not help.

@networkimprov
Copy link

I've suggested a fix in #36375 (comment). Both that issue and this one should be fixed in the same CL.

@gopherbot
Copy link

Change https://golang.org/cl/263538 mentions this issue: os: on Windows, implement fixLongPath also for relative paths

@gopherbot
Copy link

Change https://golang.org/cl/291291 mentions this issue: os: support long paths without fixup on Windows 10 >= 1607

@sandreas
Copy link
Author

I would like to mention a highly opinonated article about this topic: https://fasterthanli.me/articles/i-want-off-mr-golangs-wild-ride
Although it may have its caveats its worth reading...

I would suggest a Path type with ToString and another PathMetaData type to solve this problem on another layer...

gopherbot pushed a commit that referenced this issue Mar 23, 2021
Windows 10 >= 1607 allows CreateFile and friends to use long paths if
bit 0x80 of the PEB's BitField member is set.

In time this means we'll be able to entirely drop our long path hacks,
which have never really worked right (see bugs below). Until that point,
we'll simply have things working well on recent Windows.

Updates #41734.
Updates #21782.
Updates #36375.

Change-Id: I765de6ea4859dd4e4b8ca80af7f337994734118e
Reviewed-on: https://go-review.googlesource.com/c/go/+/291291
Trust: Jason A. Donenfeld <Jason@zx2c4.com>
Run-TryBot: Jason A. Donenfeld <Jason@zx2c4.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Windows
Projects
None yet
Development

No branches or pull requests

8 participants