Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

syscall: Go Build Fails on Windows With Long Paths #18468

Closed
cretz opened this issue Dec 29, 2016 · 53 comments
Closed

syscall: Go Build Fails on Windows With Long Paths #18468

cretz opened this issue Dec 29, 2016 · 53 comments

Comments

@cretz
Copy link

@cretz cretz commented Dec 29, 2016

Opening by request in #3358

What version of Go are you using (go version)?

go version go1.8beta2 windows/amd64

What operating system and processor architecture are you using (go env)?

Win64

What did you do?

Tried to compile several files, many with long names, in a folder via go build

What did you expect to see?

Successful compile

What did you see instead?

go build github.com/cretz/myproj: C:\dls\go1.8\go\pkg\tool\windows_amd64\compile.exe: fork/exec C:\dls\go1.8\go\pkg\tool\windows_amd64\compile.exe: The filename or extension is too long.
@bradfitz bradfitz changed the title Go Build Fails on Windows With Long Paths syscall: Go Build Fails on Windows With Long Paths Dec 29, 2016
@bradfitz bradfitz added this to the Go1.9 milestone Dec 29, 2016
@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Dec 29, 2016

My guess is that problem is that the magic we added to the os package doesn't help the syscall package, since syscall (where the exec happens) doesn't use os.

Maybe we need to move the os package's support routines (building the UNC paths) into an internal support package usable by both os and syscall.

/cc @quentinmit

@bradfitz bradfitz added the OS-Windows label Dec 29, 2016
@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Dec 29, 2016

And /cc @alexbrainman

@cretz
Copy link
Author

@cretz cretz commented Dec 29, 2016

@bradfitz - I think this would be handled by the caller of Exec instead of in syscall directly since they are opaque strings and the difference between a file path and any other arbitrary string is determined by the caller. I do agree that this and the os package should share the same path-fixing code. I also agree it should be an internal function for now until more thought is given to expose (maybe os/exec.OSSafePath but it's really annoying to abstract a Windows -only issue).

@harshavardhana
Copy link
Contributor

@harshavardhana harshavardhana commented Dec 29, 2016

Why not have filepath.LongPath() function to be used for these situations?

@kardianos
Copy link
Contributor

@kardianos kardianos commented Dec 29, 2016

@harshavardhana If the go std lib handles windows long filepaths, which I think is great, I don't think it would be good to expose it anywhere except (maybe) in syscall, as this is a really platform specific issue where the fix doesn't need to affect API.

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Dec 29, 2016

@harshavardhana
Copy link
Contributor

@harshavardhana harshavardhana commented Dec 29, 2016

@harshavardhana If the go std lib handles windows long filepaths, which I think is great, I don't think it would be good to expose it anywhere except (maybe) in syscall, as this is a really platform specific issue where the fix doesn't need to affect API.

👍

@quentinmit
Copy link
Contributor

@quentinmit quentinmit commented Dec 29, 2016

We very consciously did not make syscall transparently handle long paths, because it would be confusing if the resulting syscall was called with different arguments than the caller intended.

Probably the right thing to do is to teach os/exec to use the same long path handling as os.

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Dec 29, 2016

Actually, os/exec just uses os.StartProcess. So this is in the os package after all.

I'll send out a fix.

@gopherbot
Copy link

@gopherbot gopherbot commented Dec 29, 2016

CL https://golang.org/cl/34725 mentions this issue.

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Dec 29, 2016

Not trivial, it turns out. Somebody should tackle it during Go 1.9. https://golang.org/cl/34725 might be a good starting point. It has some debugging notes in it and a test.

@kardianos
Copy link
Contributor

@kardianos kardianos commented Dec 30, 2016

If I was to guess, the UNC meses with the quote logic; guess only.

@ALTree
Copy link
Member

@ALTree ALTree commented Jul 24, 2017

Note: #21145, just closed as a dup of this issue, has an handy Powershell script that creates a folder that triggers the problem.

@kardianos
Copy link
Contributor

@kardianos kardianos commented Jul 24, 2017

After reviewing #21145 I think the CL that Brad had started ( https://golang.org/cl/34725 ) is barking up the wrong tree. In that CL it attempts to fixup the process name to use the UNC path equivalent that the rest of the os functions can use. However, as noted in the CL, CreateProcess doesn't accept UNC paths for the process name.

Looking at this issue again the process name is short C:\dls\go1.8\go\pkg\tool\windows_amd64\compile.exe and less then 250 chars. This means that no long path fixup would be applied even if it could work.

I suspect that the error is actually referring to the argument length (Windows doesn't really differentiate between the two in error message).

From https://msdn.microsoft.com/en-us/library/windows/desktop/ms682425(v=vs.85).aspx

lpCommandLine [in, out, optional]
The command line to be executed. The maximum length of this string is 32,768 characters, including the Unicode terminating null character. If lpApplicationName is NULL, the module name portion of lpCommandLine is limited to MAX_PATH characters.

Two ways to fix this would be to instrument the Go compilers to take a new argument "-args-on-stdin=true" flag where the exec would read from stdin for args or pass the argument list file to the executable "@C:\Temp\arglist" where the given file is read off of disk and used as arguments.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Jul 25, 2017

It would be nice to solve the general problem, but perhaps Windows does not provide a mechanism for that. If we can only solve the problem for cmd/go, then of course we should do that.

For the record, GCC and friends do this by treating any argument that starts with '@' as a response file. The file is turned into a sequence of white-space separated strings, and the response file argument is replaced by the resulting set of arguments. This seems to be a standard Windows concept, documented at https://msdn.microsoft.com/en-us/library/windows/desktop/aa367156(v=vs.85).aspx .

@ellismg
Copy link

@ellismg ellismg commented Sep 28, 2017

I’m running into this issue and I’d like to contribute a fix.

If the maintainers would like me to teach the toolchain about response files, I’m happy to take a swing at it, but I have some questions:

  1. It seems like it may be worthwhile to teach the flag package how to understand response files. If that is reasonable, my approach would be when looking for flags, also treat strings that start with @ specially. Is it reasonable to make that change or would it need to be guarded behind some check to ensure existing invocations which pass strings start start with @ in a position where a flag would be legal continue to work? It so, I assume adding a new function like AllowResponseFile(bool) is the way to go?
  2. How do you want to do word splitting in the response file? Would we want to support multiple arguments per line in the response file or say that each argument is delimited by a newline. Regardless of what we choose, how does someone escape a character that would cause word breaking?
  3. If adding this feature to the flag package is unacceptable should I just teach the compiler a new argument like -sourcefiles ?
@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented Sep 28, 2017

@ellismg before deciding how to fix this, can you provide me with steps to reproduce it.

Thank you

Alex

@ellismg
Copy link

@ellismg ellismg commented Sep 28, 2017

@alexbrainman

Sure. In the case I'm hitting, the problem is just that we construct a command line that is too long (>32K characters) when we try to exec into the compiler.

This works:

C:\Users\Matt G. Ellis\go\src\github.com\ellismg\scratch>dir /B
empty.go
hello.go

C:\Users\Matt G. Ellis\go\src\github.com\ellismg\scratch>type empty.go
package main
C:\Users\Matt G. Ellis\go\src\github.com\ellismg\scratch>type hello.go
package main

import "fmt"

func main() {
    fmt.Printf("hello, world\n")
}
C:\Users\Matt G. Ellis\go\src\github.com\ellismg\scratch>go build

C:\Users\Matt G. Ellis\go\src\github.com\ellismg\scratch>

If you add a bunch of additional files (depending on how deep your source code is, you may need more copies of the dummy file, to construct a command line that is over the windows limit):

C:\Users\Matt G. Ellis\go\src\github.com\ellismg\scratch>for /l %i IN (1,1,2048) DO copy /y empty.go %i.g
o
[...lots of copy spew elided...]

Then go build will fail when trying to run compiler.exe

C:\Users\Matt G. Ellis\go\src\github.com\ellismg\scratch>go build
go build github.com/ellismg/scratch: C:\Go\pkg\tool\windows_amd64\compile.exe: fork/exec C:\Go\pkg\tool\w
indows_amd64\compile.exe: The filename or extension is too long.

Here's my system information:

C:\Users\Matt G. Ellis\go\src\github.com\ellismg\scratch>go version
go version go1.9 windows/amd64

C:\Users\Matt G. Ellis\go\src\github.com\ellismg\scratch>go env
set GOARCH=amd64
set GOBIN=
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=C:\Users\Matt G. Ellis\go
set GORACE=
set GOROOT=C:\Go
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0
set CXX=g++
set CGO_ENABLED=1
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config

Locally, I did a hacky thing where I just taught compiler.exe about a -srcfiles argument that would read a response file and then passed every source file via that list. With that set of changes, I was able to build.

Nothing long path specific here, as @kardianos surmised. Just generating a very long command that ends up being over a Windows limit.

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Sep 28, 2017

Something like flag.(*FlagSet).ResponseFiles(bool) makes sense to me, with flag.ResponseFiles calling that method on CommandLine.

The response files should be parsed exactly as the Microsoft tools parse them. I believe they permit multiple options on a single line, and permit quoting with backslash, single quote, and double quote.

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Apr 30, 2018

@arizvisa, thanks for testing and noting the .exe bug. I've uploaded a new version and also using it with "link".

arizvisa added a commit to arizvisa/golang that referenced this issue May 1, 2018
…ments into a response file if the total length of all the args is larger than the limitation imposed by windows. Also Modified runOut to use this when executing a sub-command.

Closes issue golang#18468
@arizvisa
Copy link
Contributor

@arizvisa arizvisa commented May 1, 2018

@bradfitz, hey man. I noticed that one of the reviewers of your patch suggested the idea of going the path of response files. I made a branch that goes down this route, and adds '@'-prefixed response files to the same place that your patch added your fix.

I essentially re-implemented the buildargv function (from libiberty in gcc) in golang, but at the moment it doesn't do recursive response files which I believe gnu-tools supports. It also doesn't explicitly test the tool that's being dispatched to as your patch does.

At the moment its dev'd against go1.10.1. If the community prefers this solution for some reason, though, I can rebase it against master and create a PR so it can be merged upstream. However, I could use another pair of eyes to sanity check how I'm escaping things when writing to the response file (since it's pretty common to have mistakes here).

The branch I'm referring to is at arizvisa/golang#GH-18468

arizvisa added a commit to arizvisa/golang that referenced this issue May 1, 2018
arizvisa added a commit to arizvisa/golang that referenced this issue May 1, 2018
…ments into a response file if the total length of all the args is larger than the limitation imposed by windows. Also Modified runOut to use this when executing a sub-command.

Closes issue golang#18468
arizvisa added a commit to arizvisa/golang that referenced this issue May 1, 2018
@arizvisa
Copy link
Contributor

@arizvisa arizvisa commented May 1, 2018

Actually, another update. I didn't realize go1.10.2 came out, so I rebased it for that tag. lmk..

arizvisa added a commit to arizvisa/golang that referenced this issue May 1, 2018
arizvisa added a commit to arizvisa/golang that referenced this issue May 1, 2018
This adds support for parsing response files to all tools.

This was done by re-implementing the buildargv func from gcc's
libiberty/argv.c and then updating Flagparse to look for '@'-
prefixed filenames. If one is found, then buildargv is used to
expand the command line args from the response file into the
tool's arguments.

On windows, there's a space limitation for commandline args
due to windows internally using UNICODE_STRING which has a 32k
limit. Adding support for response files allows compile and link
to act on packages that have more than 32k worth of filenames.

This includes a capability that is necessary to close golang#18468.

src/cmd/internal/objabi/flag.go: added buildargv and updated Flagparse
arizvisa added a commit to arizvisa/golang that referenced this issue May 1, 2018
This modifies golang's tools to check if the total number of
arguments will result in a commandline larger than 32k. If
we're on windows, this will then replace the commandline args
with a response file which should bypass the 32k limitation.

This was done by adding a function that copies the arguments
into a response file if the total length of the args is larger
than the windows limit. Each of the arguments written to the
response file are newline delimited, double-quoted, and escaped.

Closes issue golang#18468.

src/cmd/go/internal/work/exec.go: added buildLongCommand and escapeString
@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented May 1, 2018

@arizvisa if you want your change looked at, you have to use tools that this this repo uses - https://golang.org/doc/contribute.html on how to contribute. Or just create a PR here on master branch. It was @ianlancetaylor who suggested to use response files on https://golang.org/cl/110395. If you post your change here, maybe it will be prefered to CL 110395.

Alex

@arizvisa
Copy link
Contributor

@arizvisa arizvisa commented May 1, 2018

@alexbrainman, I don't want to step on bradfitz's toes as it was his original fix, I was just offering some direction as it was based on his original work and his response to my complaining. It works for me, but I also don't really want to maintain it or anything ;-) as I'm not really a developer.

Plus, I had realized that I implemented it from a GPL-licensed piece of software which I'm pretty sure is a licensing conflict.

@alexbrainman
Copy link
Member

@alexbrainman alexbrainman commented May 1, 2018

It works for me,

Sounds good to me. I gave Brad +2 now, so he can submit his CL.

I had realized that I implemented it from a GPL-licensed piece of software which I'm pretty sure is a licensing conflict.

It could well be.

Alex

@gopherbot gopherbot closed this in 17fbb83 May 4, 2018
@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented May 4, 2018

I've submitted 17fbb83 (https://go-review.googlesource.com/c/go/+/110395). @arizvisa, please double check that it works for you?

I at least made it such that when it runs on our build farm, 10% of the compile/link invocations go through this code path, so it will get some exercise even for non-large build arguments.

@arizvisa
Copy link
Contributor

@arizvisa arizvisa commented May 5, 2018

@bradfitz, sorry to get back to you so late as I've been ill for a bit.

It seems to work without issue. So, thanks!

One thing not too important but worth thinking about in the future, the way you split lines in expandArgs just culls out the <CR>. I know it's dickheadish, but <CR> is actually a valid character in a filename. Regardless of that weirdness tho, bufio.Scanner might be able to be used in order to scan for newlines more effectively. That was the reason why I ripped buildargv out of libiberty anyways.

Regardless though. Thanks for getting this fixed!

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented May 5, 2018

<CR> is not really a valid character in a Windows filename per https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx which says...

Characters whose integer representations are in the range from 1 through 31, except for alternate data streams where these characters are allowed.

I don't think we need to care about "alternate data streams", which I assume are like Mac "resource forks"?

@arizvisa
Copy link
Contributor

@arizvisa arizvisa commented May 6, 2018

Ah, the api call suceeds. (Although I'm using touch to avoid writing a program).

[528] user@skinny ~/build/go1.10.2$ uname -a
MSYS_NT-6.3 skinny 2.10.0(0.325/5/3) 2018-02-09 15:25 x86_64 Msys
[529] user@skinny ~/build/go1.10.2$ touch $'hola\nmundo.'
[530] user@skinny ~/build/go1.10.2$ ls -l $'hola\nmundo.'
-rw-r--r-- 1 user None 0 May  6 16:02 'hola'$'\n''mundo.'
[531] user@skinny ~/build/go1.10.2$ echo hola*
hola
mundo.

Result of the file in the old cmd prompt.

C:\users\user\build\go1.10.2\src>dir /x ..\hola*
 Volume in drive C has no label.
 Volume Serial Number is FA87-0BF0

 Directory of C:\users\user\build\go1.10.2

05/06/2018  04:02 PM                 0 HOLAMU~1     hola?mundo.
               1 File(s)              0 bytes
               0 Dir(s)  13,513,748,480 bytes free

Then a streams which isn't related, plus I couldn't get a newline added as a stream (even using the api)

C:\users\user\build\go1.10.2>echo hi > stream:woot
C:\users\user\build\go1.10.2>dir /r stream
 Volume in drive C has no label.
 Volume Serial Number is FA87-0BF0

 Directory of C:\users\user\build\go1.10.2

05/06/2018  04:12 PM                 0 stream
                                     5 stream:woot:$DATA
               1 File(s)              0 bytes
               0 Dir(s)  13,508,710,400 bytes free

I'm not really a mac guy so my exp w/ resource forks are pretty spotty. but alternate data streams are typically used for things like metadata as they have no particular data structure. NTFS exposes some interesting default ADS which one can use to get some pretty interesting side effects out of software on windows. But as an example for a legitimate thing, browsers will add an ADS to a downloaded file so that a file can be noted as being "unsafe".

But regardless about platform differences/semantics, I'm just happy things are fixed and I just have tendencies to note side effects about things which is what that stripping of <CR> looked like to me. :)

The implementation of buildargv that I ripped attempted to parse out these things by handling quoting and backslash-escapes. However in order to use that implementation effectively, when I had to build the response file I spent most of the time trying to figure out how to properly quote, escape, and separate each argument in the response file so that buildargv would properly decode those characters that were explicitly specified which allows for their compiler to work independently of filesystem contraints. That's how the gnu folk handle it anyways.

Again though, thanks for fixing this!

(edited to fix some type-o's and the missing backticks around CR)

@simchev
Copy link

@simchev simchev commented May 8, 2018

Is this supposed to be fixed in 1.10.2? I just upgraded to the latest version and I'm still getting the error.
Edit: I just saw the 1.11 milestone, how can I have access to this fix before the release?

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented May 8, 2018

Edit: I just saw the 1.11 milestone, how can I have access to this fix before the release?

You could patch it in and recompile Go if it's urgent.

But I suppose we could also backport it to Go 1.10.x if we do another one.

@gopherbot, please backport.

@gopherbot
Copy link

@gopherbot gopherbot commented May 8, 2018

Backport issue(s) opened: #25292 (for 1.10).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.