New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: GOPROXY using file:// protocol not working on Windows #27698

Open
chowey opened this Issue Sep 15, 2018 · 9 comments

Comments

Projects
None yet
8 participants
@chowey

chowey commented Sep 15, 2018

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.11 windows/386

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

GOARCH=386
GOOS=windows

What did you do?

git clone https://github.com/pkg/errors c:/repos/github.com/pkg/errors
set GOPROXY=file:///c:/repos
echo package main;import _ "github.com/pkg/errors";func main(){}> main.go
go mod init test
go run .

What did you expect to see?

Successfully imported the package and ran the program. This is documented in cmd/go, where it says:

A Go module proxy is any web server that can respond to GET requests for URLs of a specified form. The requests have no query parameters, so even a site serving from a fixed file system (including a file:/// URL) can be a module proxy.

Also from the module wiki:

Run go help goproxy for more about the module proxy, including a pure file-based option via a file:/// URL.

What did you see instead?

build test: cannot find module for path github.com/pkg/errors

I may be doing something wrong, but the documentation seems to indicate file:/// is supported.

@dmitshur

This comment has been minimized.

Member

dmitshur commented Sep 15, 2018

@magical

This comment has been minimized.

Contributor

magical commented Sep 15, 2018

It looks like you're trying to point GOPROXY at a source tree. That's not how it works. GOPROXY is expected to point at a tree of .zip files and .mod files, not source files. I believe you should be able to point GOPROXY at your module cache, which is $GOPATH/pkg/mod/cache/download, though I haven't tried this myself.

This is all more or less explained in the documentation that you linked: https://golang.org/cmd/go/#hdr-Module_proxy_protocol.

@chowey

This comment has been minimized.

chowey commented Sep 15, 2018

Yes I missed that part:

Even when downloading directly from version control systems, the go command synthesizes explicit info, mod, and zip files and stores them in its local cache, $GOPATH/pkg/mod/cache/download, the same as if it had downloaded them directly from a proxy. The cache layout is the same as the proxy URL space, so serving $GOPATH/pkg/mod/cache/download at (or copying it to) https://example.com/proxy would let other users access those cached module versions with GOPROXY=https://example.com/proxy.

I am specifically trying to use go modules in an environment where there is no direct internet connection. I was hoping that it would be simple, but no such luck.

@chowey

This comment has been minimized.

chowey commented Sep 15, 2018

So I've done more testing. I set GOPATH to e.g. C:\gopath, cached the modules, and tried:

set GOPROXY=file:///C:/gopath/pkg/mod/cache/download
go run .

and that didn't work!

But, after accidentally making a typo, I discovered that THIS works:

set GOPROXY=file://C:/gopath/pkg/mod/cache/download
go run .

Note the two leading slashes instead of three.

Now per my recollection, and Wikipedia supports me here, I'd expect the former to work and the later not to work. So there still seems to be a bug in here somewhere.

@odeke-em odeke-em changed the title from GOPROXY using file:// protocol not working on Windows to x/vgo: GOPROXY using file:// protocol not working on Windows Sep 17, 2018

@gopherbot gopherbot added this to the vgo milestone Sep 17, 2018

@myitcv myitcv changed the title from x/vgo: GOPROXY using file:// protocol not working on Windows to cmd/go: GOPROXY using file:// protocol not working on Windows Sep 17, 2018

@myitcv myitcv modified the milestones: vgo, Go1.12 Sep 17, 2018

@bcmills

This comment has been minimized.

Member

bcmills commented Sep 17, 2018

I am specifically trying to use go modules in an environment where there is no direct internet connection.

FWIW, go mod download is the intended command for that use-case: you should be able to run go mod download on a machine with a network connection, then copy GOPATH/pkg/mod/cache/download to your disconnected builder.

@bcmills

This comment has been minimized.

Member

bcmills commented Sep 17, 2018

https://blogs.msdn.microsoft.com/ie/2006/12/06/file-uris-in-windows/ does seem to support the triple-slash as canonical.

Probably we need to do something more robust to translate the URI path to a file path here:

if strings.HasPrefix(url, "file:") {
body, err := ioutil.ReadFile(req.URL.Path)

(CC @alexbrainman, in case he has any advice on Go idioms to convert Windows URL paths to file paths.)

@bcmills

This comment has been minimized.

Member

bcmills commented Sep 17, 2018

https://play.golang.org/p/5uOiwDwU0n4 for reference on the behavior of http.NewRequest.

@alexbrainman

This comment has been minimized.

Member

alexbrainman commented Sep 22, 2018

https://blogs.msdn.microsoft.com/ie/2006/12/06/file-uris-in-windows/ does seem to support the triple-slash as canonical.

LGTM.

You could also look in net/url package - we already have some tests for urls that starts with file:///C:/.... Just remember that these paths are case insensitive on Windows (just like normal file paths on Windows).

Alex

@rsc

This comment has been minimized.

Contributor

rsc commented Oct 24, 2018

OK, so the URL parser is putting the leading slash in, and on Windows we should take it out if the result passes filepath.IsAbs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment