Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: improve documentation of remote import paths for developers (implementing the go-get protocol) #37867

Open
perillo opened this issue Mar 15, 2020 · 13 comments

Comments

@perillo
Copy link

@perillo perillo commented Mar 15, 2020

What version of Go are you using (go version)?

$ go version
go version go1.14 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN="/home/manlio/.local/bin"
GOCACHE="/home/manlio/.cache/go-build"
GOENV="/home/manlio/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE="*.local"
GONOPROXY=""
GONOSUMDB="*.local"
GOOS="linux"
GOPATH="/home/manlio/.local/lib/go:/home/manlio/src/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build583242652=/tmp/go-build -gno-record-gcc-switches"
GOROOT/bin/go version: go version go1.14 linux/amd64
GOROOT/bin/go tool compile -V: compile version go1.14
uname -sr: Linux 5.5.8-arch1-1
/usr/lib/libc.so.6: GNU C Library (GNU libc) stable release version 2.31.
gdb --version: GNU gdb (GDB) 9.1

What did you do?

I'm implementing an HTTP server supporting the go-get protocol.
The documentation at https://golang.org/pkg/cmd/go/#hdr-Remote_import_paths says:

For example,

import "example.org/pkg/foo"
will result in the following requests:

https://example.org/pkg/foo?go-get=1 (preferred)
http://example.org/pkg/foo?go-get=1  (fallback, only with -insecure)

However this is not how go get behave.

Assuming that, for an importpath test.local/example/a/b/c, both
http://test.local/example/a/b/c?go-get=1 and http://test.local/example/?go-get=1 return the same vcs repository, go get test.local/example/a/b/c will concurrently request all the possible paths.

I was assuming the requests were serialized, thus optimizing the detection of the repository (in case it is a bare repository) so that the first request to http://test.local/example/a/b/c?go-get=1 can return the repository URL, instead of 404 Not Found.

The documentation should be updated, to document how go get behave.

@perillo perillo changed the title cmd/go: improve documentation of remote import paths for developer (implementing the go-get protocol) cmd/go: improve documentation of remote import paths for developers (implementing the go-get protocol) Mar 15, 2020
@dmitshur dmitshur added this to the Backlog milestone Mar 15, 2020
@dmitshur

This comment has been minimized.

Copy link
Member

@dmitshur dmitshur commented Mar 15, 2020

I suspect the behavior may differ depending whether module mode or GOPATH mode is used. The documentation you're quoting there was written for GOPATH mode.

/cc @jayconrod @bcmills @matloob per owners.

@perillo

This comment has been minimized.

Copy link
Author

@perillo perillo commented Mar 15, 2020

Thanks.

What about documenting it in https://golang.org/ref/modules, in the Retrieving modules section?

@jayconrod

This comment has been minimized.

Copy link
Contributor

@jayconrod jayconrod commented Mar 16, 2020

The protocol is the same for GOPATH-mode and for modules in GOPROXY=direct mode. It's a little different when a module proxy is used.

This will be described in https://golang.org/ref/modules (which by the way is about to be renamed to https://golang.org/ref/mod).

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Mar 16, 2020

Hmm, there may be a bug in direct mode: we arguably really should only request the package path (rather than all possible module paths), since the server is already supposed to know the package-to-repository mapping. (We still need to search all modules contained within that repository to figure out which one contains the package, though.)

@perillo

This comment has been minimized.

Copy link
Author

@perillo perillo commented Mar 16, 2020

A direct server (not a proxy) probably don't know the package-to-repository mapping. It may use a cache, however.

@perillo

This comment has been minimized.

Copy link
Author

@perillo perillo commented Mar 16, 2020

With the current behavior of go get, the implementation of the server is more simple, since it can simply discard paths that are not the root of a VCS repository.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Mar 16, 2020

The protocol already does not allow the server to discard paths that are not the root of a VCS repository. (For example, that fails badly for nested modules.)

@perillo

This comment has been minimized.

Copy link
Author

@perillo perillo commented Mar 16, 2020

Where is documented?
The protocol is all about the VCS repository, not Go modules, as far as I know.
From the documentation:

The import-prefix is the import path corresponding to the repository root. 

However it is true that a nested module don't work if the server discard paths that are not the root of a VCS repository.

go get test.local/submodule/sub
go get test.local/submodule/sub: module test.local/submodule@upgrade found (v0.0.0-20200316185411-e1e1e5e813d1), but does not contain package test.local/submodule/sub

The submodule directory contains:

go.mod
main.go
sub/go.mod
sub/sub.go

I find the message confusing, unless I'm doing something wrong.

@dmitshur

This comment has been minimized.

Copy link
Member

@dmitshur dmitshur commented Mar 16, 2020

A direct server (not a proxy) probably don't know the package-to-repository mapping.

I don't understand this point. Given the server needs to serve responses to ?go-get=1 requests, wouldn't it have to know what repository is providing each package?

@perillo

This comment has been minimized.

Copy link
Author

@perillo perillo commented Mar 16, 2020

Sorry; with know I mean to know a priori. So, for go get test.local/submodule/sub, the server does not know a priori that sub is a sub module.

In fact my implementation just return the URI to the repository using the file scheme; it does not check if the repository is a Go module, since it is go get responsibility.

I did check in the past, but it was only a waste of effort if go get requests all the possible paths; and it will make the implementation more complex if I need to check for submodules, especially with bare Git repositories. I was assuming that submodules were handled by go get.

Here is the HTTP handler implementing the go-get protocol: https://gist.github.com/perillo/83645fa5087916cb2f619f9bc2ec3cbf.

Thanks.

@perillo

This comment has been minimized.

Copy link
Author

@perillo perillo commented Mar 17, 2020

I have improved the HTTP handler so that for each request it finds the Git repository containing the import path, instead of returning 404 Not Found if the import path don't identify the root of the Git repository.

For a module containing

go.mod
main.go
sub/go.mod
sub/pkg/pkg.go
sub/sub.go
$ go get test.local/submodule
go: downloading test.local/submodule v0.1.0
go: test.local/submodule upgrade => v0.1.0
$ go get test.local/submodule/sub
go: downloading test.local/submodule/sub v0.3.0
go: test.local/submodule/sub upgrade => v0.3.0
$ go get test.local/submodule/sub/pkg
go get test.local/submodule/sub/pkg: module test.local/submodule/sub@upgrade found (v0.3.0), but does not contain package test.local/submodule/sub/pkg

For all the request done by go get, the returned metadata is always the same:

<meta name="go-import"
      content="test.local/submodule git file:///home/manlio/src/go/src/test.local/submodule">
</meta>

What is the reason for this behavior? Since the import-prefix is not test.local/submodule/sub it should probably report an error.

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented Mar 17, 2020

The import-prefix portion indicates the path to the root of the repo, not the path to the module root.

@perillo

This comment has been minimized.

Copy link
Author

@perillo perillo commented Mar 17, 2020

I tried again by explicitly requesting the latest revision @HEAD and now go get test.local/submodule/sub/pkg@HEAD works correctly. Probably before go get was using a cached copy.

Thanks for your time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
You can’t perform that action at this time.