Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: AWS CodeCommit repo URL incompatible with vgo major version import paths? #25792

Closed
computermouth opened this issue Jun 8, 2018 · 15 comments

Comments

Projects
None yet
6 participants
@computermouth
Copy link

commented Jun 8, 2018

What version of Go are you using (go version)?

$ vgo version
go version go1.10.3 linux/amd64 vgo:2018-02-20.1

Does this issue reproduce with the latest release?

Yes. I was initially reporting the issue on go1.10, but the issue persists on latest.

What operating system and processor architecture are you using (go env)?

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/byoung/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/byoung/go"
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build461856878=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Created a private repository on AWS CodeCommit for a library.

$ cat go.mod 
module git-codecommit.us-east-1.amazonaws.com/v1/repos/return2

$ cat main.go 
package return2

func Doit() int {
	return 2
}

Created a program to use it.

$ cat go.mod 
module git-codecommit.us-east-1.amazonaws.com/v1/repos/test

$ cat main.go 
package main

import "fmt"
import "git-codecommit.us-east-1.amazonaws.com/v1/repos/return2"

func main(){
	fmt.Println(return2.Doit())

}

Created an ssh redirect in my ~/.gitconfig

[url "ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/"]
        insteadOf = https://git-codecommit.us-east-1.amazonaws.com/v1/repos/

Included this library

What did you expect to see?

Direct clones work just fine. Neither go get nor vgo build are able to import:

$ git clone https://git-codecommit.us-east-1.amazonaws.com/v1/repos/return2
Cloning into 'return2'...
remote: Counting objects: 19, done.
Receiving objects: 100% (19/19), 1.67 KiB | 0 bytes/s, done.
Resolving deltas: 100% (1/1), done.

$ go get git-codecommit.us-east-1.amazonaws.com/v1/repos/return2
package git-codecommit.us-east-1.amazonaws.com/v1/repos/return2: unrecognized import path "git-codecommit.us-east-1.amazonaws.com/v1/repos/return2" (parse https://git-codecommit.us-east-1.amazonaws.com/v1/repos/return2?go-get=1: no go-import meta tags ())

$ vgo build
vgo: resolving import "git-codecommit.us-east-1.amazonaws.com/v1/repos/return2"
vgo: import "github.com/computermouth/runret" ->
	import "git-codecommit.us-east-1.amazonaws.com/v1/repos/return2": unknown module git-codecommit.us-east-1.amazonaws.com/v1/repos/return2: no go-import tags

What did you see instead?

Everything works flawlessly with private repos on Gitlab:

$ git clone https://gitlab.com/computermouth/return2
Cloning into 'return2'...
remote: Counting objects: 16, done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 16 (delta 1), reused 0 (delta 0)
Receiving objects: 100% (16/16), done.
Resolving deltas: 100% (1/1), done.

$ go get gitlab.com/computermouth/return2

$ vgo build
vgo: resolving import "gitlab.com/computermouth/return2"
vgo: finding gitlab.com/computermouth/return2 (latest)
vgo: adding gitlab.com/computermouth/return2 v0.0.0-20180607214308-f8814bf22457
vgo: finding gitlab.com/computermouth/return2 v0.0.0-20180607214308-f8814bf22457
vgo: downloading gitlab.com/computermouth/return2 v0.0.0-20180607214308-f8814bf22457

I have a sneaking suspicion that the cause is actually based in the way that AWS handles git authentication.

[url "ssh://git@gitlab.com/"]
        insteadOf = https://gitlab.com/
[url "ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/"]
        insteadOf = https://git-codecommit.us-east-1.amazonaws.com/v1/repos/

Note that the codecommit urls skip the git@. I presume that since I have yet to see any kind of restrictions in regards to repository names, and since my user/org is not accounted for in the repo URL, that the repository is presented to the user after some Amazon authentication middleware.

But I thought I'd post this here anyways.

Love vgo, it's the package management tool Go has always needed, thanks for all your work!

-- Ben

@gopherbot gopherbot added this to the vgo milestone Jun 8, 2018

@ziyel

This comment has been minimized.

Copy link

commented Jun 8, 2018

The https://git-codecommit.us-east-1.amazonaws.com/v1/repos/return2?go-get=1 need to return meta tag to let the vgo identify the module.
You can refer to the blog:

Similar to current go get, vgo expects a page with a <meta> tag to help translate from a module name to the tree of files for that module. For example, to look up swtch.com/testmod, the vgo command fetches the usual page:

$ curl -sSL 'https://swtch.com/testmod?go-get=1'
<!DOCTYPE html>
<meta name="go-import" content="swtch.com/testmod mod https://storage.googleapis.com/gomodules/rsc">
Nothing to see here.
$ 
@myitcv

This comment has been minimized.

Copy link
Member

commented Jun 8, 2018

@computermouth just as @ziyel mentioned, this is all happening pre-git being invoked.

Also take a look at https://golang.org/cmd/go/#hdr-Remote_import_paths.

I'll close this for now because I don't think there's a bug/issue per se, particularly not with vgo (the go-import meta-tag custom import paths is a general feature of the go tool).

@myitcv myitcv closed this Jun 8, 2018

@computermouth

This comment has been minimized.

Copy link
Author

commented Jun 12, 2018

Interesting, I've never seen the vcs type used in the imports before.

$ vgo build
vgo: resolving import "git-codecommit.us-east-1.amazonaws.com/v1/repos/return2.git"
vgo: import "git-codecommit.us-east-1.amazonaws.com/v1/repos/testtest" ->
	import "git-codecommit.us-east-1.amazonaws.com/v1/repos/return2.git": unknown module git-codecommit.us-east-1.amazonaws.com/v1/repos/return2.git: no go-import tags

$ go get git-codecommit.us-east-1.amazonaws.com/v1/repos/return2.git
(succeeds)

Having changed all the path references to use return2.git instead of return2. The go get yields a normal checkout and build, but vgo still reports the same error.

So I can only assume that vgo either has different behaviour that I'm still not understanding, or this is a bug, as this normally works with the go tool.

@myitcv

This comment has been minimized.

Copy link
Member

commented Jun 13, 2018

@computermouth

I've never seen the vcs type used in the imports before

Just to confirm, what do you mean by "the vcs type"?

Quoting from the import paths link:

The vcs is one of "git", "hg", "svn", etc,

Back to the issue you're having:

Having changed all the path references to use return2.git instead of return2. The go get yields a normal checkout and build, but vgo still reports the same error.

Please can you include here the HTML <meta> tags that are returned?

@computermouth

This comment has been minimized.

Copy link
Author

commented Jun 13, 2018

By vcs type I meant adding the .git or .svn.

After reading the linked pages, it sounded like adding the .git would be the solution to hitting a provider that doesn't provide the meta tags.

Following the example from @ziyel :

$ curl -sSL 'https://git-codecommit.us-east-1.amazonaws.com/v1/repos/return2?go-get=1'
<NotAuthorizedException>
  <Message>SPNEGO token required</Message>
</NotAuthorizedException>

As I mentioned before, CodeCommit has no organization/user namespacing in the URL. As such, if you were to open an account, and create a git repo called return2 on us-east-1, you would clone from the same URL. All repositories are private, and so AWS serves different content from that URL based on the credentials used when requesting the git contents. So it would make sense that this curl request does not give the desired response.

You mentioned that this happens before any git processing starts, so I was under the impression that adding the .git skips this request for the <meta> tags, and goes directly to the git processing, where git would request the files using my git helper cached credentials.

So it seemed to me that the following was kind of an oxymoron, where the .git suffix should prevent the search for <meta> tags.

unknown module git-codecommit.us-east-1.amazonaws.com/v1/repos/return2.git: no go-import tags

Reproducible command summary :
(full repo path git-codecommit.us-east-1.amazonaws.com/v1/repos/return2 truncated to return2)

cmd import result
go get return2 fails - no import tags
go get return2.git succeeds!
vgo get return2 fails - no import tags
vgo get return2.git fails - no import tags
@myitcv

This comment has been minimized.

Copy link
Member

commented Jun 16, 2018

After reading the linked pages, it sounded like adding the .git would be the solution to hitting a provider that doesn't provide the meta tags.

@computermouth I have to admit, I'd forgotten that go get does some parsing of the import path to try to determine the VCS to use (hence why it succeeds in your table) which is why the go-import meta tag is not necessary there.

I have no real experience with CodeCommit but I would make the following observations:

You are looking to use the URL git-codecommit.us-east-1.amazonaws.com/v1/repos/return2 as your import path. This URL has a /v1 element in it which I suspect will cause issues with respect to vgo version resolution (even though the /v1 has nothing to do with your module version, and everything to do with the API version of CodeCommit).

It's also unlikely that you'll be able to return custom <meta> tags via CodeCommit.

The better thing to do here therefore would be to instead map a domain you own to the the CodeCommit repo via a custom import.

Let's say you own the domain example.com and you want to map the import path "example/return2" to git-codecommit.us-east-1.amazonaws.com/v1/repos/return2. You would then simply need to ensure that:

curl -sSL 'https://example.com/return2?go-get=1'

returns the <meta> tag:

<meta name="go-import" content="example.com/return2 git https://git-codecommit.us-east-1.amazonaws.com/v1/repos/return2">

Your main.go would then be rewritten as:

package main

import "fmt"
import "example.com/return2"

func main(){
	fmt.Println(return2.Doit())
}
@computermouth

This comment has been minimized.

Copy link
Author

commented Jun 18, 2018

Mmm, noted. I appreciate the suggested workaround.

Given what we know here, should this then be kept open as a bug in vgo? Or should I expect that this is simply the way that the behavior will remain?

If it's the latter, I wonder if this won't cause a more widespread failure when vgo is rolled into the go distribution. Assuming vgo get would replace the go get, wouldn't this cause a regression for folks using go get from CodeCommit?

@myitcv myitcv changed the title x/vgo: Unable to reach private AWS CodeCommit repo x/vgo: AWS CodeCommit repo URL incompatible with vgo major version import paths? Jun 19, 2018

@myitcv

This comment has been minimized.

Copy link
Member

commented Jun 19, 2018

@rsc @bcmills - re-opening for a decision from you on this:

  • do we want to introduce something similar to the go get import path parsing to determine remote repo type (e.g. .git)?
  • if we do, then do we want to support import paths of the form git-codecommit.eu-west-1.amazonaws.com/v1/repos/myitcvtest.git, where the v1 is the AWS CodeCommit API version number?

@myitcv myitcv reopened this Jun 19, 2018

@oiooj

This comment has been minimized.

Copy link
Member

commented Jun 20, 2018

Duplicate of #25654

@oiooj oiooj marked this as a duplicate of #25654 Jun 20, 2018

@oiooj oiooj closed this Jun 20, 2018

@myitcv

This comment has been minimized.

Copy link
Member

commented Jun 20, 2018

@oiooj thanks, I hadn't spotted #25654.

I don't think this is quite a duplicate, however. Yes, the .git part is covered by #25654, but the second point in #25792 (comment) is definitely relevant and separate from #25654 if we do implement the import path parsing.

@myitcv myitcv reopened this Jun 20, 2018

@computermouth

This comment has been minimized.

Copy link
Author

commented Jun 20, 2018

if we do, then do we want to support import paths of the form git-codecommit.eu-west-1.amazonaws.com/v1/repos/myitcvtest.git, where the v1 is the AWS CodeCommit API version number?

Keep in mind here, that this would mean tracking the list of AWS regions. It looks like there are approximately 20 right now.

Adding the .git functionality seems to be the most reasonable option.

@myitcv

This comment has been minimized.

Copy link
Member

commented Jun 20, 2018

Adding the .git functionality seems to be the most reasonable option.

That was my first point; if we do want to support the import path parsing, the second point was dealing with the fact the import path has v1 in it, and that v1 has nothing to do with our module version; it's the API version of CodeCommit.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Jul 6, 2018

The latest vgo does use the same logic now as "go get", including support for ".git" to mark the end of the git repo portion of an import path. I don't believe the /v1/ in the repo URL matters at all.

@computermouth could you go get -u golang.org/x/vgo and see if the return2.git form starts working for you?

Note that if you have a domain you like, you could serve the Go <meta> tags on that domain and point to the AWS git server, and then you won't be forced to have things like "us-east-1" in your import paths. (curl https://rsc.io/quote for example.)

@rsc rsc modified the milestones: vgo, Go1.11 Jul 12, 2018

@rsc rsc added the modules label Jul 12, 2018

@rsc rsc changed the title x/vgo: AWS CodeCommit repo URL incompatible with vgo major version import paths? cmd/go: AWS CodeCommit repo URL incompatible with vgo major version import paths? Jul 12, 2018

@rsc

This comment has been minimized.

Copy link
Contributor

commented Jul 17, 2018

Going to assume this was fixed by reusing the existing go get resolver.

@rsc rsc closed this Jul 17, 2018

@computermouth

This comment has been minimized.

Copy link
Author

commented Jul 17, 2018

Hi @rsc, sorry about the late reply.

It's fixed with one small detail.

Package imports as git-codecommit.us-east-1.amazonaws.com/v1/repos/return2.git require that packages go.mod contains module git-codecommit.us-east-1.amazonaws.com/v1/repos/return2.git.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.