New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: go get doesn't work with private gitbucket repo #19766

Open
tundratim opened this Issue Mar 29, 2017 · 10 comments

Comments

Projects
None yet
6 participants
@tundratim

tundratim commented Mar 29, 2017

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

1.8

What operating system and processor architecture are you using (go env)?

x64 RHEL7

What did you do?

Attempt to do a deep import (go get) on private gitbucket repo

What did you expect to see?

Correct import

What did you see instead?

Fails because clone URL is not getting correctly parsed from gitbucket

I am doing devops support for a client doing a bunch of new go stuff. They are using an internal instance of gitbucket for their corporate code. I am not a go expert by any means, but as I understand it ....

  • Ordinary "go get" works fine, but
  • A "deep" get - somewhere down the repo tree fails because the resultant URL go tries to use is wrong

One of the devs showed me the list of rexeges baked into go for the various vcs products out there, but gitbucket seems not to be on there as best as I could tell.

So ... I guess the ask is either:

  • Fix go to properly interact with gitbucket
  • Provide some clean way for devops folk to add new vcs regex definitions without having to recompile the language.

Any thoughts here would be welcome.

@bradfitz

This comment has been minimized.

Member

bradfitz commented Mar 29, 2017

See go help importpath (or https://golang.org/cmd/go/#hdr-Remote_import_paths). While there are some specially-recognized cases, any URL works fine if it has the right <meta ...> tags.

gitbucket is not a hosted service, right? So we couldn't regexp it anyway, since users can run it on any hostname?

So I think the best option is for you to modify gitbucket to add the right <meta> tags when cmd/go makes a request to it. The cmd/go command always sends a request with ?go-get=1, which gitbucket can check for and go into a special code path.

Let me know if I misunderstand, but I think there's nothing for us to do here so I'm going to close this. Holler if I should reopen.

@bradfitz bradfitz closed this Mar 29, 2017

@tundratim

This comment has been minimized.

tundratim commented Mar 29, 2017

I think the essence of the problem occurs when we use a git repo not in your list of public servers, we default to the regex found at line 902 in:

https://golang.org/src/cmd/go/vcs.go

This regex creates the following problem as I understand it: For any git repository, if you go get domain.of.repo/repo/name.git, go tries to clone domain.of.repo/repo/name instead of name.git.

IOW, the belief by our devs is that this is incorrect behavior for the default case.

As an aside - I see more and more customers spinning up their own repos on their own hosts wherein this default behavior might come back to bite them.

P.S. We also believe there is an issue with GitBucket itself here as well:

If you try to import domain.of.repo/repo/name/deep/import, go tries to http get domain.of.repo/repo/name/deep/import, which 404s, because gitbucket serves that path at domain.of.repo/repo/name/tree/master/deep/import.

Thanks for the quick turnaround here!

@bradfitz

This comment has been minimized.

Member

bradfitz commented Mar 29, 2017

What is your concrete problem?

And you didn't reply to my suggestion of using the Go <meta> tags and adding support for that to gitbucket.

@bradfitz

This comment has been minimized.

Member

bradfitz commented Mar 29, 2017

(And by concrete problem, I mean to give me an actual command go get command that fails.)

@tundratim

This comment has been minimized.

tundratim commented Mar 29, 2017

I have contacted the devs involved and asked for more specific details.

@bentucker

This comment has been minimized.

bentucker commented Mar 31, 2017

Hey Brad,

Here's the issue that I'm seeing:

$  go get -v git.example.local/repo/repo-name.git
# cd .; git ls-remote https://git.example.local/repo/repo-name
fatal: repository 'https://git.example.local/repo/repo-name/' not found
# cd .; git ls-remote git+ssh://git.example.local/repo/repo-name
Unknown command: git-upload-pack '/repo/repo-name'
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
# cd .; git ls-remote ssh://git.example.local/repo/repo-name
Unknown command: git-upload-pack '/repo/repo-name'
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Please note that git ls-remote ssh://git.example.local/repo/repo-name.git works as expected.

From what I can tell, the default vcsPath is being matched, and the regexp is stripping the '.git' from the vcs repo. (At line 902 in vcs.go)

@bradfitz

This comment has been minimized.

Member

bradfitz commented Apr 2, 2017

What does:

git ls-remote https://git.example.local/repo/repo-name.git

... say?

@bentucker

This comment has been minimized.

bentucker commented Apr 5, 2017

That returns the same references that are returned by git ls-remote ssh://...

@bradfitz bradfitz reopened this Apr 5, 2017

@bradfitz bradfitz changed the title from Possible Bug And/Or Feature Request to cmd/go: go get doesn't work with private gitbucket repo Apr 5, 2017

@rsc

This comment has been minimized.

Contributor

rsc commented Apr 5, 2017

If your Git server is not one of the recognized ones (say, Github) and not pointed at by a custom domain using tags, then the final fallback is that we'll recognize x.git/y as meaning checkout x with git and then look at y inside the repo. Note that some Git servers (like Github) make the .git optional so that x and x.git mean the same thing, which can be a source of confusion here, but the .git in this syntax is so go get knows to use git at all, not sent to the server. Otherwise it would not be possible to use a repo that didn't end in .git.

If on your server x needs to end in .git, you need to say it twice, once for your server and once for go get:

go get -v git.example.local/repo/repo-name.git.git

Of course that looks awful but it's implied by the use of this fallback. In general it's far preferable to set up an HTTPS redirector to let you separate the import paths from the exact location of the git servers (for example curl https://rsc.io/pdf). I'd suggest doing that if .git.git is deemed too ugly.

The fallback here predates the custom URLs. If I had it to do again I'd probably just remove the fallback entirely and require HTTPS tags.

@hex108

This comment has been minimized.

hex108 commented Mar 23, 2018

Hi @bradfitz @rsc , could we add a configure to add our own git servers like the public serves in code(https://golang.org/src/cmd/go/vcs.go)? Then go get could skip sending a request with ?go-get=1. It is very useful for those private repos that does not response ?go-get=1 request correctly.

@andybons andybons added this to the Unplanned milestone Mar 23, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment