Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: cmd/go: private repos on hard-coded hosting providers require special configuration #26134

Open
brunetto opened this issue Jun 29, 2018 · 27 comments
Labels
Projects
Milestone

Comments

@brunetto
Copy link

@brunetto brunetto commented Jun 29, 2018

What version of Go are you using (go version)?

$ go get -u golang.org/x/vgo
$ vgo version
go version go1.10 darwin/amd64 vgo:2018-02-20.1

Does this issue reproduce with the latest release?

yes (coming from #25590)

What operating system and processor architecture are you using (go env)?

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/brunetto.ziosi/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/brunetto.ziosi/Code"
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/m3/145jfhdd5qsf66_41wt9p02mhc4hp5/T/go-build518939426=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I ran

vgo get github.com/MY_ORGANIZATION/MY_REPO

where:

  • I ran the command in a folder with a working golang app and a proper go.mod file -> vgo is working here and I can vgo build the app
  • the repo is a private repo with SSO enabled I can access (git pull/push, ...)

What did you expect to see?

The package code in MY_REPO or the go module downloaded

What did you see instead?

vgo get github.com/MY_ORGANIZATION/MY_REPO: git ls-remote -q https://github.com/MY_ORGANIZATION/MY_REPO in /Users/brunetto.ziosi/Code/src/mod/cache/vcs/2cb00fa3547080bb301974fe87cf912af55d50f7a6d8a9955e11fed7a20ac6d3: exit status 128:
	remote: Repository not found.
	fatal: repository 'https://github.com/MY_ORGANIZATION/MY_REPO/' not found
@gopherbot gopherbot added this to the vgo milestone Jun 29, 2018
@bcmills
Copy link
Member

@bcmills bcmills commented Jun 29, 2018

What vgo commit did you build from?

What happens if you attempt to go get the same repository with the ordinary go tool?

Are you using the credential helper?

@brunetto
Copy link
Author

@brunetto brunetto commented Jul 2, 2018

What vgo commit did you build from?

I don't know, I just updated it before running it... how can I know the commit?

What happens if you attempt to go get the same repository with the ordinary go tool?

The simple go get does some stuff in the background and says nothing, if I run go get -v -u I see the repo itself downloaded and some dependencies downloaded, few others blocked by our firewall (-insecure seems to allow the download for everything).

Are you using the credential helper?

I don't think so, I exchanged the keys with github so my remote is like

$ git remote -v
origin	git@github.com:MY_ORGANIZATION/MY_REPO.git (fetch)
origin	git@github.com:MY_ORGANIZATION/MY_REPO.git (push)
@rsc
Copy link
Contributor

@rsc rsc commented Jul 6, 2018

The error text shows the command being run. It is one of the few git commands that doesn't actually care about having a git repo to run, so you can just try it in any directory you want:

git ls-remote -q https://github.com/MY_ORGANIZATION/MY_REPO

Does it work?

@rsc
Copy link
Contributor

@rsc rsc commented Jul 6, 2018

Possible duplicate of #26145 but the error message from git is different (no mention of terminal prompts disabled).

@brunetto
Copy link
Author

@brunetto brunetto commented Jul 6, 2018

Ok, so

$ git ls-remote -q https://github.com/MY_ORGANIZATION/MY_REPO
remote: Repository not found.
fatal: repository 'https://github.com/MY_ORGANIZATION/MY_REPO/' not found

but

$ git ls-remote git@github.com:MY_ORGANIZATION/MY_REPO.git
3fe3e3f8b07ff40cdf5de8685360715c4c06df9c	HEAD
1fd67741650c052e30bce6d780cd26c273ba0ab2	refs/heads/dev
1498d237221b9e19d55148dded8fafc7316a47bf	refs/heads/evo
3fe3e3f8b07ff40cdf5de8685360715c4c06df9a	refs/heads/master
3e86bf596a1f63631b91cc9cbbc0866b88cb67c3	refs/pull/1/head
14aef40c003sw32ab1402b2471d164c549624e5c	refs/pull/2/head
...

Maybe I am able to reach the repository only via ssh?

@rsc
Copy link
Contributor

@rsc rsc commented Jul 10, 2018

If you can make the https form work then vgo (and old go get) will be happy. The two possible ways to do that are:

  1. Add to $HOME/.gitconfig:

    [url "ssh://git@github.com/MYORGANIZATION/"]
    insteadOf = https://github.com/MYORGANIZATION/

  2. Add to $HOME/.netrc:

    machine github.com login YOU password APIKEY

where APIKEY is an API key obtained from the GitHub API page with access to private repos.

@rsc rsc changed the title x/vgo: Private repository not found cmd/go: github private repos require special configuration Jul 10, 2018
@rsc rsc modified the milestones: vgo, Go1.12 Jul 10, 2018
@rsc
Copy link
Contributor

@rsc rsc commented Jul 10, 2018

Retitling this to "cmd/go: github private repos require special configuration".
Maybe for Go 1.12 we should think about some way to get this right by default. I'm not sure exactly how.

@brunetto
Copy link
Author

@brunetto brunetto commented Jul 10, 2018

The first option, adding

[url "ssh://git@github.com/MYORGANIZATION/"]
    insteadOf = https://github.com/MYORGANIZATION/

to $HOME/.gitconfig work like a charm!!
Thanks!

@krak3n
Copy link

@krak3n krak3n commented Jul 24, 2018

Hi Guys,

I don't know if this is the correct issue to comment on so please forgive me if it's not correct, I ended up here because of an Issue I am having with Gitlab. We have a project structure where we use subgroups for project organisation by client/project/repo. So our package import paths are 3 levels deep rather than the standard 2 you see on Github. This is fine except for an issue with go get. Gitlab has a strict security policy where requests for the go-import meta data is incorrect for private sub groups by design for unauthenticated requests. So for example if I wanted to go get my.gitlab.com/foo/bar/fizz the meta data request would be https://my.gitlab.com/foo/bar/fizz?go-get=1 and the meta data returned would be:

<html><head><meta name="go-import" content="my.gitlab.com/foo/bar git https://my.gitlab.com/foo/bar.git" /></head></html>

Gitlab recently did implement support for a HTTP header to override this behaviour over here: https://gitlab.com/gitlab-org/gitlab-ce/issues/42817 so we can send a Private-Token header to get the correct meta. However I cannot find a way where I can get go get to send that header to resolve the source path.

We could update our import paths to include a .git suffix but that feels a little broken?

I guess what we need is a way for go get to find crednetial information for these sorts of projects, perhaps another file that lives alongside go.mod and go.sub like go.creds or something, which could contain credential information (you wouldn't want to check that into a repo tho). Or perhaps a way to override go-import meta data locally? 🤷‍♂️

@myitcv
Copy link
Member

@myitcv myitcv commented Aug 8, 2018

To add to the list of options in this space (#26134 (comment)), we should also add use of a keychain/keyring as a third option (which is independent of remote VCS). Instructions vary quite widely between platforms, unsurprisingly:

I've switched back from using the ssh solution on Linux to using Gnome keyring (even though I don't use Gnome) to great effect... not least because it can be used to unify all credentials.

@FiloSottile are there are any security implications that I'm unaware of with respect to the use of keychain/keyrings for this sort of thing?

@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Aug 9, 2018

With Go tip (f2131f6) I can use private repos without modules (with the url "ssh://git insteadOf config above), but not with modules:

dev:barlogaamp $ GO111MODULE=on go install -x -v .
# /home/bradfitz/pkg/mod/cache/vcs/61e3c76780847e514802ec6af8f940f641c6017f711444f05c59cb17ac46d456 for git2 https://github.com/bradfitz/private
go: finding github.com/bradfitz/private v0.0.0-20180808000000-9069f54cff2af5e9fc291df126155504d71d1fd9
cd /home/bradfitz/pkg/mod/cache/vcs/61e3c76780847e514802ec6af8f940f641c6017f711444f05c59cb17ac46d456; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' 9069f54cff2af5e9fc291df126155504d71d1fd9
0.002s # cd /home/bradfitz/pkg/mod/cache/vcs/61e3c76780847e514802ec6af8f940f641c6017f711444f05c59cb17ac46d456; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' 9069f54cff2af5e9fc291df126155504d71d1fd9
cd /home/bradfitz/pkg/mod/cache/vcs/61e3c76780847e514802ec6af8f940f641c6017f711444f05c59cb17ac46d456; git tag -l
0.002s # cd /home/bradfitz/pkg/mod/cache/vcs/61e3c76780847e514802ec6af8f940f641c6017f711444f05c59cb17ac46d456; git tag -l
cd /home/bradfitz/pkg/mod/cache/vcs/61e3c76780847e514802ec6af8f940f641c6017f711444f05c59cb17ac46d456; git ls-remote -q https://github.com/bradfitz/private
0.177s # cd /home/bradfitz/pkg/mod/cache/vcs/61e3c76780847e514802ec6af8f940f641c6017f711444f05c59cb17ac46d456; git ls-remote -q https://github.com/bradfitz/private
cd /home/bradfitz/pkg/mod/cache/vcs/61e3c76780847e514802ec6af8f940f641c6017f711444f05c59cb17ac46d456; git fetch -f --depth=1 https://github.com/bradfitz/private 9069f54cff2af5e9fc291df126155504d71d1fd9:refs/dummy
0.165s # cd /home/bradfitz/pkg/mod/cache/vcs/61e3c76780847e514802ec6af8f940f641c6017f711444f05c59cb17ac46d456; git fetch -f --depth=1 https://github.com/bradfitz/private 9069f54cff2af5e9fc291df126155504d71d1fd9:refs/dummy
go: github.com/bradfitz/private@v0.0.0-20180808000000-9069f54cff2af5e9fc291df126155504d71d1fd9: git fetch -f --depth=1 https://github.com/bradfitz/private 9069f54cff2af5e9fc291df126155504d71d1fd9:refs/dummy in /home/bradfitz/pkg/mod/cache/vcs/61e3c76780847e514802ec6af8f940f641c6017f711444f05c59cb17ac46d456: exit status 1
go: error loading module requirements

That final git fetch fails, but that hash exists in the private repo:

$ git cat-file -t 9069f54cff2af5e9fc291df126155504d71d1fd9
commit

And it's on the remote (it's remote/origin/master^), but it doesn't have a remote tag or other ref, if that's the problem.

dev:src $ git --version
git version 2.11.0
@myitcv
Copy link
Member

@myitcv myitcv commented Aug 9, 2018

@bradfitz I think you're running up against #26894

@twmb
Copy link
Contributor

@twmb twmb commented Oct 30, 2018

Is it possible to add a flag that causes go to clone over ssh (or any other potential future protocol)?

A flag would be easy to use in CI where necessary and is more easily explained than editing a .gitconfig or .netrc.

@anuragdhingra
Copy link

@anuragdhingra anuragdhingra commented Jan 29, 2019

Wrote a script if any of you guys need to containerize a similar app using private github repo imports:
https://github.com/anuragdhingra/pdocker-go

@bcmills
Copy link
Member

@bcmills bcmills commented Feb 6, 2019

Note that this issue only applies to the hard-coded hosting sites.

If we have some sort of general HTTPS auth mechanism (#26232), HTTPS servers that use <meta> tags should provide ssh URLs as appropriate to begin with.

@bcmills bcmills changed the title cmd/go: github private repos require special configuration cmd/go: private repos on hard-coded hosting providers require special configuration Feb 11, 2019
@stub42
Copy link

@stub42 stub42 commented Apr 11, 2019

Issue #31376 was duplicated here, but is not using one of the hard-coded hosting sites. The matching HTTPS server provides no tags, and documented methods to make 'go get' skip VCS detection step do not seem to be working.

@bcmills
Copy link
Member

@bcmills bcmills commented Apr 11, 2019

@stub42, my apologies. We do have a special case for launchpad.net; I had missed the fact that git.launchpad.net should not match that special case.

@hazcod
Copy link

@hazcod hazcod commented Jun 5, 2019

Thank you for this issue, I was debugging the access to my private github repos for a week..

$HOME/.gitconfig:

[url "ssh://git@github.com/MYORGANIZATION/"]
insteadOf = https://github.com/MYORGANIZATION/
@andybons andybons modified the milestones: Go1.13, Go1.14 Jul 8, 2019
@millergarym
Copy link

@millergarym millergarym commented Jul 28, 2019

@rsc I have a huntch that capturing private repo information in the go.mod file might help. This information and auth details can get used directly or passed to Athens.

Maybe instead of insteadOf

Add to $HOME/.gitconfig:

[url "ssh://git@github.com/MYORGANIZATION/"]
insteadOf = https://github.com/MYORGANIZATION/

we could write something like

module my/package

replace github.com/MYORGANIZATION => ssh://git@github.com/MYORGANIZATION/someprivatepackage v1.1.1 

require github.com/MYORGANIZATION/someprivatepackage v1.1.1

or

module my/package

require github.com/MYORGANIZATION/someprivatepackage v1.1.1 at git@github.com:MYORGANIZATION/someprivatepackage
@marksamman
Copy link

@marksamman marksamman commented Jul 29, 2019

I'd also like to see a solution to this where the private repo information can be stored in the repository. This is currently holding me back from using Go modules for projects that depend on private repositories because it's not seamless enough (requires modifying $HOME/.gitconfig locally).

@millergarym
Copy link

@millergarym millergarym commented Jul 29, 2019

@marksamman any ideas on how authentication should be done?

Is putting a public key in the go.mod file a security issue?

eg

module my/package

require github.com/MYORGANIZATION/someprivatepackage v1.1.1 at git@github.com:MYORGANIZATION/someprivatepackage using ssh-rsa AAAAB3NzaC1yc2EAAAADAQA...2JN millergarym@development_machine_and_ci
@hazcod
Copy link

@hazcod hazcod commented Jul 29, 2019

@marksamman any ideas on how authentication should be done? ...

@millergarym if you mean a private key, yes.

@marksamman
Copy link

@marksamman marksamman commented Jul 29, 2019

I am not suggesting including authentication credentials in the repository, just a way of pulling over SSH instead of HTTPS and leaving the rest for Git to handle.

EDIT: I've learned that my problem can be solved with the go-import meta tag, so this is no longer a blocker for me to use Go modules with private repositories.

@millergarym
Copy link

@millergarym millergarym commented Jul 30, 2019

@marksamman any ideas on how authentication should be done? ...

@millergarym if you mean a private key, yes.

No, only public keys

@millergarym
Copy link

@millergarym millergarym commented Jul 30, 2019

EDIT: I've learned that my problem can be solved with the go-import meta tag, so this is no longer a blocker for me to use Go modules with private repositories.

Can meta tags point to private repos? If so please provide an example.

@stub42
Copy link

@stub42 stub42 commented Jul 30, 2019

Meta tags require serving public document referencing the private branch or repo, which is data leakage and not acceptable at some sites. The Launchpad web site returns a 404 for private branches, unless you are both logged in and have been granted read access. A tool can't tell if the repo exists unless it has a valid oauth token or speaks SSH.

My current workaround is to embed my private branches as git subtrees, as 'replace' in go.mod already allows me to reference a relative local path:

replace api/fooclient v0.0.0 => ../api/fooclient
require api/fooclient v0.0.0

I think allowing 'replace' to specify full URLs would be a workable solution, or a new command, ideally without the v0.0.0 version pin. Say, theoretically:

rewrite api/fooclient => git+ssh://git.launchpad.net/~stub/foo/+git/foo-client
require api/fooclient

(Interestingly, you should be able to use the existing replace to force use of a fork without changing imports, perhaps even a fork of parts of the standard library)

@DeaglePC
Copy link

@DeaglePC DeaglePC commented Jul 7, 2020

The first option, adding

[url "ssh://git@github.com/MYORGANIZATION/"]
    insteadOf = https://github.com/MYORGANIZATION/

to $HOME/.gitconfig work like a charm!!
Thanks!

DO NOT MISSING THE LAST '/' CHAR!

@rsc rsc changed the title cmd/go: private repos on hard-coded hosting providers require special configuration propsoal: cmd/go: private repos on hard-coded hosting providers require special configuration Feb 2, 2021
@rsc rsc changed the title propsoal: cmd/go: private repos on hard-coded hosting providers require special configuration proposal: cmd/go: private repos on hard-coded hosting providers require special configuration Feb 2, 2021
@rsc rsc added this to Incoming in Proposals Feb 2, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Proposals
Incoming
Linked pull requests

Successfully merging a pull request may close this issue.

None yet