wiki: add page on using private repositories (e.g., on-prem GitLab, private github.com repos, etc.)

[thepudds]

Consider adding a new wiki page on using Go with private repositories such as on-prem GitLab, on-prem GitHub, on-prem bitbucket, private github.com repos, etc.

This seems to be a fairly frequent source of confusion, and often it is when someone is new or somewhat new to Go, which means it both impacts early impressions of Go, as well as means someone at that point in time is less versed in Go overall (e.g., it is more challenging to work through how to set up something like an on-prem GitLab instance if someone is asking "what is an import path?" at more or less the same time).

It also comes up in the context of modules (because of a different error with modules, or because someone initially assumes they are seeing a module-specific issue, etc.). I would wager that at this point in time, a private repo problem that is reported in the context of modules the majority of the time ends up not actually being a module-specific issue (at least as of 1.11), but there are also module-specific aspects such as how to have an internal corporate mirror via the replacedirective, etc.

Part of the current complexity comes from the fact that it is a moving target with products like GitLab and GitHub Enterprise improving over time (or in some cases, having new bugs), which also means currently people can end up talking past each other in terms of a specific solution if they are on different versions. A similar piece of current complexity is that sometimes a solution described for GitHub Enterprise also works for GitLab and other solutions, but sometimes that is not the case.

Having a wiki page could help on all of those fronts, including helping to draft off of community experience. It might also cut down on people reinventing solutions from scratch. I personally would not expect this type of material to be in the core Go documentation, nor would I expect the core Go team to be experts in something like on-prem GitLab.

I am not sure of the best title, but perhaps "Private Repositories", or "Working with Private Repositories"?

Some current material

• https://golang.org/doc/faq#git_https
• https://golang.org/cmd/go/#hdr-Remote_import_paths
• cmd/go: private repos on hard-coded hosting providers require special configuration #26134 "cmd/go: github private repos require special configuration"
• cmd/go: custom import path private repos require special configuration #27254 (comment) "cmd/go: custom import path private repos require special configuration"
• "gitlab not support go-get with sub-groups"
• "Jan 22 2019 GitLab release notes: Support for private Go packages in subgroups"

Some older items

• x/vgo: not working with GitHub Enterprise #24076 (comment)
• x/vgo: not working with GitHub Enterprise #24076 (comment)
• cmd/go: add support for GitHub Enterprise import path #17898 (comment)
• cmd/go: using git over ssh fails to load modules requirements #26894 "cmd/go: using git over ssh fails to load modules requirements"
• cmd/go: cannot fetch gitlab repo using git:// using insteadOf (macOS git 2.19.1) #28422 "cmd/go: insteadOf broken with MacOS git 2.19.1 and go version 1.11.1?"
• cmd/go mod falls to download dependencies due to git not using ssh #28653 "cmd/go: mod falls to download dependencies due to git not using ssh""

Some sample community commentary

The first four are taken from here (including as an example of people sharing different solutions that don't always work for the original reporter). The remainder are from various other discussions.

I have to do this to get to private repos in Github using go get/dep/go mod:
git config --global url."git@github.com:".insteadOf "https://github.com/"

I have that in my .gitconfig - but it seems to ignore it. I also created a private repo token and added that as well. Both seemed to do nothing.

We setup a nginx proxy with ssl cert in front of gitlab. Also I setup a url responder to respond back from "go get" requests. There is no need to have any edit to a ~/.git/config to have insteadOf etc

I solved these problems for GitHub Enterprise. I put together a small go service modelled after https://github.com/niemeyer/gopkg for my team.

https://help.github.com/enterprise/2.14/admin/guides/user-management/changing-authentication-methods/ says in Git authentication: “LDAP supports password-based Git authentication by default, but we recommend that you disable that method and force authentication via a personal access token or SSH key.” Access with personal access tokens is described on https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/

for gitlab:
$ git config --global -l url.gitlab@gitlab.int.company.com:.insteadof=https://gitlab.int.company.com/

CC @bcmills @andybons @myitcv @mvdan @dmitris @leitzler

[mvdan]

Seems reasonable to me. Every week I see someone having issues with modules and on-site git servers on the #modules slack channel.

[bcmills]

Private repos are one of the major areas I want to push forward in 1.13. Specifically:

#26232
#26134
#25982
#20604
#27344
#27088
#29888

You're welcome to update the wiki in the meantime, but I expect that at least a few current workarounds will go away.

[bcmills]

I would probably title the page PrivateRepos. You're welcome to go ahead and create it if you like.

[javasgl]

why go modules use HTTP(s) to download repos instead of ssh?

tools like dep or glide don`t have this problem when fetching private repos

[dmitris]

dep or glide allowed you to specify the git URL (which I found really useful) so you can make you pick of git@ or https://. The go tools insists on https:// thus forcing you to use the insteadOf tricks (you, and all the users / developers of your package). I remember I read that folks making the decision prefer git authorization tokens to the SSH authentication (in the issues I asked about it, not sure if there is a reference that states this; possibly will be linked in the new PrivateRepos page).

I wish it would be possible to specify in go.mod's replace section the git URL of the module you are replacing the dependency with - that would keep feature parity with glide / dep. It would also allow to specify the configuration in one file - go.mod - instead of getting ~/.gitconfig involved which causes some headaches in the CI / headless setups.

[dmitris]

@thepudds suggested to add an example of what I'd like to have and how it could be useful.

here's an excerpt from one of our previous glide.yaml files:

import:
- package: github.com/nlopes/slack
  version: 0afac90b917f77d7c59d638420d5d40fe65eee61
  repo: git@mirror.foobarxyz.com:nlopes/slack.git
- package: github.com/DATA-DOG/godog
  repo: git@mirror.foobarxyz.com:DATA-DOG/godog.git

Our company uses a GHE instance with ssh authentication - both humans and CI setups. With glide, anyone was able to check the repository out, run glide update and build. Compare now with go.mod - to start playing the go install game, you have to "pay the entry price" by setting up your ~/.gitconfig with that annoying insteadOf command that everyone has to look up in the docs. Got on a new VM or Vagrant instance? don't forget to set that insteadOf setup; etc. Yikes.

So my wish - likely both belated and unrealistic - would be to have a place in go.mod where you can specify the git URL for the replacement module, for example:
github.com/DATA-DOG/godog => git@mirror.foobarxyz.com:DATA-DOG/godog.git v1.0.0 which would make clear to the go tool that it should grab the godog code from the given git URL (and please don't try to connect to github.com which can be network cut-off on the CI build instances).
/cc @rsc

[thepudds]

Good discussion of some of the issues and compares a few alternative solutions:
https://medium.com/@tim_raymond/fetching-private-dependencies-with-go-modules-1d65afe47c62

[thepudds]

#25792 is older AWS CodeCommit vgo bug, and includes final comment:

It's fixed with one small detail.

Package imports as git-codecommit.us-east-1.amazonaws.com/v1/repos/return2.git require that packages go.mod contains module git-codecommit.us-east-1.amazonaws.com/v1/repos/return2.git.

[thepudds]

Another recent blog post on using modules with private git repositories that steps through the details of some alternatives:
https://medium.com/cloud-native-the-gathering/go-modules-with-private-git-repositories-dfe795068db4

[ZaynJarvis]

Any update on this?

[lpvm]

The solution that worked for me was to set the environmental variable:
GOPRIVATE=github.com/my-github-user-name/*
Or to make it permanent, add this to .bashrc:
export GOPRIVATE=github.com/my-github-user-name/*

[dmitris]

Or to make it permanent, [...]

can also run go env -w GOPRIVATE="github.com/my-github-user-name/" (I believe the trailing star is implied)

[arschles]

If this happens, I'm happy to write a section on how to set up an Athens server for private modules

[n-insaidoo]

Any info on what configuration to use to resolve BitBucket Cloud private repositories?

[gudvinr]

Using both GOPRIVATE=bitbucket.org and git config --global url."ssh://git@bitbucket.org/".insteadOf "https://bitbucket.org/" doesn't seem to work inside docker despite of correct ssh key and record in .known_hosts.

go mod download still tries to use https.

EDIT: It works only when you provide key without password

[ionling]

@gudvinr You can use ssh-add your-private-key to work with password, but still may not work in Docker.

[gudvinr]

You can use ssh-add your-private-key to work with password, but still may not work in Docker.

@ionling SSH agent passthrough works only for BuildKit-enabled builds and somewhat experimental. It didn't work for me either but docker-related questions aren't related to this topic so that's where I end this conversation.

[RichDavis1]

This would be very helpful. I can't seem to get go to pass the creds in my .netrc file when performing go get. Very frustrating trying to figure it out.

[rmrfslashbin]

Another (seemingly) broken use-case: AWS Codecommit private repos. It almost works, but not quite.

From various references, I've set the GOPROVATE env:
$ go env |grep aws GONOPROXY="git-codecommit.us-east-1.amazonaws.com" GONOSUMDB="git-codecommit.us-east-1.amazonaws.com" GOPRIVATE="git-codecommit.us-east-1.amazonaws.com"

And set the insteadOf in my ~/.gitconfig:
[url "XXXAWSKEYXXX@git-codecommit.us-east-1.amazonaws.com"] insteadOf = https://git-codecommit.us-east-1.amazonaws.com/

This scenario fails:
$ go get git-codecommit.us-east-1.amazonaws.com/v1/repos/go-mod-test go get: unrecognized import path "git-codecommit.us-east-1.amazonaws.com/v1/repos/go-mod-test": reading https://git-codecommit.us-east-1.amazonaws.com/v1/repos/go-mod-test?go-get=1: 401
So I add the .git to the end:
$ go get git-codecommit.us-east-1.amazonaws.com/v1/repos/go-mod-test.git go: downloading git-codecommit.us-east-1.amazonaws.com/v1/repos/go-mod-test.git v0.0.1 go get: git-codecommit.us-east-1.amazonaws.com/v1/repos/go-mod-test.git@none updating to git-codecommit.us-east-1.amazonaws.com/v1/repos/go-mod-test.git@v0.0.1: parsing go.mod: module declares its path as: git-codecommit.us-east-1.amazonaws.com/v1/repos/go-mod-test but was required as: git-codecommit.us-east-1.amazonaws.com/v1/repos/go-mod-test.git
It seems like private repos are just not ready for general use?

[weitzj]

This seems still broken in Go 1.17.4 for private Gitlab projects, which have subgroups, i.e. gitlab.com/org/team/tools/mytool.go

[LUC18fknU7P]

This seems still broken in Go 1.17.4 for private Gitlab projects, which have subgroups, i.e. gitlab.com/org/team/tools/mytool.go

Yup. A total nightmare to deal with...

[weitzj]

I have contacted our GitLab enterprise support to reopen: https://gitlab.com/gitlab-org/gitlab-foss/-/issues/30785