Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wiki: add page on using private repositories (e.g., on-prem GitLab, private github.com repos, etc.) #29953

Open
thepudds opened this issue Jan 27, 2019 · 9 comments

Comments

@thepudds
Copy link

commented Jan 27, 2019

Consider adding a new wiki page on using Go with private repositories such as on-prem GitLab, on-prem GitHub, on-prem bitbucket, private github.com repos, etc.

This seems to be a fairly frequent source of confusion, and often it is when someone is new or somewhat new to Go, which means it both impacts early impressions of Go, as well as means someone at that point in time is less versed in Go overall (e.g., it is more challenging to work through how to set up something like an on-prem GitLab instance if someone is asking "what is an import path?" at more or less the same time).

It also comes up in the context of modules (because of a different error with modules, or because someone initially assumes they are seeing a module-specific issue, etc.). I would wager that at this point in time, a private repo problem that is reported in the context of modules the majority of the time ends up not actually being a module-specific issue (at least as of 1.11), but there are also module-specific aspects such as how to have an internal corporate mirror via the replacedirective, etc.

Part of the current complexity comes from the fact that it is a moving target with products like GitLab and GitHub Enterprise improving over time (or in some cases, having new bugs), which also means currently people can end up talking past each other in terms of a specific solution if they are on different versions. A similar piece of current complexity is that sometimes a solution described for GitHub Enterprise also works for GitLab and other solutions, but sometimes that is not the case.

Having a wiki page could help on all of those fronts, including helping to draft off of community experience. It might also cut down on people reinventing solutions from scratch. I personally would not expect this type of material to be in the core Go documentation, nor would I expect the core Go team to be experts in something like on-prem GitLab.

I am not sure of the best title, but perhaps "Private Repositories", or "Working with Private Repositories"?

Some current material

Some older items

Some sample community commentary

The first four are taken from here (including as an example of people sharing different solutions that don't always work for the original reporter). The remainder are from various other discussions.

I have to do this to get to private repos in Github using go get/dep/go mod:
git config --global url."git@github.com:".insteadOf "https://github.com/"

I have that in my .gitconfig - but it seems to ignore it. I also created a private repo token and added that as well. Both seemed to do nothing.

We setup a nginx proxy with ssl cert in front of gitlab. Also I setup a url responder to respond back from "go get" requests. There is no need to have any edit to a ~/.git/config to have insteadOf etc

I solved these problems for GitHub Enterprise. I put together a small go service modelled after https://github.com/niemeyer/gopkg for my team.

https://help.github.com/enterprise/2.14/admin/guides/user-management/changing-authentication-methods/ says in Git authentication: “LDAP supports password-based Git authentication by default, but we recommend that you disable that method and force authentication via a personal access token or SSH key.” Access with personal access tokens is described on https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/

for gitlab:
$ git config --global -l url.gitlab@gitlab.int.company.com:.insteadof=https://gitlab.int.company.com/

CC @bcmills @andybons @myitcv @mvdan @dmitris @leitzler

@mvdan

This comment has been minimized.

Copy link
Member

commented Jan 28, 2019

Seems reasonable to me. Every week I see someone having issues with modules and on-site git servers on the #modules slack channel.

@bcmills

This comment has been minimized.

Copy link
Member

commented Jan 28, 2019

Private repos are one of the major areas I want to push forward in 1.13. Specifically:

#26232
#26134
#25982
#20604
#27344
#27088
#29888

You're welcome to update the wiki in the meantime, but I expect that at least a few current workarounds will go away.

@bcmills

This comment has been minimized.

Copy link
Member

commented Jan 29, 2019

I would probably title the page PrivateRepos. You're welcome to go ahead and create it if you like.

@bcmills bcmills added this to the Unreleased milestone Jan 29, 2019

@javasgl

This comment has been minimized.

Copy link

commented Jan 30, 2019

why go modules use HTTP(s) to download repos instead of ssh?

tools like dep or glide don`t have this problem when fetching private repos

@dmitris

This comment has been minimized.

Copy link
Contributor

commented Feb 1, 2019

dep or glide allowed you to specify the git URL (which I found really useful) so you can make you pick of git@ or https://. The go tools insists on https:// thus forcing you to use the insteadOf tricks (you, and all the users / developers of your package). I remember I read that folks making the decision prefer git authorization tokens to the SSH authentication (in the issues I asked about it, not sure if there is a reference that states this; possibly will be linked in the new PrivateRepos page).

I wish it would be possible to specify in go.mod's replace section the git URL of the module you are replacing the dependency with - that would keep feature parity with glide / dep. It would also allow to specify the configuration in one file - go.mod - instead of getting ~/.gitconfig involved which causes some headaches in the CI / headless setups.

@dmitris

This comment has been minimized.

Copy link
Contributor

commented Feb 1, 2019

@thepudds suggested to add an example of what I'd like to have and how it could be useful.

here's an excerpt from one of our previous glide.yaml files:

import:
- package: github.com/nlopes/slack
  version: 0afac90b917f77d7c59d638420d5d40fe65eee61
  repo: git@mirror.foobarxyz.com:nlopes/slack.git
- package: github.com/DATA-DOG/godog
  repo: git@mirror.foobarxyz.com:DATA-DOG/godog.git

Our company uses a GHE instance with ssh authentication - both humans and CI setups. With glide, anyone was able to check the repository out, run glide update and build. Compare now with go.mod - to start playing the go install game, you have to "pay the entry price" by setting up your ~/.gitconfig with that annoying insteadOf command that everyone has to look up in the docs. Got on a new VM or Vagrant instance? don't forget to set that insteadOf setup; etc. Yikes.

So my wish - likely both belated and unrealistic - would be to have a place in go.mod where you can specify the git URL for the replacement module, for example:
github.com/DATA-DOG/godog => git@mirror.foobarxyz.com:DATA-DOG/godog.git v1.0.0 which would make clear to the go tool that it should grab the godog code from the given git URL (and please don't try to connect to github.com which can be network cut-off on the CI build instances).
/cc @rsc

@bcmills bcmills added the NeedsFix label Mar 1, 2019

@bcmills bcmills changed the title wiki: consider new wiki page on using private repositories (e.g., on-prem GitLab, private github.com repos, etc.) wiki: add page on using private repositories (e.g., on-prem GitLab, private github.com repos, etc.) Mar 1, 2019

@thepudds

This comment has been minimized.

Copy link
Author

commented Mar 27, 2019

Good discussion of some of the issues and compares a few alternative solutions:
https://medium.com/@tim_raymond/fetching-private-dependencies-with-go-modules-1d65afe47c62

@thepudds

This comment has been minimized.

Copy link
Author

commented May 17, 2019

#25792 is older AWS CodeCommit vgo bug, and includes final comment:

It's fixed with one small detail.

Package imports as git-codecommit.us-east-1.amazonaws.com/v1/repos/return2.git require that packages go.mod contains module git-codecommit.us-east-1.amazonaws.com/v1/repos/return2.git.

@thepudds

This comment has been minimized.

Copy link
Author

commented May 21, 2019

Another recent blog post on using modules with private git repositories that steps through the details of some alternatives:
https://medium.com/cloud-native-the-gathering/go-modules-with-private-git-repositories-dfe795068db4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.