Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/vgo: not working with GitHub Enterprise #24076

Closed
richardwilkes opened this issue Feb 23, 2018 · 20 comments

Comments

@richardwilkes
Copy link

commented Feb 23, 2018

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.10 darwin/amd64 vgo:2018-02-20.1

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/rich/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/rich/code/go"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.10/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.10/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/s9/9s3fk2mx5f936dkcp66ltwsc0000gn/T/go-build062786983=/tmp/go-build -gno-record-gcc-switches -fno-common"
VGOMODROOT=""

What did you do?

Attempted to build some code that had dependencies on one or more packages that come from our internal GitHub repo as well as from the public github.com using the experimental vgo tool. As this refers to an internal repo (and that is the basis of the problem), there is no way for me to give you code that reproduces the problem that you can actually try. Suffice to say, though, that even the simplest import causes the issue.

What did you expect to see?

I was hoping that GitHub Enterprise would work as well as normal GitHub.

What did you see instead?

vgo: resolving import "jaxf-github.fanatics.corp/forge/furnace"
findRepo: XML syntax error on line 25: unescaped < inside quoted string
vgo: import "jaxf-github.fanatics.corp/forge/ratelimiter" ->
import "jaxf-github.fanatics.corp/forge/furnace": unknown VCS, Repo: git,
https://jaxf-github.fanatics.corp/forge/furnace.git

@ALTree ALTree changed the title vgo not working with GitHub Enterprise x/vgo: not working with GitHub Enterprise Feb 23, 2018

@gopherbot gopherbot added this to the vgo milestone Feb 23, 2018

@ALTree

This comment has been minimized.

Copy link
Member

commented Feb 23, 2018

go get has the same issue: #17898

@richardwilkes

This comment has been minimized.

Copy link
Author

commented Feb 23, 2018

Yes, I just realized that as well. This is still workable when using the normal go build, though, because we can (and do) manually checkout our internal repos into our $GOPATH/src tree. That doesn't seem to be an option with vgo, unless I've missed something.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Feb 23, 2018

I don't know much about GitHub Enterprise. Does it have similar APIs to public GitHub? It is possible this could work better with vgo than old go get. If the tag said something like githubenterprise to tell us it was that kind of server, we could use those APIs.

@richardwilkes

This comment has been minimized.

Copy link
Author

commented Feb 23, 2018

Can you give me an example request? I'm happy to make it against something in our repo and give you the resulting output.

@mikkeloscar

This comment has been minimized.

Copy link

commented Feb 23, 2018

The APIs are the same (except for the base path) but vgo would have to know that something like https://github.mycompany.com is a Github Enterprise installation.

It would be great to think about this case from the beginning!

I opened a similar issue when dep was released: golang/dep#174 but this is unfortunately still an issue preventing us from adopting dep.

@amnonbc

This comment has been minimized.

Copy link

commented Feb 24, 2018

This is an issue which has caused me a fair amount of pain.

See https://go-review.googlesource.com/c/go/+/37176 for a simple fix for the go get case in go-1.10

@sdboyer

This comment has been minimized.

Copy link
Member

commented Feb 25, 2018

FWIW, i have been collecting a group of company GHE customers to try to influence GitHub to actually fix this problem. (it exists in GitHub itself, as well, but it's masked by having static rules that cover it)

@amnonbc

This comment has been minimized.

Copy link

commented Mar 4, 2018

Hi @sdboyer
I work for a GHE customer, and raised the issue a couple of months ago with them.
This is what they told me:

"[Thaddeus Cortez] and his team will work with Engineering to establish contact with Google's Go team to help them provide a solution on their end.
We've discussed various ways of fixing this on GitHub's side, but none of the potential solutions are ideal. The meta tag approach comes with the problem of information leakage which we consider a serious security risk. It's a trade-off we are not willing to make. And, a toggleable option in the web UI would require us to turn our static error pages into dynamic ones which would come with several different issues, most notably performance and UX implications as well as added complexity in terms of code maintenance.

We really think, the Go team needs to integrate support for other sources like GitHub Enterprise (or one of our competitor's products), and not just GitHub.com."

@cjs

This comment has been minimized.

Copy link

commented Mar 6, 2018

Does vgo have the same reliance on the meta tag returned by the ?go-get=1 query string that go get has?

@james-r-smith

This comment has been minimized.

Copy link

commented Mar 8, 2018

@amnonbc & @sdboyer,
I work for a GHE customer as well, and this was their response to us:

The "go-import" meta tag is used for installing Go applications from a URL, and is indeed supported on GitHub Enterprise, there is a caveat however:
 
To avoid leaking information about repositories when the appliance is in private mode, GitHub Enterprise doesn't return the meta tag unless you're authenticated. The ideal solution is for Go to be modified to allow authentication options, but for now, you can work around this by adding .git to your repository paths. For example:
import "[ghe-hostname]/bevns/HelloWorld.git"

Or for go get:
go get -v [ghe-hostname]/bevns/HelloWorld.git

Alternatively, another workaround is to manually clone the repository into your $GOPATH and then run go-install. For example:

git clone https://[ghe-hostname]/bevns/HelloWorld $GOPATH/src/[ghe-hostname]/bevns/HelloWorld
go install [ghe-hostname]/bevns/HelloWorld

Some other customers using Go have implemented a workaround where Go import statements refer to an internal server whose sole purpose is to direct go get to the proper repository URLs. The proxy server would respond to both HTTP and HTTPS and can return a meta tag that will point to your GitHub Enterprise instance.

For example, you could have an import statement of the form import my.ghe.domain/user/project, which should result in the following HTTP request:
https://my.ghe.domain/user/project?go-get=1

This should then return a meta tag that looks something like this:
<meta name="go-import" content="my.ghe.domain git https://my.ghe.domain/user/project.git">

That should then result in the correct repository being cloned.

They also added this:

Please also keep in mind that this proxy server would need to be a standalone server and should not be configured on your GitHub Enterprise server.
 
The best solution will come when the Go team accepts and merges #17898. That will detect the headers sent from both the GitHub Enterprise appliance and GitHub.com, and use that for verification that a Git repository is being requested.
 
I would also like to mention that some alternative Go package managers such as https://glide.sh/ work with the Go toolchain and support private Git servers and repositories.

@amnonbc

This comment has been minimized.

Copy link

commented Mar 12, 2018

@james-r-smith yes we use the former hack as a workaround at the moment.
The problem is that it forces the users of your library to modify their import paths in a way that is not obvious (i.e. appending .git). This adds unnecessary friction, especially to new users.

@james-r-smith

This comment has been minimized.

Copy link

commented Mar 14, 2018

@amnonbc, I wholeheartedly agree.

@cjs

This comment has been minimized.

Copy link

commented Mar 27, 2018

With the GitHub Enterprise 2.13 release, on-premises GitHub Enterprise instances will now send the <meta> tags expected by go get for private instances and all repositories.

For instances that are in private mode, you will still need to manage authentication for the git pull phase of go get, but there are easy workarounds such as putting an authentication token in your .netrc or the insteadOf technique to force SSH as mentioned above.

x/vgo, from what I can tell currently only supports GitHub.com, but that's to be expected considering it is still in proposal status.

@rsc rsc added the NeedsFix label Mar 30, 2018

@rsc

This comment has been minimized.

Copy link
Contributor

commented Mar 30, 2018

There's a header we can look for in the go-get response to understand that this is a GitHub Enterprise server and can be accessed using the usual GitHub code. We should do that.

@gopherbot

This comment has been minimized.

Copy link

commented Apr 18, 2018

Change https://golang.org/cl/107657 mentions this issue: cmd/go/internal/modfetch: update for new gitrepo backend

@gopherbot

This comment has been minimized.

Copy link

commented Apr 18, 2018

Change https://golang.org/cl/107656 mentions this issue: cmd/vgo/internal/modfetch/gitrepo: add general git repo access

gopherbot pushed a commit to golang/vgo that referenced this issue Apr 25, 2018
cmd/vgo/internal/modfetch/gitrepo: add general git repo access
A goal of introducing modules was to move away from invoking
version control tools directly, but it has become clear that we're not
ready to do that today. GitHub in particular imposes draconian limits
on HTTPS API access that they don't impose on ordinary Git access.
And we can avoid for now breaking company setups using private
Git servers.

Because GitHub Enterprise now serves ?go-get=1 queries in a way
that is compatible with old go get, dropping back to version control
tools makes that compatible with vgo too.

The next CL hooks this code into the rest of vgo.

For golang/go#24915.
For golang/go#23955.
For golang/go#24076.

Change-Id: I76bea30081047ab68286a5d095a0d55872c5a1a3
Reviewed-on: https://go-review.googlesource.com/107656
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@solderjs

This comment has been minimized.

Copy link

commented Sep 20, 2018

I see that this is marked as closed, but I've been unable to find an example of the solution.

I have private repos that will require API keys in deployment environments and I'm trying to figure out where I tell go.mod "when you see this import, construct this url with this token instead of whatever you would normally do".

So in the case of

go get awesome.io/go/sauce

The end result should be

git clone https://token:xxxxx@git.awesome.com/awesome/go-sauce.git

Can anyone give me an update on the best practice way to accomplish this? Something that doesn't require modifying an existing server would be preferred.

@dmitris

This comment has been minimized.

Copy link
Contributor

commented Sep 25, 2018

@coolaj86 - could you use SSH keys instead of tokens (that's how I believe the authentication to GitHub Enterprise server normally works)? If so, you can add to your (and everyone's in your org 😭 )~/.gitconfig file:

[url "ssh://git@git.awesome.com"]
    insteadOf = https://git.awesome.com

if (with a bit of luck such as GHE >= 2.13 - https://enterprise.github.com/releases/2.13.0/notes) your Git server replies to requests from the go tool to https://git.awesome.com/awesome/go-sauce?go-get=1 with a meta tag <meta name="go-import" content="git.awesome.com/awesome/go-sauce git https://git.awesome.com/awesome/go-sauce.git", then the go tool will end up cloning or updating from ssh://git@git.awesome.com/awesome/go-sauce.git.

I really wish #17898 and @matope's https://go-review.googlesource.com/c/go/+/37176 could be revisited and finished - so that it wouldn't be necessary for all users to "hack" their .gitconfig files with the insteadOf mantras before they can use go get and build with Go modules.

Now suppose you don't have yet such a wonderful server support (GHE < 2.13 etc.) - you can still make the go tol fetch your repos and use them in modules. You would need to add a replace section to go.mod where for each "real" package (module) listed in the require section that you want to replace:

require (
      git.awesome.com/awesome/go-sauce v1.0.0
)
replace (
     git.awesome.com/awesome/go-sauce => git.awesome.com/awesome/go-sauce.git v1.0.0
     // other module rewrites
)
  • notice the .git suffix in the replacemend module - I believe it is necessary (to prevent go from sending ?go-get=1 requests that the server doesn't know how to handle).
  • the insteadOf settings in ~/.gitconfig are still needed as above so that git would use the ssh keys for authentication

The replace pattern can also be applied to use custom git mirrors (let's assume opensource.git.awesome.com mirrors https://github.com/golang/net into its golang/net repo) with:

replace (
   golang.org/x/net => opensource.git.awesome.com/golang/net.git v0.0.0-20180911220305-26e67e76b6c3
)
  • once again, notice the .git suffix in the replacement module which be not necessary with GHE >= 2.13 but I believe in all cases speeds up the checkouts/updates by avoiding an extra roundtrip with the ?go-git=1 request.
@dmitris

This comment has been minimized.

Copy link
Contributor

commented Sep 25, 2018

another option is to use personal access tokens with a credential helper - then you don't need to put insteadOf in ~/.gitconfig -see https://help.github.com/enterprise/2.14/user/articles/creating-a-personal-access-token-for-the-command-line/ , https://help.github.com/enterprise/2.14/user/articles/updating-credentials-from-the-osx-keychain/ and https://help.github.com/enterprise/2.14/user/articles/caching-your-github-password-in-git/ (the last page has tabs for Mac, Windows, and Linux / All)

@coolaj86 - I think this would fit your use case:

Would be curious if that works for you.

@yulrizka

This comment has been minimized.

Copy link

commented Nov 30, 2018

Since I'm using jenkins which stored the username & password/access-token, my workaround is using a GIT_ASKPASS wrapper script

git-askpass-wrapper.sh

#! /bin/sh
case "$1" in
  Username*)
    echo $GIT_USERNAME
    ;;
  Password*)
    echo $GIT_PASSWORD
    ;;
  *)
    echo "unknown argument: '$1'"
esac
$ export GIT_USERNAME=some-user
$ export GIT_PASSWORD=some-password
$ GIT_ASKPASS=the/path/to/git-askpass-wrapper.sh go [command here]

It works because when git require credentials, it will call the wrapper script. You can modified the script to your need (for example, if you have different creds for different repo).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.