Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/vgo: use vcs tools instead of https APIs of code hosting sites #24915

Closed
rsc opened this issue Apr 18, 2018 · 14 comments

Comments

@rsc
Copy link
Contributor

commented Apr 18, 2018

In https://research.swtch.com/vgo-module I wrote that

We want to move away from invoking version control tools such as bzr, fossil, git, hg, and svn to download source code. These fragment the ecosystem: packages developed using Bazaar or Fossil, for example, are effectively unavailable to users who cannot or choose not to install these tools. The version control tools have also been a source of exciting security problems. It would be good to move them outside the security perimeter.

We still do, but it seems clear that this will be a large burden especially for corporate users. It is probably better to separate the introduction of versioning from the imposition of this restriction. We can focus on making the migration to versioned packages as painless as possible and do drop version control tool support at some later date, if at all. (One possibility is that if we get a reliable global proxy then you'd have to explicitly ask not to use the proxy, at which point the security and fragmentation concerns are significantly lessened.)

@rsc rsc added this to the vgo milestone Apr 18, 2018

@gopherbot

This comment has been minimized.

Copy link

commented Apr 18, 2018

Change https://golang.org/cl/107657 mentions this issue: cmd/go/internal/modfetch: update for new gitrepo backend

@gopherbot

This comment has been minimized.

Copy link

commented Apr 18, 2018

Change https://golang.org/cl/107656 mentions this issue: cmd/vgo/internal/modfetch/gitrepo: add general git repo access

@lrewega

This comment has been minimized.

Copy link

commented Apr 19, 2018

Has there been any consideration given to the idea of allowing a command to be invoked in lieu of an HTTP proxy? E.g.: GOPROXY_CMD=foo vgo get rsc.io/sampler@v1.3.1 could perhaps invoke a binary foo with the request URL that vgo would usually make:

foo rsc.io/sampler/@v/1.31.zip

In the minimal case, this would allow for use of tools such as curl to inject cookies into the request: GOPROXY_CMD="curl -fsSL -b my-cookie:sekret"

In the more complex case, a script or program could be invoked and decide to use version control or other tools for certain domains. E.g. if foo was a script containing:

#!/bin/sh
case "$1" in
  rsc.io/*)    curl -sSL "$1";;
  code.corp/*) ssh code.corp -- cat "/some/corporate/repos/$1";;
  *)           echo >&2 "no fetching of 3rd party code: $1"; exit 1;;
esac

Git's GIT_SSH and GIT_SSH_COMMAND environment variables inspired this concept.

@sdwarwick

This comment has been minimized.

Copy link

commented Apr 21, 2018

Why does it seem like the whole concept of URLs and URNs is being re-invented? URLs include b access protocol, and URNs provide a broadly understood concept of uniqueness.

gopherbot pushed a commit to golang/vgo that referenced this issue Apr 25, 2018
cmd/go/internal/modfetch: update for new gitrepo backend
Drop API-based access to Bitbucket, Gerrit, GitHub,
now that we can (again) talk directly to their Git repos.

Also drop general LatestAt query in favor of plain Latest
(at current time on default branch), which is all that we
needed and all that makes sense when talking directly
to Git servers.

For golang/go#24915 (still need hg, svn, bzr, fossil).
Fixes golang/go#23955 (no GitHub API calls, so no rate limits).
Fixes golang/go#24085 (no GitHub API calls, so no 403 responses).
Fixes golang/go#24076 (GitHub enterprise's go-get=1 pages are now OK).
Fixes golang/go#24102 (no more canonical GitHub import path checks).

Change-Id: I99fadb09a565398da3ec88f17691217aca7bf571
Reviewed-on: https://go-review.googlesource.com/107657
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@rsc

This comment has been minimized.

Copy link
Contributor Author

commented Apr 25, 2018

@lrewega At least to start, no, we don't plan to do that. Let's see how things go with just ordinary network-level proxy support instead of command-level.

@sdwarwick, I don't understand what you're getting at. The notation here is the same as go get always has been. We're not changing that.

gopherbot pushed a commit to golang/vgo that referenced this issue Apr 25, 2018
cmd/vgo/internal/modfetch/gitrepo: add general git repo access
A goal of introducing modules was to move away from invoking
version control tools directly, but it has become clear that we're not
ready to do that today. GitHub in particular imposes draconian limits
on HTTPS API access that they don't impose on ordinary Git access.
And we can avoid for now breaking company setups using private
Git servers.

Because GitHub Enterprise now serves ?go-get=1 queries in a way
that is compatible with old go get, dropping back to version control
tools makes that compatible with vgo too.

The next CL hooks this code into the rest of vgo.

For golang/go#24915.
For golang/go#23955.
For golang/go#24076.

Change-Id: I76bea30081047ab68286a5d095a0d55872c5a1a3
Reviewed-on: https://go-review.googlesource.com/107656
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@flibustenet

This comment has been minimized.

Copy link

commented Apr 26, 2018

Could be possible to keep both http api and vcs to remove issues like #25102 and begin now to experiment with http api ?

@paulbdavis

This comment has been minimized.

Copy link

commented May 30, 2018

Does this mean that modules in private repos can be fetched over ssh? I am having trouble getting things to build that use private repos as dependencies.

@rsc

This comment has been minimized.

Copy link
Contributor Author

commented Jun 6, 2018

Everything is using git again. Not sure why this wasn't closed.

@paulbdavis, The git invocations always specify https://. They don't fall back to ssh:// like old "go get" used to. If you need to avoid https:// my suggestion would be to use git's insteadOf configuration feature in your $HOME/.gitconfig, so that 'https://' is accepted and understood by git to mean ssh.

@paulbdavis

This comment has been minimized.

Copy link

commented Jun 8, 2018

@rsc thanks, I had tried this before, but that was before using vcs again.

Are there any plans to continue to allow git+ssh access going forward? (via some env var, git config as above, fallbacks instead of failure, etc.). Without SSH, I am using a combo of vendoring, GOPROXY, and @myitcv's modpub, then compiling in the GOPATH with "vanilla" go in my CI. Tedious, but works so I can use vgo now.

Going forward I would expect the proxy availability to be more robust for handling this use case, but simply allowing git+ssh to be available at least as an option would simplify using private git repos now and in the future (for git users at least).

@myitcv

This comment has been minimized.

Copy link
Member

commented Jun 8, 2018

@paulbdavis to use git with ssh, you'll need a config setting along the lines of:

git config url.git@github.com:.insteadof https://github.com/

See https://git-scm.com/docs/git-config for more details.

@paulbdavis

This comment has been minimized.

Copy link

commented Jun 12, 2018

@rsc thanks, I had tried this before, but that was before using vcs again.

Are there any plans to continue to allow git+ssh access going forward? (via some env var, git config as above, fallbacks instead of failure, etc.). Without SSH, I am using a combo of vendoring, GOPROXY, and @myitcv's modpub, then compiling in the GOPATH with "vanilla" go in my CI. Tedious, but works so I can use vgo now.

Going forward I would expect the proxy availability to be more robust for handling this use case, but simply allowing git+ssh to be available at least as an option would simplify using private git repos now and in the future (for git users at least).

@myitcv

This comment has been minimized.

Copy link
Member

commented Jun 13, 2018

@paulbdavis did you see my response #24915 (comment)?

Looks like you reposted the same comment (#24915 (comment)) in response?

@paulbdavis

This comment has been minimized.

Copy link

commented Jun 14, 2018

Sorry, not sure how that got posted again four days later.

Yes, I got git+ssh working as you described.

What I am trying to ask is if using vcs or allowing vcs to be used is going to remain a part of vgo going forward into to 1.11 preview implementation.

From reading the blog posts, eliminating vcs usage seems to be a goal. The use case of having private repositories is a fairly common one, and using vcs tooling makes it easy to get working on any machine that has your ssh keys versus making sure you have a working proxy set up for each set of private repos being used (for example, private libraries for multiple client projects)

@myitcv

This comment has been minimized.

Copy link
Member

commented Jun 16, 2018

Yes, I got git+ssh working as you described.

Great.

What I am trying to ask is if using vcs or allowing vcs to be used is going to remain a part of vgo going forward into to 1.11 preview implementation.

Yes, it will. Indeed per #24915 I quote:

We still do, but it seems clear that this will be a large burden especially for corporate users. It is probably better to separate the introduction of versioning from the imposition of this restriction. We can focus on making the migration to versioned packages as painless as possible and do drop version control tool support at some later date, if at all.

@golang golang locked and limited conversation to collaborators Jun 16, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
7 participants
You can’t perform that action at this time.