New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: cmd/go: -mirror flag for go-get #21292

Open
Merovius opened this Issue Aug 3, 2017 · 7 comments

Comments

Projects
None yet
7 participants
@Merovius

Merovius commented Aug 3, 2017

I'd like to informally propose to add a -mirror flag to go-get (or an equivalent mechanism, say a GO_GET_MIRROR environment variable).
The prompt for this is this article; the general thought has been spooking in my head ever since the discussion around vendoring started.

The semantics I'd propose are simple: The mirror flag provides the URL of an HTTP service, hosting meta-information about where to fetch repositories from. So, e.g. when calling go get -mirror https://example.com/go-mirror vanityurl.io/pkg/leftpad, the go tool would make an HTTP request to https://example.com/go-mirror/vanityurl.io/pkg/leftpad. It would then expect that to contain a go-import meta tag, just like without -mirror. Fetching would continue as normal from there.

I think this would address the concerns both around the reliability of vanity-imports and the general concerns about Go's distributed package registry, while marking off all requirements in the (very reasonable) checklist from the article above:

  • Discovery would work as before. If you don't use -mirror, nothing changes. A simple go get vanityurl.io/pkg/foo would find the package.
  • Auditability is preserved, because a company can provide an internal mirror of the audited go code, pointing the served meta-tags to internal code repositories containing only audited versions
  • Publishing works as before. It also still enables third-party companies with a vested interest in go (say, Cloudflare, GitHub or Google) to run highly reliable vanity-domain hosting.
  • Ownership is given, if vanity URLs are used. Even if, say Cloudflare, where to be hosting the HTTP for my vanity URLs as a service, I would still own the domain itself, so could, at any point, redirect the requests to a different service.
  • Namespacing stays as before decentralized and using the DNS.

It would also mean, any entity could run a public mirror of their own. So, for example, archive.org could decide to provide publicly hosted mirror. The decentralized nature of that would effectively prevent depublishing or censorship; even if an author depublishes vanityurl.io/leftpad, existing packages using that import path wouldn't even necessarily break, they would just need a -mirror, that still keeps that package around.

Of course the mirror would have full control over what code is served, so it needs to be fully trusted by the person running go-get. Which is why I think an explicit -mirror flag would be the right way. We might consider requiring mirrors to be served via https.

@gopherbot gopherbot added this to the Proposal milestone Aug 3, 2017

@gopherbot gopherbot added the Proposal label Aug 3, 2017

@rakyll

This comment has been minimized.

Member

rakyll commented Aug 3, 2017

Is this proposal suggesting a federated package serving service implicitly? It is hard to connect some dots if not.

the reliability of vanity-imports

It reads like this section explains the reliability of the package serving service rather than reliability of the vanity imports. The original reliability of vanity-import paths problem comes from the necessity to run a server. A mirror is a more complicated server to maintain.

The users don't want to maintain a server but we need some sort of centralized naming to have the flexibility to move code between providers. Today some of us use DNS (via vanity import paths) for naming, but configuring and maintaining a vanity server is a lot of work.

If we will depend on a centralized service to provision names, we surely need to consider some mirroring and flexibility to run the service on your own.

If this is not about a centralized package serving service, does the mirror can override what's being served at github.com/com/rakyll/pizza?

$ go get -mirror https://example.com/go-mirror github.com/rakyll/pizza

Then, this will lock me to mirrors with the right configuration if github.com goes away one day, right? I don't think without sorting out what naming will be handled (can be federated), it is hard to move forward in this problem.

@SamWhited

This comment has been minimized.

Member

SamWhited commented Aug 3, 2017

This is great stuff to think about; thanks for reading my thoughts and following up with a proposal!

I personally think this is an important step to get right and not have to redo anytime soon; before any concrete proposals can be considered (if they're needed at all, maybe it turns out everything is fine, who knows?), we probably want some experience reports of what the actual problems with the current system are (my blog post was a broad overview of some theoretical problems, but it would be good to hear from organizations and individuals about specific real world problems they have).

Maybe you'd like to write up something about your experiences using go get and what went wrong (or right)?

@slrz

This comment has been minimized.

slrz commented Aug 3, 2017

Note that you can achieve this effect today without any modification to Go tooling by making use of e.g. Git's URL rewriting functionality. I'm sure other VCS have similar options.

@Merovius

This comment has been minimized.

Merovius commented Aug 3, 2017

@rakyll I should have made that assumption explicit: From a technical perspective, there isn't anything preventing me paying a third party for reliably hosting the go-imports HTML reliably already, while staying independent of them (in the simplest case, I can host it as a github-page and add a corresponding record to my DNS. If I want to move away from them later, I can change the records). There is the additional problem, that this currently requires any other HTTPS to also be served by that third party, which is true. I am not so much personally concerned with that (I view it more as "whoever serves the page also serves the go-imports meta tag"). This proposal, in any case, isn't about making that easier or harder; exactly because I already consider it totally possible and fine with the current tooling.

The interaction with this proposal is, that mirrors could alleviate most of the problems caused by an unreliable vanity-url hosting, namely that code isn't reachable, if the domain or the HTTPS server behind it or the hoster of the repo breaks. Mirroring, as laid out here, would means neither the DNS server of the domain, nor the HTTPS server nor the repo hoster would ever need to be hit. It wouldn't make it any easier to run a vanity domain reliably, but it would make it less of a problem if it isn't. And yes, it does mean you'd need a mirror set up pre-emptively.

The idea is, to get all the benefits of something like crates.io (reliable hosting of the code). Just like there is a godoc.org, I'd imagine someone running gopackages.org as a public, transparently caching mirror. But since mirrors are just HTTP servers and a command line flag, it isn't susceptible to the same kind of attacks as crates.io (or npm, or cpan, or…) due to its central nature is.

@SamWhited TBQH, I'm not experiencing a lot of pain myself. This was originally meant to solve most of the problems as vendoring (and the idea is from the time before that existed). I mostly want to have an alternative to centralized code hosting on record, for the problems people are discussing now.

I agree that calling this a proposal at this stage was probably premature, though… It probably makes more sense to at least implement this as a separate tool first.

@slrz That still requires go-get to actually resolve vanity.io, the HTTPS request to it succeeding and only then can the remote be overwritten - after the rewrite got manually configured, of course.

At that point, we are basically at the level of "you can also just not use go-get and git clone into your GOPATH". Technically correct, but clearly not ergonomic.

@rakyll

This comment has been minimized.

Member

rakyll commented Aug 4, 2017

The idea is, to get all the benefits of something like crates.io (reliable hosting of the code).

I agree with this aspect.

(if they're needed at all, maybe it turns out everything is fine, who knows?)

I know that some companies are hacking around go-get (or replacing go-get with their own) to be able to only checkout from a trusted source/version and there was an earlier conversation about a "mirror" flag might be the fix. I am a bit skeptical about the experience reports at this point because (a) I haven't seen some common complaints represented there yet, (b) I am on the lazier side and even I haven't contributed anything so far even though I have a list of 10 items at least.

@rsc

This comment has been minimized.

Contributor

rsc commented Aug 7, 2017

I want to solve this problem as part of the work on package management, but it probably won't be exactly a flag. Will put this on hold for now.

@rsc rsc added the Proposal-Hold label Aug 7, 2017

@bweston92

This comment has been minimized.

bweston92 commented Nov 2, 2017

@Merovius if you use git specifically.
For each remote like GitHub/BitBucket etc you can have the following.

git config --global url."https://internal.domain.com/".insteadOf "https://github.com/"

@rsc rsc changed the title from Proposal: cmd/go: -mirror flag for go-get to proposal: cmd/go: -mirror flag for go-get Nov 2, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment