Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
proposal: cmd/go: -mirror flag for go-get #21292
I'd like to informally propose to add a -mirror flag to go-get (or an equivalent mechanism, say a GO_GET_MIRROR environment variable).
The semantics I'd propose are simple: The
I think this would address the concerns both around the reliability of vanity-imports and the general concerns about Go's distributed package registry, while marking off all requirements in the (very reasonable) checklist from the article above:
It would also mean, any entity could run a public mirror of their own. So, for example, archive.org could decide to provide publicly hosted mirror. The decentralized nature of that would effectively prevent depublishing or censorship; even if an author depublishes
Of course the mirror would have full control over what code is served, so it needs to be fully trusted by the person running go-get. Which is why I think an explicit
Is this proposal suggesting a federated package serving service implicitly? It is hard to connect some dots if not.
It reads like this section explains the reliability of the package serving service rather than reliability of the vanity imports. The original reliability of vanity-import paths problem comes from the necessity to run a server. A mirror is a more complicated server to maintain.
The users don't want to maintain a server but we need some sort of centralized naming to have the flexibility to move code between providers. Today some of us use DNS (via vanity import paths) for naming, but configuring and maintaining a vanity server is a lot of work.
If we will depend on a centralized service to provision names, we surely need to consider some mirroring and flexibility to run the service on your own.
If this is not about a centralized package serving service, does the mirror can override what's being served at
Then, this will lock me to mirrors with the right configuration if github.com goes away one day, right? I don't think without sorting out what naming will be handled (can be federated), it is hard to move forward in this problem.
This is great stuff to think about; thanks for reading my thoughts and following up with a proposal!
I personally think this is an important step to get right and not have to redo anytime soon; before any concrete proposals can be considered (if they're needed at all, maybe it turns out everything is fine, who knows?), we probably want some experience reports of what the actual problems with the current system are (my blog post was a broad overview of some theoretical problems, but it would be good to hear from organizations and individuals about specific real world problems they have).
Maybe you'd like to write up something about your experiences using
@rakyll I should have made that assumption explicit: From a technical perspective, there isn't anything preventing me paying a third party for reliably hosting the go-imports HTML reliably already, while staying independent of them (in the simplest case, I can host it as a github-page and add a corresponding record to my DNS. If I want to move away from them later, I can change the records). There is the additional problem, that this currently requires any other HTTPS to also be served by that third party, which is true. I am not so much personally concerned with that (I view it more as "whoever serves the page also serves the go-imports meta tag"). This proposal, in any case, isn't about making that easier or harder; exactly because I already consider it totally possible and fine with the current tooling.
The interaction with this proposal is, that mirrors could alleviate most of the problems caused by an unreliable vanity-url hosting, namely that code isn't reachable, if the domain or the HTTPS server behind it or the hoster of the repo breaks. Mirroring, as laid out here, would means neither the DNS server of the domain, nor the HTTPS server nor the repo hoster would ever need to be hit. It wouldn't make it any easier to run a vanity domain reliably, but it would make it less of a problem if it isn't. And yes, it does mean you'd need a mirror set up pre-emptively.
The idea is, to get all the benefits of something like crates.io (reliable hosting of the code). Just like there is a godoc.org, I'd imagine someone running gopackages.org as a public, transparently caching mirror. But since mirrors are just HTTP servers and a command line flag, it isn't susceptible to the same kind of attacks as crates.io (or npm, or cpan, or…) due to its central nature is.
@SamWhited TBQH, I'm not experiencing a lot of pain myself. This was originally meant to solve most of the problems as vendoring (and the idea is from the time before that existed). I mostly want to have an alternative to centralized code hosting on record, for the problems people are discussing now.
I agree that calling this a proposal at this stage was probably premature, though… It probably makes more sense to at least implement this as a separate tool first.
@slrz That still requires go-get to actually resolve vanity.io, the HTTPS request to it succeeding and only then can the remote be overwritten - after the rewrite got manually configured, of course.
At that point, we are basically at the level of "you can also just not use go-get and git clone into your GOPATH". Technically correct, but clearly not ergonomic.
I agree with this aspect.
I know that some companies are hacking around go-get (or replacing go-get with their own) to be able to only checkout from a trusted source/version and there was an earlier conversation about a "mirror" flag might be the fix. I am a bit skeptical about the experience reports at this point because (a) I haven't seen some common complaints represented there yet, (b) I am on the lazier side and even I haven't contributed anything so far even though I have a list of 10 items at least.