proposal: cmd/go: add a sub command to manage authentication #40189
The current state
The Go command utilitizes the ~/.netrc file, both implicitly and explicitly, to authenticate against remote servers when downloading Go code.
Go uses ~/.netrc implicitly when using "direct" downloads because git uses libcurl to fetch dependencies through https which in turn uses the ~/.netrc file to forward credentials.
Go uses ~/.netrc explicitly when authenticating against a GOPROXY server by looking for a matching "machine" URL with a valid login and password and forwards those credentials as a BasicAuth header.
A Go programmer who wants to set their credentials (whether against a proxy or VCS) must know how to create/edit the .netrc file in their home directory manually. This is has a few problems:
Many languages and tools abstract authentication management in their command line:
And the list goes on.
Go should provide a more pleasant, and less error prone, way to configure user's credentials for downloading private module dependencies.
Specifically, Go should be able to create and edit the ~/.netrc file without the user's direct manipulation of it.
Go should be able to add/edit/remove specific lines in the ~/.netrc through the Go command line.
Please note: The following syntax is arbitrary and can definitely be changed. This proposal is more about getting agreement that we should let the Go command manage the .netrc file and is not picky about what the syntax will look like.
That said, suggestion on what the command syntax would look like is welcome here.
The text was updated successfully, but these errors were encountered:
To address a couple points:
That said, I'm not sure we should rely on
#26232 is a feature request for an extensible authentication mechanism in the style of
That would provide a lot of flexibility, but I'm not sure it would make the user experience any simpler. Perhaps a
Agreed. The GOAUTH env var would probably be even more complex for users to configure so a
Could we not have a
That said, I would prefer to keep the
From a UX perspective, I don't think is a good idea. A nice gain of having a GOPROXY is that we can tell users they don't need to have "git" and other VCS binaries in their build step because a GOPROXY will take care of module fetching for you. If we had to tell them they need to make sure they have "git" just to authenticate against a GOPROXY, it feels like a step in the wrong direction. But definitely correct me if I missed something.
Agreed. But telling Go developers to install a third party tool just so they can make Go understand and pass credentials is not the most ideal. Though it's certainly better than telling Go developers to hand craft a ~/.netrc file.
Other than the above, I see two options:
I like the idea of a
According to the Cloud Functions documentation for Go, the only way to use a private Go module is to vendor it (which is very constraining).
Another problem, as it pertains to HTTP Basic Auth, is that (in accordance with RFC 7617) the
To the extent that this issue is about maintaining ~/.netrc, it sounds like we should encourage prototyping a netrc manager outside the Go command.
It would be nice to have more of a problem statement about how people authenticate to their source repos and what they need. There are many things netrc does not address.
@rsc below is my problem statement and more likely an experience report stemming from the last 3 companies I have worked at where we needed to authenticate and fetch private modules across different VCS repositories.
In all 3 companies, there were 2-3 areas when we needed to access private Go dependencies:
Problem statement: This is by far the most frequently reported issues in our internal slack channels at work. People always run into issues trying to get/upgrade a private dependency and they have forgotten to update their GOPRIVATE or they have misconfigured their .netrc or git config.
Furthermore, people rarely know the difference between GOPRIVATE and GONOSUMDB. To make somewhat of a strong statement: If they are using a GOPROXY for their private modules, they really shouldn't be setting GOPRIVATE without a GONOPROXY and vice versa. But the underlying issue is the same: more often than not, my colleagues always got their configuration wrong and got frustrated until someone else helped them fix their configuration.
This also has the side effect of dividing our internal Go community into two categories: the super-users and the non-super-users. I ultimately don't want to make people feel that they don't have sufficient knowledge or that other colleagues are more knowledgable than them. Instead, I'd want them to understand and enjoy Go's simplicity the way I have because Go tends to abstract the really difficult parts with simple and easy to understand APIs.
Problem statement: I'd be more than happy to go into detail here. But I think the downsides of the vendor folder is quite well understood to the Go team and the Go community. To name some high level issues with vendoring: bloated repositories, stale/disappearing dependencies, inability to upgrade any module that has a stale/missing transitive dependency, poor lsp/editor integration (I'd be super happy to expand on this one) and more.
Problem statement: a lot of companies might deploy their GOPROXY within VPN so that there wouldn't be a need for authentication, but then their CI/CD may not be within their VPN which deems this solution impossible.
On the other hand, if their internal GOPROXY was reachable from their CI/CD but required authentication, then they must make sure to configure GONOSUMDB correctly as well as programmatically creating a .netrc file in their CI/CD so that their private dependencies can be properly downloaded.
Problem statement: because you cannot configure a .netrc file when you deploy a Google Cloud Function, GCF forces you to always vendor your dependencies which makes for a less ideal solution. We can definitely argue that this is a GCF issue so I won't focus on this too much but just thought I'd bring it up as another example where Go tooling can enable these platforms to be more friendly.
To summarize here are the pain points from a Go developer's perspective:
This brings me to my last point:
My core angle here is to address the user experience. I'd love for the Go team and the community to tackle this issue from a UX perspective.
The only reason this issue addresses the ".netrc" file is mainly because this is what we have today. But the underlying issue is the same: how can we make the user experience for fetching private modules wonderful, pleasant and also secure!
The last 3 companies I have been employed at all used private Go modules and servers spread out over multiple git repositories and all 3 companies have faced the same struggle due to the lack of a good user experience.
To solve the above issues, there are arguably infinite ways to approach this problem. However, here's one basic suggestion that I hope it really shouldn't be a hard sell:
If we can achieve the above, we would be in a much better position than we are today.
But if we want to go further and because the design space is quite vast, I think we can look into making the experience even better by doing a little more heavy lifting such as the following suggestion:
Maybe the GOPROXY Download Protocol can be extended so that it can declare what private module paths it supports. This way the Go command can know when to consult the sumdb for a particular module coming from a particular GOPROXY.
I think the new Swift Registry Proposal is considering such option where a registry server declares the package paths it supports.
Finally, regarding this:
I started building out a CLI that basically just "edits" the .netrc file. But then I realized this only goes as far as making sure the user doesn't make syntactical-netrc errors. What if that same tool can actually do more:
I'm currently building a tool and an API that does exactly the above by consulting/authenticating with a central API which will return the GOPROXY URL as well as the GONOSUMDB configuration so that the user would not have to worry about those details and they can just run
This is still early days and the API is not open source/available yet but I hope it shows some ideas in how we can make private modules a much better experience for Go developers.
Apologies for the long winded response but I wanted to make sure I addressed everything from the last 2.5 years of working with private Go modules and I'd be very happy to talk to anyone directly because writing all of the above in a GitHub issue may not be the best medium.
It's clear we need an answer for the underlying problem that both this issue and #45611 are trying to solve, but neither really solves it completely. We don't have a coherent plan right now. This is something the go command team hopes to look at in the next cycle. Putting on hold until then.