cmd/go: permit marking a module as private in go.mod

[donatj]

The new proxy system requires individual machines environmental variables to be set up for private modules. This means knowledge about the build that is not kept in the repo. This is not ideal.

I believe a good feature would be the ability to mark a module as private in the go.mod rather than depending on each machine setting it's environmental variables correctly.

Originally posted by @donatj in #33980 (comment)

[tommyknows]

Definitely agree. Telling everybody at your company to "set the GOPRIVATE=..." seems impractical, and having this information lying in the go.mod file, and thus checked in, would be a more straightforward solution.

That would make it possible to pull a repo from a git server and built it, without configuring you environment.

Although I wonder how to solve the syntax-problem.
The modules-syntax seems to be fairly simple and I cannot think of an intuitive way to mark modules as private.
For example:

module myprivate.git/module

require (
    myprivate.git/dependency // private
    myprivate.git/another // private indirect
)

Feels clunky (and embedding machine-information in comments is not a nice solution imho)
Also, could you set the main module to private, too (is there a need for it)? If it is coming from a private repo, should it be noted as such?
e.g. module myprivate.git/module // private?

[henderjon]

It feels like we got rid of GOPATH only to replace it by a convoluted series of new GOPATHs (i.e. GOPRIVATE, GOPROXY, GONOPROXY, GOSUMDB).

[ianlancetaylor]

CC @bcmills @jayconrod

[bcmills]

(I could've sworn I'd seen this issue before, but I can't find a duplicate right now...)

CC @hyangah @marwan-at-work @thepudds @rogpeppe

[bcmills]

The major problem with indicating private repos in the go.mod file is that “private” is relative to the proxy.

For example: if your company is running its own (authenticated) corp module proxy, then it is entirely reasonable for the go command to be able to fetch your private-to-corp modules from that proxy when it is in use. If you switch to some other proxy — regardless of whether it is proxy.golang.org or something else — then those modules become “private” (because the proxy doesn't have access to them).

So “private” is fundamentally relative to the proxy, not relative to the module dependency.

[marwan-at-work]

I remember the issue very well, but I can't seem to find it either :/

However, I do sympathize with the issue here, because you can't force all of your team members to set up the correct configuration and environment variables on their machines.

And since "private" is relative to the proxy, maybe it's worth considering adding the GOPROXY variable to the go.mod file itself (as well as the GONOSUMDB and friends)

Or maybe, just be able to set any of the go env options in the go.mod file itself. However, the non zero value of an environment variable could take precdence.

This way, you don't need to set a "private directive" in the go.mod file, but still resolve the issue mentioned above.

That said, one can simply have a .env file in their repo and indicate in the README that those environment variables must be set before running a go command, so I'm also happy to not add that kind of complexity to go.mod.

[thepudds]

(I could've sworn I'd seen this issue before, but I can't find a duplicate right now...)

I think it was discussed as part of the larger #25530 discussion, such as in #25530 (comment) and some related comments there, as well as some of that discussion I think then triggered some additions to the proposal document, such as:

One possibility to further reduce exposure of private module path text is to provide additional ways to set $GONOSUMDB, although it is not clear what those should be. A top-level module's source code repository is an attractive place to want to store configuration such as $GONOSUMDB and $GOPROXY, but then that configuration changes depending on which version of the repo is checked out, which would cause interesting behavior when testing old versions, whether by hand or using tools like git bisect.

I don't recall a specific issue aside from #25530.

[tommyknows]

So “private” is fundamentally relative to the proxy, not relative to the module dependency.

That is true.
However, from your statement, it seems like running your own proxy will be "expected"?

For example, at a company with ~50 people, we have our own (private) Gitlab instance running. However, we are only ~5 devs actually using Go.
Setting up and maintaining a proxy for 5 people seems like overkill.
(Obviously, coordinating 5 people to set the same environment variable is doable, but is it nice?)

Normally, public projects will not depend on private repos.
This means that private projects will most likely live in the same environment as their dependencies.

That said, setting GOPROXY in the modules file would be needed for this to work.
Having a private proxy indicated in the go.mod file would mean no env-config would be needed, and "private" repos would not need to be declared.
Having no proxy (or a public one) set in the go.mod file would mean one could use "private" within the go.mod file to indicate that go should talk directly to the specified git server for these repos.

[heschi]

@tommyknows: I think it's expected that some organizations will want to, but certainly not everybody. However, keep in mind that (as touched on in #25530 (comment)) the go.mod file is immutable in your module's version control. If your company grows to the point where it does make sense for you to run your own proxy, you probably won't want it to be bypassed if someone happens to depend on an old module version from before the proxy was set up.

[bcmills]

@tommyknows, if you configure a proxy in your go.mod file, and I write some other module that depends on yours, and we have annotated private modules separately, what should happen?

Moreover: suppose that I have five developers but fifty repos: now adding the environment variable is actually less work overall than annotating all of the go.mod files individually.

At the other extreme, suppose that I have fifty developers and only one repo: now I don't need to annotate dependencies at all, because there's only one module containing private things and it doesn't depend on any other private dependencies.

The cost/benefit tradeoff you're describing is only favorable to the go.mod annotation in between those two extremes. Do we have solid evidence that that's where most developers fall? (Absent evidence, we should arguably bias toward not adding new annotations.)