cmd/go: load $GOROOT/go.env before loading user go.env

[rsc]

I've been looking at the patches that distributions apply to Go. We'd like to remove as many as possible, so that once reproducible builds land (#24904, #57007), Linux distributions that rebuild from source should still be shipping the identical binaries that are distributed on https://go.dev/dl.

Debian patches are here: https://sources.debian.org/patches/golang-1.15/1.15.15-1~deb11u4/
Fedora patches are here: https://src.fedoraproject.org/rpms/golang/tree/rawhide

Debian's are all test changes. Fedora has one non-test change in cmd/link that maybe we should pick up if gcc has been fixed. Fedora also changes the default values for GOPROXY and GOSUMDB. Setting aside the fact that they are opening their users to supply chain attacks, it would be good if they didn't have to modify the binaries to do it. We also expect over on #57001 that Fedora will want to default GOTOOLCHAIN=local, and perhaps Debian will too.

Today the default Go environment variables are modified using 'go env -w' which writes to filepath.Join(os.UserConfigDir(), "go/env"). On my Mac that's $HOME/Library/Application Support/go/env. On Linux that's $HOME/.config/go/env.

I propose that when the go command loads that file, it first loads $GOROOT/go.env and then the user file. Any setting in the user file overwrites an equivalent setting in the $GOROOT/go.env, and the environment overwrites both (as always).

Then distributions that want to modify the defaults do not need to edit source code and create non-standard binaries. They can just write a $GOROOT/go.env.

[seankhliao]

naming it go.env, would people think that it's like go.mod / go.work and want to use it locally (ask for it as a feature request) for per project (build) config?

[mvdan]

They could come to that wrong conclusion with any filename, I think :) Even with base.env, one might think that it would work in any module directory.

I personally think proper docs would be enough. This would realistically only appear in downstream Go distributions and packages, and in general, users wouldn't go look inside GOROOT directly.

[rsc]

Perhaps we should accept a go.env in the work module or next to go.work as well. There have been various issues asking for per-module configuration in go.mod. Go.mod is not the right place for that, which is why those were declined, but if we already have the search order $GOROOT/go.env, $userconfigdir/go.env, it might make sense to insert $workdir/go.env between them, where $workdir is the directory containing the go.work (if any) or else the one containing the work module's go.mod.

Of course, go.env would only apply in the work module and would be ignored in dependencies. There are security concerns to think through for people who clone other repos and then open them in editors. We'd also have to decide what GOENV means and especially what GOENV=off means.

This would address:

• cmd/go: permit marking a module as private in go.mod #33985
• proposal: cmd/go: allow GOPRIVATE, GONOSUMDB, GONOPROXY to be defined in go.mod #48441
• proposal: cmd/go: add project config to go.mod #42343
• proposal: cmd/go: record default build tags in go.mod #43288 (perhaps, using GOFLAGS=)
• proposal: cmd/go: introduce a build configurations file #39005 (partially; only one configuration)

In #57001, there is some complication due to having both the GOTOOLCHAIN environment variable and the toolchain go.mod line. If the work module's go.env file were a thing, we could probably drop the toolchain line and rely on the ability to set GOTOOLCHAIN in the work module go.env instead.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[rsc]

The reactions here are uniformly positive. Does anyone object to accepting this proposal?

[thepudds]

Hi Russ, minor clarification -- is a module/workspace go.env as described above in #57179 (comment) currently considered part of this proposal, or is that just a possible future extension?

[hyangah]

is a module/workspace go.env as described above in #57179 (comment) currently considered part of this proposal, or is that just a possible future extension?

I think the work module go.env #57179 (comment) should be a separate proposal. Introducing $GOROOT/go.env is less controversial and may be necessary to help the reproducible builds & extended forward compatibility support move forward smoothely. But I guess the work module go.env idea has more details to clarify.

• why introducing a separate file (go.env) instead of extending go.work?
• will all go env vars be allowed in the work module's go.env? (GOROOT/GOPATH/GOBIN/CC/CXX/AR/GOPROXY/GOSUMDB/*FLAGS...)
• can there be an option to disable the work module's go.env? Or should this go.env be ignored by default?
  (For example, an editor may choose to ignore the go.env until a user explicitly approve it is trusted - like VS Code's workspace trust mode).
• meaning of GOENV.
• tool to help users debug how they ended up with certain go env vars. (We saw many users confused after running go env -w long ago and didn't realize their global go env was affected. Now there will be two more sources).

[rsc]

Thanks @hyangah. I started answering these questions but there are too many significant problems with the per-module go.env that don't arise with the $GOROOT/go.env. Let's keep this proposal scoped to just the $GOROOT/go.env.

The $GOROOT/go.env source is an alternative to editing the source code to change the hard-coded defaults, so that's less of a change.

[rsc]

Marking this likely accept but only for $GOROOT/go.env, not a per-module go.env.

[rsc]

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

[gopherbot]

Change https://go.dev/cl/462198 mentions this issue: cmd/go: introduce GOROOT/go.env and move proxy/sumdb config there

[rsc]

I confused myself and submitted this before it was accepted. Reopening.

[rsc]

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

[rsc]

Done.

[thockin]

Is there a write-up of the issues with a per-module or per-workspace go.env? That seems like it would be helpful in Kubernetes, especially once we adopt go workspaces fully. Our need for wrapper scripts will approach zero.

[thepudds]

Hi @thockin, to my knowledge, the most complete list of concerns are the ones @hyangah asked about above in #57179 (comment).

A couple of days ago in #57001 (comment), Ian wrote the following, which references that comment above:

[...] we decided against enabling go.env per module. See the concerns in #57179 (comment) . There is no current proposal for go.env per module, and it seems hard to do.

[ianlancetaylor]

@thockin Unfortunately many environment variables are inherently global, and it's unclear what should happen if they are set for a particular module. What would it mean to set CC, GOARCH, GOBIN, GOROOT, CGO_ENABLED, GOEXPERIMENT for a single module?

It may make sense to permit setting environment variables in a go.work file, I'm not sure.

[thockin]

ACK that there's a lot more in go env than just the things I care about. It still feels a little baby-with-the-bathwater, to me. That said, my needs are forward-looking - we have not even adopted workspaces yet, and that is FAR more valuable.