cmd/go: load $GOROOT/go.env before loading user go.env

[rsc]

I've been looking at the patches that distributions apply to Go. We'd like to remove as many as possible, so that once reproducible builds land (#24904, #57007), Linux distributions that rebuild from source should still be shipping the identical binaries that are distributed on https://go.dev/dl.

Debian patches are here: https://sources.debian.org/patches/golang-1.15/1.15.15-1~deb11u4/
Fedora patches are here: https://src.fedoraproject.org/rpms/golang/tree/rawhide

Debian's are all test changes. Fedora has one non-test change in cmd/link that maybe we should pick up if gcc has been fixed. Fedora also changes the default values for GOPROXY and GOSUMDB. Setting aside the fact that they are opening their users to supply chain attacks, it would be good if they didn't have to modify the binaries to do it. We also expect over on #57001 that Fedora will want to default GOTOOLCHAIN=local, and perhaps Debian will too.

Today the default Go environment variables are modified using 'go env -w' which writes to filepath.Join(os.UserConfigDir(), "go/env"). On my Mac that's $HOME/Library/Application Support/go/env. On Linux that's $HOME/.config/go/env.

I propose that when the go command loads that file, it first loads $GOROOT/go.env and then the user file. Any setting in the user file overwrites an equivalent setting in the $GOROOT/go.env, and the environment overwrites both (as always).

Then distributions that want to modify the defaults do not need to edit source code and create non-standard binaries. They can just write a $GOROOT/go.env.

[seankhliao]

naming it go.env, would people think that it's like go.mod / go.work and want to use it locally (ask for it as a feature request) for per project (build) config?

[mvdan]

They could come to that wrong conclusion with any filename, I think :) Even with base.env, one might think that it would work in any module directory.

I personally think proper docs would be enough. This would realistically only appear in downstream Go distributions and packages, and in general, users wouldn't go look inside GOROOT directly.

[rsc]

Perhaps we should accept a go.env in the work module or next to go.work as well. There have been various issues asking for per-module configuration in go.mod. Go.mod is not the right place for that, which is why those were declined, but if we already have the search order $GOROOT/go.env, $userconfigdir/go.env, it might make sense to insert $workdir/go.env between them, where $workdir is the directory containing the go.work (if any) or else the one containing the work module's go.mod.

Of course, go.env would only apply in the work module and would be ignored in dependencies. There are security concerns to think through for people who clone other repos and then open them in editors. We'd also have to decide what GOENV means and especially what GOENV=off means.

This would address:

• cmd/go: permit marking a module as private in go.mod #33985
• proposal: cmd/go: allow GOPRIVATE, GONOSUMDB, GONOPROXY to be defined in go.mod #48441
• proposal: cmd/go: add project config to go.mod #42343
• proposal: cmd/go: record default build tags in go.mod #43288 (perhaps, using GOFLAGS=)
• proposal: cmd/go: introduce a build configurations file #39005 (partially; only one configuration)

In #57001, there is some complication due to having both the GOTOOLCHAIN environment variable and the toolchain go.mod line. If the work module's go.env file were a thing, we could probably drop the toolchain line and rely on the ability to set GOTOOLCHAIN in the work module go.env instead.

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[rsc]

The reactions here are uniformly positive. Does anyone object to accepting this proposal?

[thepudds]

Hi Russ, minor clarification -- is a module/workspace go.env as described above in #57179 (comment) currently considered part of this proposal, or is that just a possible future extension?

[hyangah]

is a module/workspace go.env as described above in #57179 (comment) currently considered part of this proposal, or is that just a possible future extension?

I think the work module go.env #57179 (comment) should be a separate proposal. Introducing $GOROOT/go.env is less controversial and may be necessary to help the reproducible builds & extended forward compatibility support move forward smoothely. But I guess the work module go.env idea has more details to clarify.

• why introducing a separate file (go.env) instead of extending go.work?
• will all go env vars be allowed in the work module's go.env? (GOROOT/GOPATH/GOBIN/CC/CXX/AR/GOPROXY/GOSUMDB/*FLAGS...)
• can there be an option to disable the work module's go.env? Or should this go.env be ignored by default?
  (For example, an editor may choose to ignore the go.env until a user explicitly approve it is trusted - like VS Code's workspace trust mode).
• meaning of GOENV.
• tool to help users debug how they ended up with certain go env vars. (We saw many users confused after running go env -w long ago and didn't realize their global go env was affected. Now there will be two more sources).

[rsc]

Thanks @hyangah. I started answering these questions but there are too many significant problems with the per-module go.env that don't arise with the $GOROOT/go.env. Let's keep this proposal scoped to just the $GOROOT/go.env.

The $GOROOT/go.env source is an alternative to editing the source code to change the hard-coded defaults, so that's less of a change.

[rsc]

Marking this likely accept but only for $GOROOT/go.env, not a per-module go.env.

[rsc]

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

[gopherbot]

Change https://go.dev/cl/462198 mentions this issue: cmd/go: introduce GOROOT/go.env and move proxy/sumdb config there

[rsc]

I confused myself and submitted this before it was accepted. Reopening.

[rsc]

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

[rsc]

Done.

[thockin]

Is there a write-up of the issues with a per-module or per-workspace go.env? That seems like it would be helpful in Kubernetes, especially once we adopt go workspaces fully. Our need for wrapper scripts will approach zero.

[gopherbot]

Change https://go.dev/cl/497437 mentions this issue: cmd/go: set default GOTOOLCHAIN in GOROOT/go.env

[alexsaezm]

I was trying to integrate this new feature in Fedora when I found that TestScript/mod_sumdb_golang fails:

vcs-test.golang.org rerouted to http://127.0.0.1:33295
https://vcs-test.golang.org rerouted to https://127.0.0.1:42771
go test proxy running at GOPROXY=http://127.0.0.1:32775/mod
--- FAIL: TestScript (0.18s)
    --- FAIL: TestScript/mod_sumdb_golang (0.06s)
        script_test.go:132: 2023-06-30T17:21:00Z
        script_test.go:134: $WORK=/tmp/cmd-go-test-2792608462/tmpdir2295187342/mod_sumdb_golang511923858
        script_test.go:156:
            # Test default GOPROXY and GOSUMDB (0.056s)
            > env GOPROXY=
            > env GOSUMDB=
            > go env GOPROXY
            [stdout]
            direct
            > stdout '^https://proxy.golang.org,direct$'
        script_test.go:156: FAIL: testdata/script/mod_sumdb_golang.txt:5: stdout ^https://proxy.golang.org,direct$: no match for `(?m)^https://proxy.golang.org,direct$` in stdout
FAIL
FAIL	cmd/go	49.471s

At first I thought it was something in the isolated Fedora's build environment but I was able to reproduce this in a fresh machine by just cloning go, setting GOPROXY=direct and GOSUMDB=off in the go.env and then run all.bash.

Is this expected? Is this new feature not meant to be used during testing and only after everything has passed?

[SuperSandro2000]

Fedora also changes the default values for GOPROXY and GOSUMDB. Setting aside the fact that they are opening their users to supply chain attacks

How is directly downloading the binaries exposing users more than trusting sum.golang.org/proxy.golang.org?

[balasanjay]

How is directly downloading the binaries exposing users more than trusting sum.golang.org/proxy.golang.org?

What if the source repository had logic that looked (roughly) like this:

if (request is from specific IP space) { return subtly evil code; }
else { return benign code; }

With GOSUMDB disabled, authors in the victim IP space would see no indication that anything bad was happening.

With GOSUMDB enabled, there's a globally unique hash for every version of a package, so the source repository could not do such a thing without one of the two versions being rejected by the global checksum database. (And clients are not "trusting" this global checksum database so much as cryptographically proving that its committing to a single hash per package version, proving that its never rolling back history, etc using the same underlying tech as Certificate Transparency). See https://www.youtube.com/watch?v=KqTySYYhPUE for a great talk at Gophercon 2019 about the exact strategy.