Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: deprecate func GOROOT #51473

Closed
rittneje opened this issue Mar 4, 2022 · 44 comments
Closed

runtime: deprecate func GOROOT #51473

rittneje opened this issue Mar 4, 2022 · 44 comments
Labels
FixPending Issues that have a fix which has not yet been reviewed or submitted. Proposal Proposal-Accepted
Milestone

Comments

@rittneje
Copy link

rittneje commented Mar 4, 2022

The runtime.GOROOT() function is currently implemented as follows:

  1. If the GOROOT environment variable is set, return its value.
  2. Else, return the value that was burned into the binary at compile time.

Unfortunately, this implementation cannot be relied upon to have any semantic meaning in general.

If the intent is to know what GOROOT the binary was compiled with, then consulting the GOROOT environment variable is wrong. Instead, the burned in value should be added to debug.BuildInfo.

If the intent is to know the GOROOT on the current machine, then consulting the burned in value is wrong, as it cannot handle the following (non-exhaustive) scenarios:

  1. The binary may have been compiled on another machine, with a totally irrelevant GOROOT.
  2. The go installation may have been moved since the binary was compiled.
  3. The PATH environment variable may have changed since the binary was compiled.
  4. The current machine might not even have a go installation.

Instead, the go env GOROOT command should be used. In particular, this change needs to be made for go/build.Default to function properly in all cases.

@mvdan
Copy link
Member

mvdan commented Mar 4, 2022

I think this is an interesting proposal, and I agree with the current confusion causing problems at times.

I assume that, if a program wants to try both approaches, they could have code that first tries one approach (like go env GOROOT), and then falls back to the other (like debug.BuildInfo).

@mvdan
Copy link
Member

mvdan commented Mar 4, 2022

Instead, the go env GOROOT command should be used. In particular, this change needs to be made for go/build.Default to function properly in all cases.

I fully agree with your suggestion to use go env GOROOT, but I'm worried about changing go/build.Default.GOROOT. Note that Default is filled at init time, so it has a noticeable flat cost added to any program that ends up importing the package. Most of that cost currently comes from many calls to os.Getenv; adding an execution of go env would make it significantly slower, making many Go programs take a few milliseconds longer to start.

So when it comes to go/build, I think we should either keep it as-is with a warning in its documentation, or fully deprecate the Context.GOROOT field, instead pointing people to better alternatives like go env GOROOT to just query that one field, or to https://pkg.go.dev/golang.org/x/tools/go/packages for a whole-sale replacement of the Context.Import API.

@rittneje
Copy link
Author

rittneje commented Mar 4, 2022

I fully agree with your suggestion to use go env GOROOT, but I'm worried about changing go/build.Default.GOROOT. Note that Default is filled at init time, so it has a noticeable flat cost added to any program that ends up importing the package

One possible solution would be to add a private flag to go/build.Context, such as isDefault bool. If true, then it won't set any environment variables when calling go list, since at best this accomplishes nothing, and at worst causes bugs.

go/src/go/build/build.go

Lines 1182 to 1188 in 81767e2

cmd.Env = append(os.Environ(),
"GOOS="+ctxt.GOOS,
"GOARCH="+ctxt.GOARCH,
"GOROOT="+ctxt.GOROOT,
"GOPATH="+ctxt.GOPATH,
"CGO_ENABLED="+cgo,
)

@mvdan
Copy link
Member

mvdan commented Mar 4, 2022

Would that help with the added init cost, though, if its init func may still call go list?

@rittneje
Copy link
Author

rittneje commented Mar 4, 2022

@mvdan I'm not suggesting that the init func for go/build call go list. I'm saying that when go/build calls go list within importGo (which already happens today), it should not attempt to set any of the environment variables in the case of build.Default.

Alternatively, at the top of importGo, it should call go env GOROOT rather than relying on the (possibly broken) value of build.Default.GOROOT.

@mvdan
Copy link
Member

mvdan commented Mar 4, 2022

Ah, I see. I was thinking mainly of end users directly reading build.Default.GOROOT. Presumably we need to do something about that one way or another.

@bcmills
Copy link
Contributor

bcmills commented Mar 4, 2022

The one case I can think of where runtime.GOROOT() may still be helpful is for tests, where it may be easier to invoke runtime.GOROOT() than to shell out to go env GOROOT. The latter may not be accurate anyway if the go in $PATH is not the one used to compile the test. (Consider ~/go-devel/bin/go test ., where ~/go-devel is a GOROOT but is not listed in $PATH.)

At least in principle, it is possible for the go command to ensure that GOROOT is set accurately in the test process's environment when it is run. Then again, if we do that then the test could run os.Getenv("GOROOT") just as easily as runtime.GOROOT(). 🤔

@rittneje
Copy link
Author

rittneje commented Mar 4, 2022

@bcmills I'm not sure what the use cases are for looking up GOROOT while unit testing, but one thing to keep in mind is that it is possible to compile a test binary with -c and then run it later. Consequently, it would run into the same general issues I mentioned above.

@bcmills
Copy link
Contributor

bcmills commented Mar 4, 2022

That's true, but if you compile a test binary with -c and then run it later, you are also responsible for replicating any conditions necessary for the test to succeed (working directory, environment, etc.).

Even so, I suspect that most tests that depend on runtime.GOROOT() only do so via go/build (for example, because they are testing analyzers for Go source code), and fixing #51483 would make those tests more robust either way.

@rittneje
Copy link
Author

rittneje commented Mar 4, 2022

Gotcha. Actually, as an optimization, since we know that $GOROOT takes precedence if it's set, then go/build can just check the env var first before shelling out to go env GOROOT. Then maybe such test cases can mandate the env var gets set for stabler results (as in the example you mentioned).

I guess another approach would be to have go test manipulate the PATH variable for the sub-process, by prefixing it with $(go env GOROOT)/bin. Not sure if that would be too surprising, but it does have the nice side effect of ensuring that any go commands that get spawned by the test do the more expected thing.

@rsc
Copy link
Contributor

rsc commented Mar 16, 2022

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

@rsc
Copy link
Contributor

rsc commented Mar 23, 2022

@bradfitz reports that he has uses of runtime.GOROOT in GitHub actions where built binaries want to find the toolchain that built them, so they can invoke it to do other things.

It seems like we probably need to enumerate the uses people have for runtime.GOROOT and what they should do instead. It doesn't help to just say "Deprecated". We have to say what to use instead.

@bradfitz
Copy link
Contributor

For example:

bradfitz@tsdev:~/src/tailscale.com$ git grep runtime.GOROOT
net/tlsdial/tlsdial_test.go:    cmd := exec.Command(filepath.Join(runtime.GOROOT(), "bin", "go"),
tstest/archtest/qemu_test.go:                   cmd := exec.Command(filepath.Join(runtime.GOROOT(), "bin", "go"),
tstest/integration/integration.go:      goBin := filepath.Join(runtime.GOROOT(), "bin", "go"+exe())
version/modinfo_test.go:        goTool := filepath.Join(runtime.GOROOT(), "bin", "go"+exe())

@rittneje
Copy link
Author

rittneje commented Mar 25, 2022

@bradfitz I'm not sure which repos those are, but this seems to at least partially align with with what @bcmills mentioned above re: tests. In that situation, we can make go test automatically set the GOROOT env var when running the test binary, and then the unit tests just look at that env var. No need for runtime.GOROOT() specifically, although it would have the same effect.

With regards to non-test binaries that intend to find the toolchain that built them, this feels somewhat ill-defined, for the reasons mentioned above, as well as issues with -trimpath. And to consult GOROOT in this case seems strange, because it could point to any arbitrary toolchain installation.

In short:

  1. If you need the specific GOROOT with which the binary was compiled, fetch the burned in value (from debug.BuildInfo). But it should be noted that (a) it will not work if compiled with -trimpath, and (b) it might refer to a non-existent folder on the current machine. So if you are trying to use GOROOT to invoke the go toolchain, your binary won't be portable. Users have no control over the GOROOT selection, other than to recompile. Also, it should be noted that even this isn't guaranteed to give you the right toolchain, as the user may have changed the installation at that path after the fact.
  2. If you need the currently configured GOROOT, run go env GOROOT. As an optimization, you may check for the GOROOT env var first. The resulting binary will be portable, but the result will have no guaranteed relation to the GOROOT your binary was compiled with. Users should set GOROOT or PATH appropriately to select different GOROOTs.
  3. As a "light" version of (2), you may exclusively rely on the GOROOT env var, and fail if it is unset.
  4. If you are in complete control of the execution environment (such as with a GitHub action), there is no need for the standard library to participate at all. Either configure your PATH appropriately (so that exec'ing go does the right thing), or set some arbitrary environment variable of your choosing (possibly GOROOT but it doesn't matter) with the path of the desired Go installation and have your application read it.

@bcmills
Copy link
Contributor

bcmills commented Apr 1, 2022

@bradfitz, thanks for those examples. I cloned the tailscale repo and ran go test -trimpath ./net/tlsdial ./tstest/... ./version with a go built from head, and those tests already fail in that mode.

(They happen to pass when run with golang.org/dl/go1.18 or golang.org/dl/gotip because the wrappers for those binaries set GOROOT explicitly in the process environment: https://cs.opensource.google/go/dl/+/master:internal/version/version.go;l=67;drc=f798e20c9ec1dfe45a3a432d168172504ecf3b88)

~/src/tailscale.com$ ASSUME_NO_MOVING_GC_UNSAFE_RISK_IT_WITH=$(go version | grep -o 'devel .* +0000') go test -trimpath ./net/tlsdial ./tstest/... ./version -count=1
--- FAIL: TestFallbackRootWorks (0.00s)
    tlsdial_test.go:54: mkcert: fork/exec bin/go: no such file or directory,
FAIL
FAIL    tailscale.com/net/tlsdial       0.027s
ok      tailscale.com/tstest    0.021s
--- FAIL: TestInQemu (0.00s)
    --- FAIL: TestInQemu/386 (0.00s)
        qemu_test.go:69: failed:
        qemu_test.go:73: output didn't contain "I am linux/386":
FAIL
FAIL    tailscale.com/tstest/archtest   0.017s
--- FAIL: TestOneNodeUpNoAuth (0.01s)
    integration_test.go:518: DERP httpsrv listener: 127.0.0.1:39475
    integration.go:63: failed to find go at bin/go
    stuntest.go:64: STUN server shutdown
--- FAIL: TestAddPingRequest (0.01s)
    integration_test.go:518: DERP httpsrv listener: 127.0.0.1:43717
    integration.go:63: failed to find go at bin/go
    stuntest.go:64: STUN server shutdown
--- FAIL: TestOneNodeUpAuth (0.01s)
    integration_test.go:518: DERP httpsrv listener: 127.0.0.1:45951
    integration.go:63: failed to find go at bin/go
    stuntest.go:64: STUN server shutdown
--- FAIL: TestCollectPanic (0.01s)
    integration_test.go:518: DERP httpsrv listener: 127.0.0.1:40909
    integration.go:63: failed to find go at bin/go
    stuntest.go:64: STUN server shutdown
--- FAIL: TestTwoNodes (0.01s)
    integration_test.go:518: DERP httpsrv listener: 127.0.0.1:36241
    integration.go:63: failed to find go at bin/go
    stuntest.go:64: STUN server shutdown
--- FAIL: TestNoControlConnWhenDown (0.01s)
    integration_test.go:518: DERP httpsrv listener: 127.0.0.1:36325
    integration.go:63: failed to find go at bin/go
    stuntest.go:64: STUN server shutdown
--- FAIL: TestControlTimeLogLine (0.01s)
    integration_test.go:518: DERP httpsrv listener: 127.0.0.1:37683
    integration.go:63: failed to find go at bin/go
    stuntest.go:64: STUN server shutdown
--- FAIL: TestNodeAddressIPFields (0.01s)
    integration_test.go:518: DERP httpsrv listener: 127.0.0.1:41095
    integration.go:63: failed to find go at bin/go
    stuntest.go:64: STUN server shutdown
--- FAIL: TestStateSavedOnStart (0.01s)
    integration_test.go:518: DERP httpsrv listener: 127.0.0.1:37661
    integration.go:63: failed to find go at bin/go
    stuntest.go:64: STUN server shutdown
--- FAIL: TestOneNodeExpiredKey (0.01s)
    integration_test.go:518: DERP httpsrv listener: 127.0.0.1:40891
    integration.go:63: failed to find go at bin/go
    stuntest.go:64: STUN server shutdown
--- FAIL: TestLogoutRemovesAllPeers (0.01s)
    integration_test.go:518: DERP httpsrv listener: 127.0.0.1:45759
    integration.go:63: failed to find go at bin/go
    stuntest.go:64: STUN server shutdown
--- FAIL: TestOneNodeUpWindowsStyle (0.01s)
    integration_test.go:518: DERP httpsrv listener: 127.0.0.1:38921
    integration.go:63: failed to find go at bin/go
    stuntest.go:64: STUN server shutdown
FAIL
FAIL    tailscale.com/tstest/integration        0.076s
?       tailscale.com/tstest/integration/testcontrol    [no test files]
ok      tailscale.com/tstest/integration/vms    0.092s
?       tailscale.com/tstest/integration/vms/gen        [no test files]
ok      tailscale.com/tstest/natlab     0.009s
--- FAIL: TestFindModuleInfo (0.00s)
    modinfo_test.go:22: failed to build tailscaled: fork/exec bin/go: no such file or directory
FAIL
FAIL    tailscale.com/version   0.014s
FAIL

There is also no guarantee in general that even a non-empty runtime.GOROOT() reports the GOROOT with which the test binary was actually built. (For example, the test could be built with go test -c and the GOROOT subsequently rebuilt at an entirely different version.)

That's of course fine for tailscale's own code if you're ok with those failure modes, but may be further evidence that we should discourage that pattern in general. 😅

A more robust approach might be to simply look for go in the usual $PATH and either skip or fail the test if go env GOVERSION does not match runtime.GOVERSION(). (See https://go.dev/play/p/5Vg4frf8pIk for a sketch. Maybe we should publish that as a supported test-helper somewhere?)

@rsc
Copy link
Contributor

rsc commented Apr 6, 2022

Is "finding the go command" the only thing we need GOROOT for? Probably?
We have internal/testenv.GoToolCmd so clearly we need it frequently.
Maybe we can solve that more limited problem?

Possibilities:

  1. New API and a new environment variable that 'go test' would start setting when running the test binary.
  2. New API using $GOROOT, which 'go test' would start setting when running the test binary.
  3. Have 'go test' put the Go bin directory at the front of $PATH before running the test binary. (No new API!)

Number 3 would fit well with non-test code like go/build and go/packages that already assume that looking for "go" in your PATH is the one you want.

Thoughts on finding the go command?

And thoughts on any other needs from GOROOT?

@bcmills
Copy link
Contributor

bcmills commented Apr 11, 2022

  1. New API and a new environment variable that 'go test' would start setting when running the test binary.

That wouldn't necessarily even require a new environment variable: a flag might suffice. I think the API is the tricky part.

Maybe:

package testing

// GOROOT reports the GOROOT with which the test was built, if known.
// This function may report a non-empty GOROOT even if runtime.GOROOT
// returns the empty string (such as if the test was built with -trimpath).
func GOROOT() (string, bool)

@bcmills
Copy link
Contributor

bcmills commented Apr 11, 2022

  1. New API using $GOROOT, which 'go test' would start setting when running the test binary.

Having go test set $GOROOT would hide erroneous existing uses of runtime.GOROOT: having the variable set would mean that runtime.GOROOT always reports the test's real GOROOT, which would mask cases where a corresponding production binary would be unable to locate it.

I think this option is not really much better than deciding not to deprecate runtime.GOROOT() at all, which IMO is unfortunate because it interacts so badly with -trimpath.

@bcmills
Copy link
Contributor

bcmills commented Apr 11, 2022

  1. Have 'go test' put the Go bin directory at the front of $PATH before running the test binary. (No new API!)

I like that approach a lot overall. It looks like GOROOT/bin normally contains only go and gofmt — neither of which should pollute the search namespace very much — and I suspect that a large fraction of tests that run go and/or gofmt already assume that the binary matches the Go version used to build and run the test itself.

That would also make it fairly trivial to identify the GOROOT location in general, since the test binary could just exec go env GOROOT.

@rsc
Copy link
Contributor

rsc commented Apr 13, 2022

It sounds like people like "Have 'go test' put the Go bin directory at the front of $PATH before running the test binary." so let's start with that. @bcmills, want to write a CL for that?

Then tests and things like go/packages that exec "go" will work fine without runtime.GOROOT.

At that point, what is left for use cases for runtime.GOROOT?

@danderson
Copy link
Contributor

@rsc making sure I understand your last response: would go test still obey the GOROOT environment variable when figuring out what Go bin directory to add to $PATH?

@bcmills
Copy link
Contributor

bcmills commented Apr 19, 2022

@danderson, the go command and runtime.GOROOT would continue to respond to the GOROOT environment variable as they already do. (We would just discourage the use of runtime.GOROOT specifically, since it is overly sensitive to the presence of that variable.)

@rsc
Copy link
Contributor

rsc commented May 4, 2022

Are there any other problems with deprecating runtime.GOROOT, once we've guaranteed that "go in the PATH" is the right go during a test?

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/404134 mentions this issue: cmd/go: place GOROOT/bin at the beginning of PATH in 'go generate' and 'go test'

@rsc rsc changed the title proposal: runtime: deprecate GOROOT proposal: runtime: deprecate fun GOROOT Feb 1, 2024
@rsc rsc changed the title proposal: runtime: deprecate fun GOROOT proposal: runtime: deprecate func GOROOT Feb 1, 2024
@rsc
Copy link
Contributor

rsc commented Feb 1, 2024

Reviving. The proposal is to deprecate GOROOT in favor of running "go" from PATH when you want to run the go command, and using 'go env GOROOT' when you need to know the GOROOT.

// GOROOT returns the root of the Go tree. It uses the
// GOROOT environment variable, if set at process start,
// or else the root used during the Go build.
//
// Deprecated: The root used during the Go build will not be 
// meaningful if the binary is copied to another machine. 
// Use the system path to locate the “go” binary, and use
// “go env GOROOT” to find its GOROOT.
func GOROOT() string

@rsc
Copy link
Contributor

rsc commented Feb 1, 2024

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

@rsc rsc removed the Proposal-Hold label Feb 1, 2024
@rsc rsc moved this from Hold to Active in Proposals Feb 1, 2024
@rittneje
Copy link
Author

rittneje commented Feb 2, 2024

@rsc It might also be good to explicitly mention what runtime.GOROOT() returns if $GOROOT is not set and the binary was compiled with -trimpath since that is another sharp edge.

@bcmills
Copy link
Contributor

bcmills commented Feb 2, 2024

@rittneje, note that that was settled in #51461 and it now returns the empty string in that case. I think that's the appropriate behavior going forward.

@rittneje
Copy link
Author

rittneje commented Feb 2, 2024

@bcmills Yeah I think that behavior is fine, it's just not currently mentioned in the documentation.

@rsc
Copy link
Contributor

rsc commented Feb 8, 2024

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

Mark runtime.GOROOT deprecated:

// GOROOT returns the root of the Go tree. It uses the
// GOROOT environment variable, if set at process start,
// or else the root used during the Go build.
//
// Deprecated: The root used during the Go build will not be 
// meaningful if the binary is copied to another machine. 
// Use the system path to locate the “go” binary, and use
// “go env GOROOT” to find its GOROOT.
func GOROOT() string

@rsc rsc moved this from Active to Likely Accept in Proposals Feb 8, 2024
@rsc
Copy link
Contributor

rsc commented Feb 14, 2024

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

Mark runtime.GOROOT deprecated:

// GOROOT returns the root of the Go tree. It uses the
// GOROOT environment variable, if set at process start,
// or else the root used during the Go build.
//
// Deprecated: The root used during the Go build will not be 
// meaningful if the binary is copied to another machine. 
// Use the system path to locate the “go” binary, and use
// “go env GOROOT” to find its GOROOT.
func GOROOT() string

@rsc rsc moved this from Likely Accept to Accepted in Proposals Feb 14, 2024
@rsc rsc changed the title proposal: runtime: deprecate func GOROOT runtime: deprecate func GOROOT Feb 14, 2024
@rsc rsc modified the milestones: Proposal, Backlog Feb 14, 2024
@dmitshur dmitshur modified the milestones: Backlog, Go1.23 Feb 14, 2024
@dmitshur dmitshur added the FixPending Issues that have a fix which has not yet been reviewed or submitted. label Feb 15, 2024
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/564142 mentions this issue: runtime: deprecate GOROOT

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/564275 mentions this issue: gopls/internal/test/marker: remove runtime.GOROOT use in format.txt case

gopherbot pushed a commit to golang/tools that referenced this issue Mar 18, 2024
The runtime.GOROOT function is about to be marked as deprecated, which
will generate a diagnostic that isn't expected in format.txt test case.
Pick another hopefully somewhat future-proof identifier, since dealing
with this doesn't seem to be in scope of this test case.

For golang/go#51473.

Change-Id: I54d7ed0b43860bcd41938328211797a0cdd60e36
Reviewed-on: https://go-review.googlesource.com/c/tools/+/564275
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Robert Findley <rfindley@google.com>
@dmitshur
Copy link
Contributor

A fix CL is available but hasn't been reviewed before Go 1.23 freeze. Moving to the next milestone.

@dmitshur dmitshur modified the milestones: Go1.23, Go1.24 May 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
FixPending Issues that have a fix which has not yet been reviewed or submitted. Proposal Proposal-Accepted
Projects
Status: Accepted
Development

No branches or pull requests

10 participants