testing: calling os.Getenv in TestMain does not factor into test cache

[rittneje]

What version of Go are you using (go version)?

$ go version
go version go1.19.8 darwin/amd64

Does this issue reproduce with the latest release?

Yes (1.20.3)

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE="auto"
GOARCH="amd64"
GOBIN=""
GOCACHE="/tmp/.gocache"
GOENV="/Users/rittneje/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/rittneje/test/pkg/mod"
GONOPROXY="[redacted]"
GONOSUMDB="[redacted]"
GOOS="darwin"
GOPATH="/Users/rittneje/test"
GOPRIVATE="[redacted]"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/Users/rittneje/go1.19.8"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/Users/rittneje/go1.19.8/pkg/tool/darwin_amd64"
GOVCS="public:git,[redacted]:git,private:off"
GOVERSION="go1.19.8"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/rittneje/test/src/envtest/go.mod"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/kf/kr7_s3xx0l12zbj3jrn082hmzy5gvy/T/go-build1303705698=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

package envtest

import (
	"os"
	"testing"
)

var foobar string

func TestMain(m *testing.M) {
	foobar = os.Getenv("FOOBAR")
	os.Exit(m.Run())
}

func TestFoo(t *testing.T) {
	if foobar != "123" {
		t.FailNow()
	}
}

$ FOOBAR=123 go test -v .
=== RUN   TestFoo
--- PASS: TestFoo (0.00s)
PASS
ok      envtest 0.142s

$ FOOBAR=1234 go test -v .
=== RUN   TestFoo
--- PASS: TestFoo (0.00s)
PASS
ok      envtest (cached)

What did you expect to see?

The second test run should fail.

What did you see instead?

The call to os.Getenv did not properly factor into the cache, so it incorrectly reported the cached result.

Note that calling os.Getenv directly inside the unit test works correctly.

package envtest

import (
	"os"
	"testing"
)

func TestFoo(t *testing.T) {
	if os.Getenv("FOOBAR") != "123" {
		t.FailNow()
	}
}

$ FOOBAR=123 go test -v .
=== RUN   TestFoo
--- PASS: TestFoo (0.00s)
PASS
ok      envtest 0.152s

$ FOOBAR=1234 go test -v .
=== RUN   TestFoo
--- FAIL: TestFoo (0.00s)
FAIL
FAIL    envtest 0.128s
FAIL

[bcmills]

Duplicate of #44625