testing: calling TempDir() from a Benchmark causes it to fail

[hannahhoward]

What version of Go are you using (go version)?

$ go version
go version go1.15 darwin/amd64

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""

GOARCH="amd64"

GOBIN=""

GOCACHE="/Users/hannah/Library/Caches/go-build"

GOENV="/Users/hannah/Library/Application Support/go/env"

GOEXE=""

GOFLAGS=""

GOHOSTARCH="amd64"

GOHOSTOS="darwin"

GOINSECURE=""

GOMODCACHE="/Users/hannah/projects/go/pkg/mod"

GONOPROXY=""

GONOSUMDB=""

GOOS="darwin"

GOPATH="/Users/hannah/projects/go"

GOPRIVATE=""

GOPROXY="https://proxy.golang.org,direct"

GOROOT="/usr/local/Cellar/go/1.15/libexec"

GOSUMDB="sum.golang.org"

GOTMPDIR=""

GOTOOLDIR="/usr/local/Cellar/go/1.15/libexec/pkg/tool/darwin_amd64"

GCCGO="gccgo"

AR="ar"

CC="clang"

CXX="clang++"

CGO_ENABLED="1"

GOMOD="/Users/hannah/projects/gosrc/src/go.mod"

CGO_CFLAGS="-g -O2"

CGO_CPPFLAGS=""

CGO_CXXFLAGS="-g -O2"

CGO_FFLAGS="-g -O2"

CGO_LDFLAGS="-g -O2"

PKG_CONFIG="pkg-config"

GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/qj/9f3_14352b1b07rxs4pm9tmc0000gn/T/go-build263434137=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

• wrote a benchmark that uses B.TempDir

What did you expect to see?

• success, assuming no other failures

What did you see instead?

• testing.go:905: TempDir: mkdir /var/folders/qj/9f3_14352b1b07rxs4pm9tmc0000gn/T/BenchmarkRoundtripSuccess_test-20-10000723322760/003: no such file or directory

---

I dug into the benchmark, determined the first run succeeded, but on a short benchmark that is run again with N>1, the failure happens.

I dug into the benchmark.go source code and I believe the problem is that after every benchmark is run, common.runCleanup is called. runCleanup will delete the top level source folder under which TempDir() creates more temp directories. However, the top level source directory is only created once due to the sync.Once mutex.

I went ahead and wrote a test and possible fix which I submitted to Gerril -- https://go-review.googlesource.com/c/go/+/250950

I have no idea if the fix is right (it does make the test pass) but the test should demonstrate the problem.

[bcmills]

Neat find! This also applies to calls to (*testing.T).TempDir within a Cleanup callback: https://play.golang.org/p/_qWZd56AwyP

CC @ianlancetaylor @rogpeppe @bradfitz

[hannahhoward]

For sure! I submitted the PR on Gerrill as a demo of the issue and a possible fix. Unfortunately, I discovered it while writing benchmarks for my job that I definitely need to get back to. So I probably won't have time to go through the revision process for the PR -- but I suspect you all will find a more ideal solution :) Please let me know if you need any more from me.

In the mean time, I'll just workaround with ioutil.

[bcmills]

Thanks for at least getting us started with a report and a solid test case!

[gopherbot]

Change https://golang.org/cl/251297 mentions this issue: testing: fix TempDir failure in Cleanup and Benchmark

[anacrolix]

Any chance this would be backported to Go 1.15?

[ianlancetaylor]

We usually only backport changes that are serious bugs that don't have a workaround, and this one has a pretty easy workaround.