testing: start test log for caching before TestMain

[bakatz]

What version of Go are you using (go version)?

go version go1.10 windows/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\ben\AppData\Local\go-build
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=C:\Users\ben\dev\go
set GORACE=
set GOROOT=C:\Go
set GOTMPDIR=
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\ben\AppData\Local\Temp\go-build167965298=/tmp/go-build -gno-record-gcc-switches

What did you do?

1. Create a test_something.go file whose tests require a main_test.go file for setup/teardown
2. Create several tests in test_something.go
3. Run all of the tests
4. Observe that the code in main_test is executed
5. Change a non-go file that the test depends on

What did you expect to see?

main_test is executed again and a cache miss occurs.

What did you see instead?

main_test is not executed again and a cache hit occurs. See https://github.com/bakatz/golangissue21422 for more info.

[ianlancetaylor]

What is test_main? Do you mean TestMain?

[bakatz]

@ianlancetaylor whoops, yes! Meant main_test.go, i.e. TestMain. Will update the issue now.

EDIT: Issue updated.

[ianlancetaylor]

Thanks. For complete clarity, can you provide a small self-contained example?

[bakatz]

Sure, give me a few minutes to throw together some sample code and I'll update you when it's ready.

[bakatz]

My fault: as it turns out I can no longer reproduce this issue today. It occurred yesterday with the same exact version of go, set of tests, etc.

I'll reopen this issue if needed once I figure out what set of steps caused the original failure (clearly the steps I gave are not accurate)

Thanks

[bakatz]

@ianlancetaylor Figured it out. Basically, the issue is with SQL migrations. My TestMain is responsible for running these migrations when the test run starts, but since .sql files are not imported by any .go files, I'm assuming that the go test runner has no idea about these migration files and doesn't know to invalidate the cache and run the TestMain code in order to run the migrations. This means that every time I add a new migration, it won't be applied unless I run go test -count=1 ./... manually

To be fair, I don't think it should be Go's responsibility to keep track of extraneous files in the repository and know which ones are critical to the testing process, but I also think it could make sense to always run TestMain (but not necessarily all other tests), even when there's a cache hit, in order to avoid issues like this. Alternatively, the CLI flag that I mentioned in my first post would be useful for me. What do you think?

[ysmolski]

@bakatz can you please provide an example of code with the issue?

[bakatz]

@ysmolsky sure, apologies on the delay on this; been really busy at work. I'll provide an example repo when I next have a free moment. Should be within the next day or two

[andybons]

Ping on that info, @bakatz