testing: fuzz tests not discovered unless package path specified or GOMAXPROCS=1

[kokes]

What version of Go are you using (go version)?

$ go version
go version devel go1.17-988d0248b4 Thu Jul 8 16:39:05 2021 +0000 darwin/amd64

Does this issue reproduce with the latest release?

Yes (release meaning dev.fuzz)

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/okokes/Library/Caches/go-build"
GOENV="/Users/okokes/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/okokes/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/okokes/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/Users/okokes/sdk/gotip"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/Users/okokes/sdk/gotip/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="devel go1.17-988d0248b4 Thu Jul 8 16:39:05 2021 +0000"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/okokes/git/smda3/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/81/4jydp7kn51n6p68z88sqnkzc0000gn/T/go-build934674902=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

When running go test -v -fuzz with GOMAXPROCS > 1 and ./... target, fuzzing doesn't start, the test binary just hangs. If I drop the // +build gofuzzbeta tag, the fuzzing sometimes does start (but it is not flushed to stdout).

If I set GOMAXPROCS=1 or I specify the exact path of my fuzzed package, fuzzing starts almost immediately and all the logs flush every three seconds.

Here's a concrete reproducer. Sorry for using a pre-existing project - when I tried a minimal reproducer... it all worked all of a sudden.

# commit 7d9672fab55920d245a0b78636e4ba512c025930
git clone https://github.com/kokes/smda -b experiments/fuzzing
cd smda
# this hangs
GOMAXPROCS=2 gotip test -v -run=NONE -fuzz=FuzzDelimiter ./...
# these two work
GOMAXPROCS=1 gotip test -v -run=NONE -fuzz=FuzzDelimiter ./...
GOMAXPROCS=2 gotip test -v -run=NONE -fuzz=FuzzDelimiter ./src/database

I even tried building go from source, introducing various fmt.Printlns around the codebase. It seemed like CoordinateFuzzing was never even triggered, but I couldn't figure out why.

---

To try and isolate as much of my local software as possible, I ran a clean golang Docker image, built Go's dev.fuzz from scratch and ran the test command than would hang on my Mac. It did the same thing on this linux/amd64 image, so I don't think it's specific to Darwin or my setup.

What did you expect to see?

Fuzzing stats from logStats

What did you see instead?

Nothing, go test hangs.

[jayconrod]

Thanks for reporting this. I believe this is a symptom of #46312: we don't yet have support for fuzzing multiple targets in multiple packages.

I tried this out, and go test did appear to hang. ps showed a couple of worker processes running on github.com/kokes/smda/src/database. I sent SIGINT to one of them, and the whole command wrapped up with normal output printed all at once:

...
fuzzing, elapsed: 36.0s, execs: 1556484 (43234/sec), workers: 2, interesting: 6
fuzzing, elapsed: 39.0s, execs: 1685642 (43220/sec), workers: 2, interesting: 6
fuzzing, elapsed: 42.0s, execs: 1817287 (43268/sec), workers: 2, interesting: 6
fuzzing, elapsed: 45.0s, execs: 1945143 (43221/sec), workers: 2, interesting: 6
fuzzing, elapsed: 47.8s, execs: 2053839 (42963/sec), workers: 2, interesting: 6
--- PASS: FuzzDelimiterInference (47.81s)
PASS
ok  	github.com/kokes/smda/src/database	47.962s
testing: warning: no targets to fuzz
PASS
ok  	github.com/kokes/smda/src/query	0.221s [no targets to fuzz]
...

So the problems here are at least:

• One only target is actually being fuzzed.
• We're not streaming log messages.

I think GOMAXPROCS=2 may have something to do with the output streaming, but I'm not sure what exactly. I think go test may not stream output when multiple packages are running concurrently.

cc @golang/fuzzing