Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testing: fuzz tests not discovered unless package path specified or GOMAXPROCS=1 #47136

Open
kokes opened this issue Jul 12, 2021 · 1 comment
Open

Comments

@kokes
Copy link

@kokes kokes commented Jul 12, 2021

What version of Go are you using (go version)?

$ go version
go version devel go1.17-988d0248b4 Thu Jul 8 16:39:05 2021 +0000 darwin/amd64

Does this issue reproduce with the latest release?

Yes (release meaning dev.fuzz)

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/okokes/Library/Caches/go-build"
GOENV="/Users/okokes/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/okokes/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/okokes/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/Users/okokes/sdk/gotip"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/Users/okokes/sdk/gotip/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="devel go1.17-988d0248b4 Thu Jul 8 16:39:05 2021 +0000"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/okokes/git/smda3/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/81/4jydp7kn51n6p68z88sqnkzc0000gn/T/go-build934674902=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

When running go test -v -fuzz with GOMAXPROCS > 1 and ./... target, fuzzing doesn't start, the test binary just hangs. If I drop the // +build gofuzzbeta tag, the fuzzing sometimes does start (but it is not flushed to stdout).

If I set GOMAXPROCS=1 or I specify the exact path of my fuzzed package, fuzzing starts almost immediately and all the logs flush every three seconds.

Here's a concrete reproducer. Sorry for using a pre-existing project - when I tried a minimal reproducer... it all worked all of a sudden.

# commit 7d9672fab55920d245a0b78636e4ba512c025930
git clone https://github.com/kokes/smda -b experiments/fuzzing
cd smda
# this hangs
GOMAXPROCS=2 gotip test -v -run=NONE -fuzz=FuzzDelimiter ./...
# these two work
GOMAXPROCS=1 gotip test -v -run=NONE -fuzz=FuzzDelimiter ./...
GOMAXPROCS=2 gotip test -v -run=NONE -fuzz=FuzzDelimiter ./src/database

I even tried building go from source, introducing various fmt.Printlns around the codebase. It seemed like CoordinateFuzzing was never even triggered, but I couldn't figure out why.


To try and isolate as much of my local software as possible, I ran a clean golang Docker image, built Go's dev.fuzz from scratch and ran the test command than would hang on my Mac. It did the same thing on this linux/amd64 image, so I don't think it's specific to Darwin or my setup.

What did you expect to see?

Fuzzing stats from logStats

What did you see instead?

Nothing, go test hangs.

@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented Jul 12, 2021

Thanks for reporting this. I believe this is a symptom of #46312: we don't yet have support for fuzzing multiple targets in multiple packages.

I tried this out, and go test did appear to hang. ps showed a couple of worker processes running on github.com/kokes/smda/src/database. I sent SIGINT to one of them, and the whole command wrapped up with normal output printed all at once:

...
fuzzing, elapsed: 36.0s, execs: 1556484 (43234/sec), workers: 2, interesting: 6
fuzzing, elapsed: 39.0s, execs: 1685642 (43220/sec), workers: 2, interesting: 6
fuzzing, elapsed: 42.0s, execs: 1817287 (43268/sec), workers: 2, interesting: 6
fuzzing, elapsed: 45.0s, execs: 1945143 (43221/sec), workers: 2, interesting: 6
fuzzing, elapsed: 47.8s, execs: 2053839 (42963/sec), workers: 2, interesting: 6
--- PASS: FuzzDelimiterInference (47.81s)
PASS
ok  	github.com/kokes/smda/src/database	47.962s
testing: warning: no targets to fuzz
PASS
ok  	github.com/kokes/smda/src/query	0.221s [no targets to fuzz]
...

So the problems here are at least:

  • One only target is actually being fuzzed.
  • We're not streaming log messages.

I think GOMAXPROCS=2 may have something to do with the output streaming, but I'm not sure what exactly. I think go test may not stream output when multiple packages are running concurrently.

cc @golang/fuzzing

@jayconrod jayconrod added this to the Backlog milestone Jul 12, 2021
@rsc rsc changed the title [dev.fuzz] testing: tests not discovered unless package path specified or GOMAXPROCS=1 testing: fuzz tests not discovered unless package path specified or GOMAXPROCS=1 Sep 21, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants