Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testing: gotip test -fuzz='^FuzzName$' appears to run regular tests as well #50432

Open
dnwe opened this issue Jan 4, 2022 · 4 comments
Open

testing: gotip test -fuzz='^FuzzName$' appears to run regular tests as well #50432

dnwe opened this issue Jan 4, 2022 · 4 comments

Comments

@dnwe
Copy link

@dnwe dnwe commented Jan 4, 2022

What version of Go are you using (gotip version)?

$ gotip version
go version devel go1.18-95b240b2 Mon Jan 3 23:45:12 2022 +0000 darwin/amd64

Does this issue reproduce with the latest release?

Yes, this was reproduced on latest tip (95b240b) and had also been seen to reproduce on my previously installed version of gotip from Sep 22nd (09d3df0) too

What operating system and processor architecture are you using (go env)?

go env Output
darwin/amd64

What did you do?

  • wrote a simple fuzz_test.go and attempted to run it with gotip test -fuzz=FuzzName as per the docs, but was surprised to find it taking a long time to run
  • added -v to see the output and noted that all of my others tests were being run too

What did you expect to see?

I expected -fuzz=^FuzzName$ to match the docs and spec and behave like -run=^TestName$:

In order to run a fuzz test with the mutation engine, -fuzz will take a regexp which must match only one fuzz test. In this situtation, only the fuzz test will run (ignoring all other tests)

What did you see instead?

gotip test proceeded to execute all of my regular tests in addition to the single fuzzing test that I'd specified. A workaround was to change the cmd to gotip test -fuzz='^FuzzName$' -run='^FuzzName$' and then it behaved as expected, running a single fuzzing test and nothing else.

@thepudds thepudds added the fuzz label Jan 4, 2022
@mvdan
Copy link
Member

@mvdan mvdan commented Jan 4, 2022

Interesting to see that we have a test which specifically locks this behavior in:

# Fuzz successful chatty fuzz target that includes a separate unit test.
go test -v chatty_with_test_fuzz_test.go -fuzz=Fuzz -fuzztime=1x
stdout ok
stdout PASS
! stdout FAIL
stdout -count=1 'all good here'
# Verify that the unit test is only run once.
stdout -count=1 'logged foo'

It seems like it was added in b894834 by @katiehockman. It sounds like a relatively straightforward fix, be it amending the docs, or updating the code to follow the docs. The simplest fix in terms of code might be to have -run default to the value of -fuzz if -fuzz is set but -run is not.

@katiehockman
Copy link
Member

@katiehockman katiehockman commented Jan 4, 2022

@dnwe thanks for filing this. Can you clarify where you saw these docs. If it is only in the design draft, that doc is out of date (see the addendum at the top of the doc), so that doesn't need to be updated.

This is intentional behavior. See #46222

@katiehockman katiehockman added this to the Backlog milestone Jan 4, 2022
@dnwe
Copy link
Author

@dnwe dnwe commented Jan 4, 2022

@katiehockman in terms of docs, it probably was only the draft proposal that made me think the behaviour wasn’t matching the design, or perhaps something I saw in the old go help fuzz output. After re-reading gotip help testflag I can see the current behaviour is specified there:

-fuzz regexp
	    Run the fuzz test matching the regular expression. When specified,
	    the command line argument must match exactly one package within the
	    main module, and regexp must match exactly one fuzz test within
	    that package. Fuzzing will occur after tests, benchmarks, seed corpora
	    of other fuzz tests, and examples have completed. See the Fuzzing
	    section of the testing package documentation for details.

but it would be great if we could cover it on https://go.dev/doc/fuzz/ too and also highlight the difference between gotip test runs that only use the seed corpus, and -fuzz=FuzzXYZ runs that run through mutations to generate fuzz testdata

EDIT: I've also now read through the latest https://pkg.go.dev/testing@master#hdr-Fuzzing which probably should have been my starting point, rather than https://go.dev/doc/fuzz/ as it does explain and even has "The -fuzz and -run flags can both be set, in order to fuzz a target but skip the execution of all other tests"

I guess one of the unexpected parts of the current behaviour is that even if I've already run through all of the tests myself via gotip test ., the testcache gets invalidated when I then subsequently run gotip test -fuzz=FuzzTest . straight after, so then it will start by re-running all of the tests again even though I've just run them and haven't changed any files. So if I'm wanting to perform some fuzztime for all of my fuzz tests (gotip test -list . | grep '^Fuzz' | parallel -j1 'gotip test -v -run=^{}$ -fuzz=^{}$ -fuzztime=10m') the regular tests, examples, benchmarks would get re-run for each one.

It's a shame we can't preserve the testcache ignoring -fuzz= parameters.

@katiehockman
Copy link
Member

@katiehockman katiehockman commented Jan 4, 2022

That's an interesting idea. I can see your point about that fact that running go test with -fuzz could potentially use some of the cached test data from previous runs of go test. @bcmills do you have opinions on the feasibility of this? I would need to think about it some more.

One possible issue is that the binary isn't exactly the same, because running go test -fuzz instruments the testing binary in a way that's different than typical runs of go test. That instrumentation should be irrelevant for the unit tests, but it's there nonetheless.

/cc @golang/fuzzing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants