Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: verify go test flags with -fuzz #49294

Open
katiehockman opened this issue Nov 2, 2021 · 1 comment
Open

cmd/go: verify go test flags with -fuzz #49294

katiehockman opened this issue Nov 2, 2021 · 1 comment

Comments

@katiehockman
Copy link
Member

@katiehockman katiehockman commented Nov 2, 2021

This is more of a tracking issue, just to have a record of how go test -fuzz behaves when also passed other flags. Let's make sure this is reasonable for the upcoming release for now, and see if we can determine what it would should look like in the long term.

For 1.18, the bare minimum is that it doesn't break anything :)

  • -cover, -covermode, -coverpkg

With -run the coverage is printed as zero. We can probably just document that it isn't supported.

godev test -fuzz Fuzz -run Fuzz -cover
...
PASS
coverage: 0.0% of statements
ok      crypto/dsa      3.551s

Without -run the coverage is shown for the other tests that were run. That's good, but I don't want people to get the impression that this coverage includes what was found during fuzzing. Maybe we should print a warning if -cover is provided with the -fuzz flag to indicate that the coverage % doesn't include anything from fuzzing.

godev test crypto/dsa -fuzz FuzzFoo -cover                     
...
PASS
coverage: 88.3% of statements
ok      crypto/dsa      29.526s
  • -cpu, -failfast, -short, -shuffle, -timeout, -vet

These simply don't affect fuzzing right now, and maybe never should. We should either document that, add a warning to the command line before fuzzing starts, or both.

  • -json, -list, -parallel, -run, -v, -trace benchmarking

Works as intended.

  • -mutexprofile

Not 100% sure if this is working as intended, but it at least isn't break anything.

@bcmills
Copy link
Member

@bcmills bcmills commented Nov 2, 2021

-short seems like it could affect fuzzing, right? (If a fuzz test calls the testing.Short() function, will it panic?)

It seems like -cpu could affect fuzzing too (by changing runtime.GOMAXPROCS(0)).

-failfast and -vet should already work in combination with fuzzing (should be completely orthogonal, IIUC).

-shuffle should probably shuffle the order of test inputs run from the existing corpus.

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants