Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: verify go test flags with -fuzz #49294

Open
katiehockman opened this issue Nov 2, 2021 · 1 comment
Open

cmd/go: verify go test flags with -fuzz #49294

katiehockman opened this issue Nov 2, 2021 · 1 comment
Labels
FeatureRequest fuzz Issues related to native fuzzing support NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made.
Milestone

Comments

@katiehockman
Copy link
Contributor

katiehockman commented Nov 2, 2021

This is more of a tracking issue, just to have a record of how go test -fuzz behaves when also passed other flags. Let's make sure this is reasonable for the upcoming release for now, and see if we can determine what it would should look like in the long term.

For 1.18, the bare minimum is that it doesn't break anything :)

  • -cover, -covermode, -coverpkg

With -run the coverage is printed as zero. We can probably just document that it isn't supported.

godev test -fuzz Fuzz -run Fuzz -cover
...
PASS
coverage: 0.0% of statements
ok      crypto/dsa      3.551s

Without -run the coverage is shown for the other tests that were run. That's good, but I don't want people to get the impression that this coverage includes what was found during fuzzing. Maybe we should print a warning if -cover is provided with the -fuzz flag to indicate that the coverage % doesn't include anything from fuzzing.

godev test crypto/dsa -fuzz FuzzFoo -cover                     
...
PASS
coverage: 88.3% of statements
ok      crypto/dsa      29.526s
  • -cpu, -failfast, -short, -shuffle, -timeout, -vet

These simply don't affect fuzzing right now, and maybe never should. We should either document that, add a warning to the command line before fuzzing starts, or both.

  • -json, -list, -parallel, -run, -v, -trace benchmarking

Works as intended.

  • -mutexprofile

Not 100% sure if this is working as intended, but it at least isn't break anything.

@katiehockman katiehockman added NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. FeatureRequest fuzz Issues related to native fuzzing support labels Nov 2, 2021
@katiehockman katiehockman added this to the Backlog milestone Nov 2, 2021
@bcmills
Copy link
Member

bcmills commented Nov 2, 2021

-short seems like it could affect fuzzing, right? (If a fuzz test calls the testing.Short() function, will it panic?)

It seems like -cpu could affect fuzzing too (by changing runtime.GOMAXPROCS(0)).

-failfast and -vet should already work in combination with fuzzing (should be completely orthogonal, IIUC).

-shuffle should probably shuffle the order of test inputs run from the existing corpus.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
FeatureRequest fuzz Issues related to native fuzzing support NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made.
Projects
Status: No status
Development

No branches or pull requests

2 participants