Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: allow fuzzing multiple targets per package #46312

Open
jayconrod opened this issue May 21, 2021 · 5 comments
Open

cmd/go: allow fuzzing multiple targets per package #46312

jayconrod opened this issue May 21, 2021 · 5 comments

Comments

@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented May 21, 2021

Currently, go test only allows fuzzing one target per package. So if package example.com/a has targets Fuzz1 and Fuzz2, this command fails:

$ go test -fuzz=. example.com/a
ok  	example.com/a	0.103s [will not fuzz, -fuzz matches more than one target]

(Actually, it still exits zero without fuzzing, which is probably bad, too).

If multiple packages have individual fuzz targets, that's okay though:

$ go test -fuzz=. example.com/a example.com/b

This seems a little strange. We should decide to either allow one fuzzing target globally across all matched packages on the go test command line, or allow multiple targets per package and fuzz them all in parallel.

@jayconrod
Copy link
Contributor Author

@jayconrod jayconrod commented May 21, 2021

For reference, the -bench flag can match multiple benchmarks in multiple packages. We should probably do that.

I think go test -bench=. ./... will run benchmarks sequentially to avoid contention. We could either do that with -fuzz (with -fuzztime saying how long to run), or we could coordinate multiple fuzz targets concurrently.

@katiehockman
Copy link
Member

@katiehockman katiehockman commented Jun 29, 2021

We should decide to either allow one fuzzing target globally across all matched packages on the go test command line, or allow multiple targets per package and fuzz them all in parallel.

My preference would be to allow multiple targets per package to be fuzzed in parallel
Edit (since I think that was unclear): What I'd like to support in the future is running several targets, perhaps even across several packages, in some kind of rotation. So that might look like running target (1) for one second, then (2) for one second, etc etc.

I think go test -bench=. ./... will run benchmarks sequentially to avoid contention. We could either do that with -fuzz (with -fuzztime saying how long to run), or we could coordinate multiple fuzz targets concurrently.

I feel like running multiple fuzz targets concurrently is just asking for trouble (e.g. races). Though I guess if each is in it's own process, maybe the risk of that isn't actually that high. We also have to decide what to do if someone wants to fuzz 10 targets at once but only use 3 processes. At that point we'll have to run them sequentially/in a round anyway.

I think it will be simpler and equally efficient to just run each target for some period of time in a round. For example, we run each target for 3 seconds before moving to the next, and so on, until we're back at the first target, then repeat. I don't see a compelling reason why the "time to run each target before moving to the next" needs to be user-configurable, but LMK if you can think of a reason.

@gopherbot
Copy link

@gopherbot gopherbot commented Sep 15, 2021

Change https://golang.org/cl/350156 mentions this issue: [dev.fuzz] cmd/go: in 'go test' don't allow multiple packages with -fuzz

gopherbot pushed a commit that referenced this issue Sep 20, 2021
Until we have a system for managing load across multiple fuzz targets
in multiple test executables, we'll only support fuzzing one target in
one package at a time. Users can still run multiple 'go test -fuzz'
commands concurrently, but this may overwhelm some systems unless
-parallel and -p are set carefully.

For #46312

Change-Id: If84c58d1b3e60498ce955eae5ad4d52100dbd4b7
Reviewed-on: https://go-review.googlesource.com/c/go/+/350156
Trust: Jay Conrod <jayconrod@google.com>
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Katie Hockman <katie@golang.org>
@nightlyone
Copy link
Contributor

@nightlyone nightlyone commented Sep 20, 2021

OK that is getting weird now: Are you now supposed to iterate over the list of fuzzable packages yourself?

Useful CI systems usually run go ACTION ./... for any major action the go command offers.

If that is not possible, could you at least provide some way to create the full list of fuzz parallel jobs with the sequential lists of each fuzz function? That together with a max fuzz time would allow CI systems to express the resource budgets itself.

@katiehockman
Copy link
Member

@katiehockman katiehockman commented Sep 20, 2021

@nightlyone

Are you now supposed to iterate over the list of fuzzable packages yourself?

For now. This isn't necessarily the long term solution for fuzzing. It's just a safety measure that's in place now until we can build this out effectively.

If that is not possible, could you at least provide some way to create the full list of fuzz parallel jobs with the sequential lists of each fuzz function?

That might be possible. We'll take a look and see if we can provide something like this before 1.18.
An initial idea to get us started: One way to do this would be to write a little tool which matches on FuzzX(*testing.F) functions in *_test.go files, and spits out a script which, if run, would fuzz runs each target in a round. Maybe each target is fuzzed for at least 10 seconds as a default, but this could be configurable. We could do this without any code changes to the fuzzing engine or go command. The concern is that we need to make sure the default is long enough. If, for example, the default is 1 second, it might take approximately that long to start everything up and run the warm-up for initial coverage. And since the go command process would completely shut down before starting the next target, we would need to do this warm-up every time. We could do this outside the release cycle (for folks who want it, this can just go in x/tools or something).

@rsc rsc changed the title [dev.fuzz] cmd/go: allow fuzzing multiple targets per package cmd/go: allow fuzzing multiple targets per package Sep 21, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants