Repository navigation
cmd/go: allow fuzzing multiple targets per package #46312
Description
Activity
For reference, the
-benchflag can match multiple benchmarks in multiple packages. We should probably do that.I think
go test -bench=. ./...will run benchmarks sequentially to avoid contention. We could either do that with-fuzz(with-fuzztimesaying how long to run), or we could coordinate multiple fuzz targets concurrently.We should decide to either allow one fuzzing target globally across all matched packages on the go test command line, or allow multiple targets per package and fuzz them all in parallel.
My preference would be to allow multiple targets per package to be fuzzed in parallel
Edit (since I think that was unclear): What I'd like to support in the future is running several targets, perhaps even across several packages, in some kind of rotation. So that might look like running target (1) for one second, then (2) for one second, etc etc.I think go test -bench=. ./... will run benchmarks sequentially to avoid contention. We could either do that with -fuzz (with -fuzztime saying how long to run), or we could coordinate multiple fuzz targets concurrently.
I feel like running multiple fuzz targets concurrently is just asking for trouble (e.g. races). Though I guess if each is in it's own process, maybe the risk of that isn't actually that high. We also have to decide what to do if someone wants to fuzz 10 targets at once but only use 3 processes. At that point we'll have to run them sequentially/in a round anyway.
I think it will be simpler and equally efficient to just run each target for some period of time in a round. For example, we run each target for 3 seconds before moving to the next, and so on, until we're back at the first target, then repeat. I don't see a compelling reason why the "time to run each target before moving to the next" needs to be user-configurable, but LMK if you can think of a reason.
Reacted by Jay Conrod and DasSkelettChange https://golang.org/cl/350156 mentions this issue:
[dev.fuzz] cmd/go: in 'go test' don't allow multiple packages with -fuzz- added a commit that references this issue
on Sep 20, 2021 OK that is getting weird now: Are you now supposed to iterate over the list of fuzzable packages yourself?
Useful CI systems usually run go ACTION ./... for any major action the go command offers.
If that is not possible, could you at least provide some way to create the full list of fuzz parallel jobs with the sequential lists of each fuzz function? That together with a max fuzz time would allow CI systems to express the resource budgets itself.
Reacted by DasSkelett and Jacek KrólikowskiAre you now supposed to iterate over the list of fuzzable packages yourself?
For now. This isn't necessarily the long term solution for fuzzing. It's just a safety measure that's in place now until we can build this out effectively.
If that is not possible, could you at least provide some way to create the full list of fuzz parallel jobs with the sequential lists of each fuzz function?
That might be possible. We'll take a look and see if we can provide something like this before 1.18.
An initial idea to get us started: One way to do this would be to write a little tool which matches onFuzzX(*testing.F)functions in *_test.go files, and spits out a script which, if run, would fuzz runs each target in a round. Maybe each target is fuzzed for at least 10 seconds as a default, but this could be configurable. We could do this without any code changes to the fuzzing engine or go command. The concern is that we need to make sure the default is long enough. If, for example, the default is 1 second, it might take approximately that long to start everything up and run the warm-up for initial coverage. And since the go command process would completely shut down before starting the next target, we would need to do this warm-up every time. We could do this outside the release cycle (for folks who want it, this can just go in x/tools or something).Reacted by Ingo Oeser and Calle Helmertz- changed the title
[-][dev.fuzz] cmd/go: allow fuzzing multiple targets per package[/-][+]cmd/go: allow fuzzing multiple targets per package[/+]on Sep 21, 2021 Here's the quick-n-dirty bash solution:
#!/bin/bash set -e fuzzTime=${1:-10} files=$(grep -r --include='**_test.go' --files-with-matches 'func Fuzz' .) for file in ${files} do funcs=$(grep -oP 'func \K(Fuzz\w*)' $file) for func in ${funcs} do echo "Fuzzing $func in $file" parentDir=$(dirname $file) go test $parentDir -run=$func -fuzz=$func -fuzztime=${fuzzTime}s done done
Reacted by Naveen and jordancurveHere's an updated script that works a little more broadly
#!/bin/bash set -e fuzzTime=${1:-10} files=$(grep -r --include='**_test.go' --files-with-matches 'func Fuzz' .) for file in ${files} do funcs=$(grep -o 'func Fuzz\w*' $file | sed 's/func //') for func in ${funcs} do echo "Fuzzing $func in $file" parentDir=$(dirname $file) go test $parentDir -run=$func -fuzz=$func -fuzztime=${fuzzTime}s done doneReacted by Lane RettigHere's the quick-n-dirty bash solution:
#!/bin/bash set -e fuzzTime=${1:-10} files=$(grep -r --include='**_test.go' --files-with-matches 'func Fuzz' .) for file in ${files} do funcs=$(grep -oP 'func \K(Fuzz\w*)' $file) for func in ${funcs} do echo "Fuzzing $func in $file" parentDir=$(dirname $file) go test $parentDir -run=$func -fuzz=$func -fuzztime=${fuzzTime}s done done
@benjivesterby does grep actually support a -P flag? ran this on macos and run into:
grep: invalid option -- P usage: grep [-abcdDEFGHhIiJLlMmnOopqRSsUVvwXxZz] [-A num] [-B num] [-C[num]] [-e pattern] [-f file] [--binary-files=value] [--color=when] [--context[=num]] [--directories=action] [--label] [--line-buffered] [--null] [pattern] [file ...]this is what i use on MacOS
#!/bin/bash fuzzTime=${1:-1h} while true; do files=$(grep -r --include='**_test.go' --files-with-matches 'func Fuzz' .) for file in ${files}; do funcs=$(grep '^func Fuzz' "$file" | sed s/func\ // | sed 's/(.*$//') for func in ${funcs}; do echo "Fuzzing $func in $file" parentDir=$(dirname "$file") go test "$parentDir" -run="$func" -fuzz="$func" -fuzztime="${fuzzTime}" done done doneI made a tool that does this: https://github.com/koonix/gofuzz
forgot about this thread. your tool looks nice, @koonix !
If anyone's looking for something to copy/paste for the
fishshell, here's what I've been using (requires you to also haveag):#! /usr/bin/env fish # Workaround to fuzz all the fuzz tests we can write, roughly stolen from https://github.com/golang/go/issues/46312#issuecomment-1153345129 # Additionally, the use of ag to match a single part of the function was informed by https://github.com/ggreer/the_silver_searcher/issues/400#issuecomment-418454903 set files (ag 'func Fuzz' -G '.*test.go' -l; or true) for file in $files echo Fuzz tests found in $file set funcs (ag -o '^func \KFuzz.*(?=\()' $file) for func in $funcs echo Fuzzing $func go test ./(dirname $file) -fuzz $func -fuzztime 1000x end end
I usually call this file from
mage- marked testing: fuzz tests not discovered unless package path specified or GOMAXPROCS=1 #47136 as a duplicate of this issue
on Apr 19, 2025
Metadata
Metadata
Assignees
Labels
Type
Projects
- Status
Currently,
go testonly allows fuzzing one target per package. So if packageexample.com/ahas targetsFuzz1andFuzz2, this command fails:(Actually, it still exits zero without fuzzing, which is probably bad, too).
If multiple packages have individual fuzz targets, that's okay though:
This seems a little strange. We should decide to either allow one fuzzing target globally across all matched packages on the
go testcommand line, or allow multiple targets per package and fuzz them all in parallel.