-
Notifications
You must be signed in to change notification settings - Fork 17.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
proposal: testing: custom mutator support for fuzzing #48815
Comments
cc @golang/fuzzing |
A friendly ping: any update? |
This is unlikely to be implemented for 1.18, but might be considered for 1.19. |
Thanks for your reply, I will maintain an implementation in my local branch(https://github.com/s3nt3/go/tree/dev.fuzz.custom_mutator). Looking forward to supporting this feature in the new development cycle. |
The interface should probably satisfy type CustomMutator interface {
Mutate() error
MarshalText() ([]byte, error)
UnmarshalText([]byte) error
} |
I believe Go 1.20 development cycle is currently open, and implementation in https://github.com/s3nt3/go/tree/dev.fuzz.custom_mutator looks like a very compact change. Any chance of this being included in 1.20? I could definitely use this capability in some of my projects :) |
Additional background: Structure-Aware Fuzzing with libFuzzer |
Users might be fuzzing binary data such as images or protobufs. Wouldn't binary marshaler/unmarshaler be a better choice? type CustomMutator interface {
encoding.BinaryMarshaler
encoding.BinaryUnmarshaler
Mutate() error
} Also, would it be useful to supply an type CustomMutator interface {
encoding.BinaryMarshaler
encoding.BinaryUnmarshaler
Mutate(seed int64) error
} |
@rhansen the corpus data is stored in a text format |
True, but if I understand correctly, that's an implementation detail. (At least I couldn't find an official specification of the |
This commit extends F.Fuzz to support fuzz functions with custom input parameter types. Custom input types must implement the MarshalBinary, UnmarshalBinary, and Mutate methods. This enables structure-aware fuzzing; see [1] for background. Custom values are encoded to disk like this: go test fuzz v1 custom("*example.com/mod/pkg.Typename", "bytes go here") [1] https://github.com/google/fuzzing/blob/master/docs/structure-aware-fuzzing.md Fixes golang#48815
This commit extends F.Fuzz to support fuzz functions with custom input parameter types. Custom input types must implement the MarshalBinary, UnmarshalBinary, and Mutate methods. This enables structure-aware fuzzing; see [1] for background. Custom values are encoded to disk like this: go test fuzz v1 custom("*example.com/mod/pkg.Typename", "bytes go here") [1] https://github.com/google/fuzzing/blob/master/docs/structure-aware-fuzzing.md Fixes golang#48815 Change-Id: Ib5c8dfadd2df9456b8a82ceec7d8e289a4c7c3de
Change https://go.dev/cl/493304 mentions this issue: |
This commit extends F.Fuzz to support fuzz functions with custom input parameter types. Custom input types must implement the MarshalBinary, UnmarshalBinary, and Mutate methods. This enables structure-aware fuzzing; see [1] for background. Custom values are encoded to disk like this: go test fuzz v1 custom("*example.com/mod/pkg.Typename", "bytes go here") [1] https://github.com/google/fuzzing/blob/master/docs/structure-aware-fuzzing.md Fixes golang#48815 Change-Id: Ib5c8dfadd2df9456b8a82ceec7d8e289a4c7c3de
Change https://go.dev/cl/493637 mentions this issue: |
I started a design doc. Feedback would be appreciated. |
How should fuzzing of complex, nested data types work where you don't control the implementation of its methods but nevertheless want/need to use them as data types in your types? |
@nightlyone As long as it is possible to marshal/unmarshal the type to/from binary (such as JSON or protobuf), you should be able to fuzz it. For example, if the type you want to fuzz is package foo_test
import (
"context"
"testing"
"me.example/mod/foo"
)
type fuzzInput struct {
foo.Foo
}
func (v *fuzzInput) MarshalBinary() ([]byte, error) { return v.MarshalJSON() }
func (v *fuzzInput) UnmarshalBinary(d []byte) error { return v.UnmarshalJSON(d) }
func (v *fuzzInput) Mutate(ctx context.Context, seed int64) error {
// mutate v.Foo as desired
return nil
}
func FuzzFoo(f *testing.F) {
// Assumption: the zero value of foo.Foo is valid. If that is not the
// case, call f.Add or create a `testdata/fuzz/FuzzFoo/*` seed file
// so that f.Fuzz has a valid value to start with.
f.Fuzz(func(t *testing.T, v *fuzzInput) {
if err := foo.FunctionYouWantToFuzz(&v.Foo); err != nil {
t.Fatal(err)
}
})
} If the third-party type is not directly marshalable, you could instead marshal a machine-readable sequence of instructions that recreate the desired state. (For example, marshal a protobuf message that lists function calls and their arguments. These calls would be replayed when unmarshaling to reconstruct the state from scratch.) The |
For golang/go#48815. Change-Id: I021e4517940ff073254d9d56fcca623f4e2ed460 Reviewed-on: https://go-review.googlesource.com/c/proposal/+/493637 Reviewed-by: Ian Lance Taylor <iant@golang.org>
Open issues in the design doc:
|
Personally I think it's clearly useful for a |
Change https://go.dev/cl/501537 mentions this issue: |
After thinking on this for a while, I think the two best options for
The design doc currently proposes Anyone else have any opinions? |
Change https://go.dev/cl/493302 mentions this issue: |
* Add rationale explaining why the `customMutator` interface is not exported. * Switch `Mutate` to take a `context.Context`, and expand the associated rationale. * Minor readability tweaks. * Delete the open issues. (They don't describe problems that lack a solution; they are more like discussion points that should be addressed in the linked issue or in merge requests.) For golang/go#48815. Change-Id: I704ab930710d1b82a42b7923dbabc4e10543f5ca Reviewed-on: https://go-review.googlesource.com/c/proposal/+/501537 Reviewed-by: Ian Lance Taylor <iant@google.com>
Hi! Would it make sense to pass a My motivation is that I only want to custom mutate a few fields in a struct for example, and leave the rest to the default mutator. |
As the official fuzzer implementation provided by golang, the native fuzzer should be well suited for various usage scenarios. However, currently native fuzzers only support general mutation algorithms for built-in types. Therefore, in many cases, the native fuzzer cannot efficiently generate test inputs. For example, when testing a DSL parser, the mutator will generate a large amount of output that cannot pass the syntax or semantic check. A possible solution is to provide support for custom mutators, so that users can implement custom mutators for various fuzz targets and reuse other parts of the native fuzzer.
I tried on the existing code and designed the following interface:
The object that implements the above interface can be passed as an argument to the
testing.F.Fuzz
method, and it will call theMutate
method to use the custom mutation algorithm. TheMarshal
andUnmarshal
methods ensure that it can be imported/exported to a corpus file.I think supporting custom mutator will bring the following benefits:
In addition, custom mutator will also bring some side effects, such as the custom mutator code will be instrumented, which may affect performance and the accuracy of coverage statistics.
/cc @jayconrod
The text was updated successfully, but these errors were encountered: