testing: minimized crasher is unrelated to original #48326
Comments
thanm
added
fuzz
rsc
changed the title
|
Checking in on this issue as it's labeled a release blocker for Go 1.18. Is there any update? |
|
Not yet. There are a lot of fuzzing issues. We'll get to them as soon as we can. |
|
I'm going to remove the release-blocker label for this issue. I think it's a nice to have, so I'll leave it for Go 1.18, but not strictly necessary for the release. We don't currently support any detection around what kind of crash occurred in order to replicate it during minimization. This is something that we'll need down the line: namely for deduplication of crashing so that we can run in a |
|
@katiehockman I think the problem here is that minimization is considered successful if the new value has at least one coverage bit in common with the original. This can lead to two inputs with little in common e.g for this code if len(input) >= 1 {
if input[0] == 'A' {
foo(input);
} else {
bar(input);
}
}
I think a minimized input should never result in fewer coverage bits than we had for the original input. I experimented with enforcing this and the following patch appears to fix the bug: diff --git a/src/internal/fuzz/coverage.go b/src/internal/fuzz/coverage.go
index 3dee73b81c..88f98a16b2 100644
--- a/src/internal/fuzz/coverage.go
+++ b/src/internal/fuzz/coverage.go
@@ -66,6 +66,17 @@ func countNewCoverageBits(base, snapshot []byte) int {
return n
}
+// isCoverageSubset returns true if all the base coverage bits are set in
+// snapshot
+func isCoverageSubset(base, snapshot []byte) bool {
+ for i, v := range base {
+ if v&snapshot[i] != v {
+ return false
+ }
+ }
+ return true
+}
+
// hasCoverageBit returns true if snapshot has at least one bit set that is
// also set in base.
func hasCoverageBit(base, snapshot []byte) bool {
diff --git a/src/internal/fuzz/worker.go b/src/internal/fuzz/worker.go
index e7d824bea1..b80e98bde7 100644
--- a/src/internal/fuzz/worker.go
+++ b/src/internal/fuzz/worker.go
@@ -908,7 +908,7 @@ func (ws *workerServer) minimizeInput(ctx context.Context, vals []interface{}, c
}
return true
}
- if keepCoverage != nil && hasCoverageBit(keepCoverage, coverageSnapshot) {
+ if keepCoverage != nil && isCoverageSubset(keepCoverage, coverageSnapshot) {
return true
}Given recent improvements in the effectiveness of the mutator by @rolandshoemaker I've been able to see that this problem with minimization actually works against the mutator. For example, this problem renders the mutator ineffective for examples which libfuzzer etc deal with easily just using coverage feedback e.g. the fuzzer can't find an input which terminates the following var leet = []byte{1, 3, 3, 7}
func magic(input []byte) bool {
if len(input) >= 4 && input[0] == leet[0] && input[1] == leet[1] && input[2] == leet[2] && input[3] == leet[3] {
input = input[4:]
return len(input) >= 4 && input[0] == leet[0] && input[1] == leet[1] && input[2] == leet[2] && input[3] == leet[3]
}
return false
}
func Fuzz(f *testing.F) {
f.Fuzz(func(t *testing.T, data []byte) {
if magic(data) {
t.Fatalf("magic is %v", data)
}
})
}With the above patch applied, the fuzzer finds a crasher right away. My interpretation is that the mutator is working to increase coverage but minimization can reduce coverage. |
|
Thanks @stevenjohnstone for looking into this so deeply. One of us should be able to take a look at this in the next couple weeks as well, and hopefully can improve this before 1.18. If not, it'll happen for 1.19. |
|
Change https://go.dev/cl/391454 mentions this issue: |
|
Change https://go.dev/cl/391614 mentions this issue: |
When minimizing a value, if the value cannot be minimized (i.e. it is the final value is the same value as was sent for minimization) return the initial coverage map, rather than the coverageSnapshot, which is actually the coverage map for the final minimization step and may not accurately reflect whether the input actually expands the coverage set or not. Updates #48326 Change-Id: I01f0eebe5841e808b6799647d2e5fe3aa45cd2e0 Reviewed-on: https://go-review.googlesource.com/c/go/+/391614 Reviewed-by: Bryan Mills <bcmills@google.com> Trust: Roland Shoemaker <roland@golang.org> Run-TryBot: Roland Shoemaker <roland@golang.org> Auto-Submit: Roland Shoemaker <roland@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org>
Minimization should result in a fuzz input which includes the same coverage bits as the original input. Updates #48326 Change-Id: I6c5f30058b57ccd1a096ad0e9452a4dfbb7d9aab Reviewed-on: https://go-review.googlesource.com/c/go/+/391454 Trust: Bryan Mills <bcmills@google.com> Reviewed-by: Roland Shoemaker <roland@golang.org> Run-TryBot: Roland Shoemaker <roland@golang.org> Auto-Submit: Roland Shoemaker <roland@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org>
|
Change https://go.dev/cl/391797 mentions this issue: |
|
Change https://go.dev/cl/391798 mentions this issue: |
…during minimization When minimizing a value, if the value cannot be minimized (i.e. it is the final value is the same value as was sent for minimization) return the initial coverage map, rather than the coverageSnapshot, which is actually the coverage map for the final minimization step and may not accurately reflect whether the input actually expands the coverage set or not. Updates #48326 Change-Id: I01f0eebe5841e808b6799647d2e5fe3aa45cd2e0 Reviewed-on: https://go-review.googlesource.com/c/go/+/391614 Reviewed-by: Bryan Mills <bcmills@google.com> Trust: Roland Shoemaker <roland@golang.org> Run-TryBot: Roland Shoemaker <roland@golang.org> Auto-Submit: Roland Shoemaker <roland@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> (cherry picked from commit bd71dee) Reviewed-on: https://go-review.googlesource.com/c/go/+/391797 Trust: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
… coverage Minimization should result in a fuzz input which includes the same coverage bits as the original input. Updates #48326 Change-Id: I6c5f30058b57ccd1a096ad0e9452a4dfbb7d9aab Reviewed-on: https://go-review.googlesource.com/c/go/+/391454 Trust: Bryan Mills <bcmills@google.com> Reviewed-by: Roland Shoemaker <roland@golang.org> Run-TryBot: Roland Shoemaker <roland@golang.org> Auto-Submit: Roland Shoemaker <roland@golang.org> TryBot-Result: Gopher Robot <gobot@golang.org> (cherry picked from commit 5003ed8) Reviewed-on: https://go-review.googlesource.com/c/go/+/391798 Trust: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org> Reviewed-by: Bryan Mills <bcmills@google.com>
|
CC @golang/fuzzing Looks like this didn't make 1.19. Moving to backlog. Please recategorize as appropriate. Thanks. |
What version of Go are you using (
go version)?Does this issue reproduce with the latest release?
n/a
What operating system and processor architecture are you using (
go env)?go envOutputWhat did you do?
Here's the simplest case I can find which reproduces the problem:
The test is rigged so it'll crash in
branch1. Minimization kicks in and it ends up crashing inbranch2:Instrumenting with this bpftrace script
I see that all the worker processes run
branch1first:The tail of the log shows what looks like the negative value
-20123being cast to a largeuintand then divided by 10 repeatedly to reach the value 18446 in the corpus. This may be another issue if the intention was to preserve the sign of integers on minimization 🤷What did you expect to see?
I expected to see the original crash recorded. The first and second crashes have different coverage and different panic messages/backtraces. There's enough information to label these as different crashes.
In a more complicated example, reducing one type of crash to another and then discarding the original wastes a lot of work. The fuzzer would need to be run again to find the original crash. It'd be really useful if the fuzzer retained the original crasher and provided a minimized version as a feature.
What did you see instead?
The original crasher is lost.
The text was updated successfully, but these errors were encountered: