internal/fuzz,testing: test_fuzz_mutator_repeat failures with "inputs are not equal" #49047
Comments
bcmills
added
NeedsInvestigation
|
The inputs in these failures look very similar to me. (Maybe this has something to do with the Unicode replacement value, if the test input happens not to be valid UTF-8?) |
|
Change https://golang.org/cl/356649 mentions this issue: |
(Temporarily, until they can be fixed.) For #49046 For #49047 Change-Id: Ib580a5e45a0955aabdfc1899ed38a262a37f66ab Reviewed-on: https://go-review.googlesource.com/c/go/+/356649 Trust: Bryan C. Mills <bcmills@google.com> Run-TryBot: Bryan C. Mills <bcmills@google.com> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Roland Shoemaker <roland@golang.org> Reviewed-by: Katie Hockman <katie@golang.org>
|
I don't think this is a UTF-8 issue. What this test is basically checking is that the coordinator can reconstruct the mutated input if it wasn't provided by the worker (e.g. because of a non-recoverable crash). It reconstructs it by using the count of mutations that is stored in shared memory. If that number is 1 off (either too big or too small), it would look a lot like these errors, since all of the incorrect inputs are only slightly different from the expected inputs. I'm not able to reproduce this locally, unfortunately, which makes debugging this more difficult. |
|
I have a theory. It's a bit of a longshot, but seems plausible? When fuzzing, if an unrecoverable error occurs, the coordinator uses the count written to memory to reconstruct the input that caused the failure. This count is increased inside fuzzOnce right before we execute fuzzFn (the fuzz function). I see one possible way this number could be off. If the 99th execution of fuzzFn increases coverage, and a crash occurs during the de-flake (i.e. the 100th execution), then the count will accurately describe how many times the fuzz function was called, but will be 1 bigger than the number of times mutate was called. This would explain why it happens so rarely. The fix for this isn't totally straightforward. We can't for example just not increase the count when we're deflaking, because the count is used for |
|
Change https://golang.org/cl/358094 mentions this issue: |
|
Unfortunately this test is still occasionally failing on
2021-11-17T17:36:36-0555ea3/linux-386-longtest |
|
From some further builds, the recent failures are looking like a regression as of CL 364214. |
|
Change https://golang.org/cl/364758 mentions this issue: |
For #49047 Change-Id: If06ce033f7bfd23d640311f1be261afab8332028 Reviewed-on: https://go-review.googlesource.com/c/go/+/364758 Trust: Roland Shoemaker <roland@golang.org> Run-TryBot: Roland Shoemaker <roland@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com> TryBot-Result: Go Bot <gobot@golang.org>
|
Change https://golang.org/cl/364954 mentions this issue: |
|
Change https://golang.org/cl/365175 mentions this issue: |
|
Change https://golang.org/cl/365294 mentions this issue: |
|
@rolandshoemaker @katiehockman Is this okay to resolve after beta1? If so, feel free to apply the label for tracking. |
When swapping two chunks of bytes in a slice, don't pick chunks which extend beyond the end of the slice. Also don't pick chunks which intersect with each other. Fixes #49047 Change-Id: I070eb1888d05ae849ec6122d01c40c45e602019f Reviewed-on: https://go-review.googlesource.com/c/go/+/365175 Trust: Roland Shoemaker <roland@golang.org> Trust: Bryan C. Mills <bcmills@google.com> Trust: Katie Hockman <katie@golang.org> Run-TryBot: Roland Shoemaker <roland@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Bryan C. Mills <bcmills@google.com> Reviewed-by: Katie Hockman <katie@golang.org>
greplogs --dashboard -md -l -e '(?ms)FAIL: TestScript/test_fuzz_mutator_repeat.*inputs are not equal'2021-10-13T16:36:59-69041c7/windows-amd64-longtest
2021-10-08T00:18:29-7cef831/linux-amd64-longtest
2021-10-05T21:25:51-6ae3afa/windows-amd64-longtest
CC @katiehockman @rolandshoemaker
The text was updated successfully, but these errors were encountered: