Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[dev.fuzz] testing: crash due to incompatible type #45593

Open
dsnet opened this issue Apr 15, 2021 · 6 comments
Open

[dev.fuzz] testing: crash due to incompatible type #45593

dsnet opened this issue Apr 15, 2021 · 6 comments

Comments

@dsnet
Copy link
Member

@dsnet dsnet commented Apr 15, 2021

Using 529e5d0.

Consider this snippet:

func FuzzFoo(f *testing.F) {
	for _, td := range myTestdata {
		f.Add(0, []byte(td))
	}

	f.Fuzz(func(t *testing.T, seed int64, b []byte) {
		...
	})
}

Currently, this crashes with:

panic: reflect: Call using int as type int64 [recovered]
	panic: reflect: Call using int as type int64

goroutine 54 [running]:
testing.tRunner.func1.2(0x5fd600, 0xc00058e220)
	testing/testing.go:1153 +0x34b
testing.tRunner.func1(0xc0000f8300)
	testing/testing.go:1156 +0x4b6
panic(0x5fd600, 0xc00058e220)
	runtime/panic.go:972 +0x1d4
reflect.Value.call(0x603a20, 0x653690, 0x13, 0x64222a, 0x4, 0xc001c9c240, 0x3, 0x4, 0x4, 0x30, ...)
	reflect/value.go:409 +0x1917
reflect.Value.Call(0x603a20, 0x653690, 0x13, 0xc001c9c240, 0x3, 0x4, 0x2, 0x4, 0x786c00)
	reflect/value.go:338 +0xb9
testing.(*F).Fuzz.func1.1(0xc0000f8300)
	testing/fuzz.go:343 +0x225
testing.tRunner(0xc0000f8300, 0xc001c9c1e0)
	testing/testing.go:1203 +0xef
created by testing.(*F).Fuzz.func1
	testing/fuzz.go:338 +0x370

The cause of the crash is because testing.F.Add is called with an int, while the Fuzz function itself is expecting an int64.

Either, the testing framework should:

  • report an with a more clearer message about what's wrong, or
  • automatically convert assignable values.

Also, with generics on the horizon, is there an API design that would guarantee type safety between testing.F.Add and testing.F.Fuzz? It's probably better to statically prevent type errors like than to have it occur at runtime.

\cc @katiehockman @jayconrod

@katiehockman
Copy link
Member

@katiehockman katiehockman commented Apr 15, 2021

Ah thanks for testing and reporting this! There's probably something better we can do here, like you said. So I'll think about that more. At the very least we can improve the error messaging to say something more direct.

It would be great to be able to use generics at some point soon. It would likely help for cases like this. If we can't, then at the very least we can add go vet checks that can verify this at build time so it isn't caught at runtime.

@dsnet
Copy link
Member Author

@dsnet dsnet commented Apr 15, 2021

It would be great to be able to use generics at some point soon. It would likely help for cases like this.

Although, I'm not sure if it's going to be possible with the currently accepted generics proposal since it doesn't support variadic type parameters. A go vet check might be necessary.

@ianlancetaylor @griesemer

@mknyszek mknyszek added this to the Backlog milestone Apr 15, 2021
@mknyszek
Copy link
Contributor

@mknyszek mknyszek commented Apr 15, 2021

I put this in backlog for now, but is it a release blocker?

@dsnet dsnet added the fuzz label Apr 16, 2021
@katiehockman katiehockman modified the milestones: Backlog, Unreleased Jun 4, 2021
@katiehockman
Copy link
Member

@katiehockman katiehockman commented Jun 4, 2021

One tricky thing to consider is that we have to decide what our source of truth is if the arguments differ.

For example, consider these two fuzz targets:
a)

func FuzzFoo(f *testing.F) {
	f.Add(int32(0))
	f.Fuzz(func(t *testing.T, i int64) { ... })
}

b)

func FuzzFoo(f *testing.F) {
	f.Add(int64(0))
	f.Fuzz(func(t *testing.T, i int32) { ... })
}

Should we pick int64 values to fuzz and write to the corpus, since that can hold the most bytes, and is thus the most likely to be able to assign to? Or should we assume that the arguments in the f.Fuzz function are the source of truth, and if whatever is passed via f.Add or in testdata can't be cast to that type, then we fail the test? I think I'd prefer the latter.

Do you have an opinion on this?

@dsnet
Copy link
Member Author

@dsnet dsnet commented Jun 4, 2021

Or should we assume that the arguments in the f.Fuzz function are the source of truth, and if whatever is passed via f.Add or in testdata can't be cast to that type, then we fail the test? I think I'd prefer the latter.

I believe f.Fuzz should be authoritative with the rationale that f.Fuzz is given explicit types in the arguments list. On the other hand, the types given to f.Add are inferred by the values given.

@katiehockman katiehockman self-assigned this Jun 7, 2021
@gopherbot
Copy link

@gopherbot gopherbot commented Jun 7, 2021

Change https://golang.org/cl/325702 mentions this issue: [dev.fuzz] testing: convert seed corpus values where possible

gopherbot pushed a commit that referenced this issue Jun 15, 2021
The types provided in f.Fuzz will be viewed as the
canonical types for fuzzing. If the type is different
for a seed corpus entry, then the testing package
will attempt to convert it. If it can't convert it,
f.Fuzz will fail.

Currently, this allows converting types that may result
in precision loss or a semantically different value.
For example, an int(-1) can be converted to uint even
though the value could be math.MaxUint64. There is a
TODO to consider improving this in the future.

Updates #45593

Change-Id: I2e752119662f46b68445d42b1ffa46dd30e9faea
Reviewed-on: https://go-review.googlesource.com/c/go/+/325702
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Roland Shoemaker <roland@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants