Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

build.sh example for go test -fuzz #102

Closed
hickford opened this issue Jul 22, 2022 · 16 comments
Closed

build.sh example for go test -fuzz #102

hickford opened this issue Jul 22, 2022 · 16 comments

Comments

@hickford
Copy link

hickford commented Jul 22, 2022 •
edited

I write fuzz tests in Go 1.18 following https://go.dev/doc/fuzz/ and run them with a command such as go test -fuzz=Range. How can I integrate them with ClusterFuzzLite?

I ran python3 infra/helper.py generate --external --language=go $PATH_TO_PROJECT but now I'm stuck. What do I put in build.sh? It would be neat to have an example in the documentation. The current example looks C++ specific https://google.github.io/clusterfuzzlite/build-integration/#buildsh
@hickford
Copy link
Author

Ah reading https://google.github.io/clusterfuzzlite/build-integration/#prerequisites it looks like ClusterFuzzLite wants some other kind of fuzz target

ClusterFuzzLite supports libFuzzer targets built with Clang on Linux.

@hickford
Copy link
Author

hickford commented Jul 22, 2022 •
edited

Perhaps I can follow https://google.github.io/oss-fuzz/getting-started/new-project-guide/go-lang/#native-go-fuzzing-support though ? It mentions compile_native_go_fuzzer

@jonathanmetzman
Copy link
Collaborator

ClusterFuzzLite definitely supports non-clang targets. I will change this, this is talking about C/C++ code.
ClusterFuzzLite does support go targets, but it supports go targets fuzzed with libFuzzer, which means I don't think we support go 1.18 fuzzing. We share a lot of the infra with oss-fuzz, and here is the unsubmitted PR: google/oss-fuzz#7519

tl;dr no unless we can fuzz this target with libfuzzer.

@jonathanmetzman
Copy link
Collaborator

@DavidKorczynski just confirming, oss-fuzz doesn't support go 1.18 fuzzing right?

@DavidKorczynski
Copy link

DavidKorczynski commented Jul 22, 2022 •
edited

tldr: yes it does and I think it should work here.

@AdamKorcz can you clarify details here?

My personal summary, although Adam knows a lot more about this: OSS-Fuzz supports fuzzing of targets written syntactically for go 1.18 native. These targets, however, are compiled such that they run by way of libFuzzer. A consequence of this is the corpus used for native fuzzing and the OSS-Fuzz style will be incompatible. i.e. you cannot upload your own corpus and reproduction of issues should happen by way of OSS-Fuzz. But besides that, OSS-Fuzz is fully compatible with go 1.18 targets and already larger projects are using it, e.g. Istio (google/oss-fuzz#7992).

@DavidKorczynski
Copy link

@AdamKorcz could you perhaps make a sample go 1.18 ClusterFuzzLite integration if this would help?

@AdamKorcz
Copy link

Go 1.18 fuzz targets can run on OSS-Fuzz using compile_native_go_fuzzer which changes &testing.F{} to run with libFuzzer. As such, in the process of building Go 1.18 fuzzers on OSS-Fuzz they are made into libFuzzer binaries. All features in libFuzzer will subsequently be available whereas not all features in Go 1.18 fuzzing will.

@DavidKorczynski
Copy link

Perhaps I can follow https://google.github.io/oss-fuzz/getting-started/new-project-guide/go-lang/#native-go-fuzzing-support though ? It mentions compile_native_go_fuzzer

This should work.

@DavidKorczynski
Copy link

tl;dr no unless we can fuzz this target with libfuzzer.

And we should be able to do that with all go 1.18 fuzzers as long as they are compiled by way of compile_native_go_fuzzer (https://github.com/google/oss-fuzz/blob/master/infra/base-images/base-builder/compile_native_go_fuzzer)

@hickford
Copy link
Author

hickford commented Jul 22, 2022 •
edited

Thanks I now have ClusterFuzzLite running in GitHub Actions hickford/merkle#1

my setup https://github.com/hickford/merkle/blob/main/.clusterfuzzlite/Dockerfile and https://github.com/hickford/merkle/blob/main/.clusterfuzzlite/build.sh

However when I tested it by deliberately breaking the test, the workflow succeeded regardless. Any ideas?

2022-07-22 17:01:27,849 - root - INFO - Running fuzzer: FuzzRangeNodes.
[20](https://github.com/hickford/merkle/runs/7472433593?check_suite_focus=true#step:5:21)22-07-22 17:01:27,849 - root - INFO - Downloading corpus for FuzzRangeNodes to /github/workspace/cifuzz-corpus/FuzzRangeNodes.
2022-07-22 17:01:28,056 - root - WARNING - Could not find artifact: cifuzz-corpus-FuzzRangeNodes.
2022-07-22 17:01:28,056 - root - WARNING - Could not download artifact: cifuzz-corpus-FuzzRangeNodes.
2022-07-22 17:01:28,057 - root - INFO - Done downloading corpus. Contains 0 elements.
2022-07-22 17:01:28,057 - root - INFO - Starting fuzzing
INFO: Seed: 1337
INFO: A corpus is not provided, starting from an empty corpus
#2	INITED ft: 43 corp: 1/1b exec/s: 0 rss: 45Mb
#4	NEW    ft: 47 corp: 2/3b lim: 4096 exec/s: 0 rss: 45Mb L: 2/2 MS: 2 ChangeBit-InsertByte-
#6	NEW    ft: 435 corp: 3/124b lim: 4096 exec/s: 0 rss: 45Mb L: 1[21](https://github.com/hickford/merkle/runs/7472433593?check_suite_focus=true#step:5:22)/121 MS: 2 CopyPart-InsertRepeatedBytes-

Fuzz test fails locally as expected within seconds

> go test -fuzz=Range
fuzz: elapsed: 0s, gathering baseline coverage: 0/42 completed
failure while testing seed corpus entry: FuzzRangeNodes/27d3f847ee8ebbcea67cc43e524feee604c2cfaf057db1c57a3d10b5bafd70cf
fuzz: elapsed: 0s, gathering baseline coverage: 0/42 completed
--- FAIL: FuzzRangeNodes (0.05s)
    --- FAIL: FuzzRangeNodes (0.00s)
        node_fuzz_test.go:16: begin=526, end=776
        node_fuzz_test.go:18: nodes=[{1 13} {2 7} {5 1} {6 1} {7 1} {8 1} {8 2} {3 96}]
        node_fuzz_test.go:25: got=26, want=526
    
FAIL
exit status 1
FAIL    github.com/transparency-dev/merkle/compact      2.868s

@AdamKorcz
Copy link

AdamKorcz commented Jul 22, 2022 •
edited

Thanks I now have ClusterFuzzLite running in GitHub Actions hickford/merkle#1

However when I tested it by deliberately breaking the test, the workflow succeeded regardless. Any ideas?

Fuzz test fails locally

> go test -fuzz=Range
fuzz: elapsed: 0s, gathering baseline coverage: 0/42 completed
failure while testing seed corpus entry: FuzzRangeNodes/27d3f847ee8ebbcea67cc43e524feee604c2cfaf057db1c57a3d10b5bafd70cf
fuzz: elapsed: 0s, gathering baseline coverage: 0/42 completed
--- FAIL: FuzzRangeNodes (0.05s)
    --- FAIL: FuzzRangeNodes (0.00s)
        node_fuzz_test.go:16: begin=526, end=776
        node_fuzz_test.go:18: nodes=[{1 13} {2 7} {5 1} {6 1} {7 1} {8 1} {8 2} {3 96}]
        node_fuzz_test.go:25: got=26, want=526
    
FAIL
exit status 1
FAIL    github.com/transparency-dev/merkle/compact      2.868s

ClusterFuzzLite is not running the libFuzzer binary here. The fuzzer is being run by way of go test -fuzz=.

Your workflow is correctly building the fuzzer with compile_native_go_fuzzer github.com/transparency-dev/merkle/compact FuzzRangeNodes FuzzRangeNodes. This produces the FuzzRangeNodes binary which must run in ClusterFuzzLite.

EDIT: Sorry, I thought that the stacktrace you added was from the CI. Please disregard this message and see the one below.

@AdamKorcz
Copy link

AdamKorcz commented Jul 22, 2022 •
edited

Thanks I now have ClusterFuzzLite running in GitHub Actions hickford/merkle#1

However when I tested it by deliberately breaking the test, the workflow succeeded regardless. Any ideas?

Fuzz test fails locally as expected

> go test -fuzz=Range
fuzz: elapsed: 0s, gathering baseline coverage: 0/42 completed
failure while testing seed corpus entry: FuzzRangeNodes/27d3f847ee8ebbcea67cc43e524feee604c2cfaf057db1c57a3d10b5bafd70cf
fuzz: elapsed: 0s, gathering baseline coverage: 0/42 completed
--- FAIL: FuzzRangeNodes (0.05s)
    --- FAIL: FuzzRangeNodes (0.00s)
        node_fuzz_test.go:16: begin=526, end=776
        node_fuzz_test.go:18: nodes=[{1 13} {2 7} {5 1} {6 1} {7 1} {8 1} {8 2} {3 96}]
        node_fuzz_test.go:25: got=26, want=526
    
FAIL
exit status 1
FAIL    github.com/transparency-dev/merkle/compact      2.868s

The fuzzer should find this bug in the CI. Could you try increasing the fuzz time to 30 minutes?

@AdamKorcz
Copy link

Okay, I see the issue here. I will get a fix ready for this.

@AdamKorcz
Copy link

@hickford This should now have been fixed in https://github.com/AdamKorcz/go-118-fuzz-build. Could you trigger the CI job again and ensure that you are using the latest version of github.com/AdamKorcz/go-118-fuzz-build?

@hickford
Copy link
Author

@AdamKorcz Thanks, yes working as expected now. hickford/merkle#1 When I try to break the test, the workflow fails. https://github.com/hickford/merkle/actions/runs/2732014085

@AdamKorcz
Copy link

@AdamKorcz Thanks, yes working as expected now. hickford/merkle#1 When I try to break the test, the workflow fails. https://github.com/hickford/merkle/actions/runs/2732014085

That is great!

@jonathanmetzman this issue can be closed.

@vroldanbet vroldanbet mentioned this issue Sep 21, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@hickford @DavidKorczynski @oliverchang @jonathanmetzman @AdamKorcz