Better support for fuzzing Go, remaining - code coverage

[Dor1s]

As of today, ClusterFuzz supports fuzzing Go in the gofuzz_libfuzzer mode: https://github.com/dvyukov/go-fuzz#libfuzzer-support

But it's not really convenient to integrate a new Golang project. I think we should do the following:

1. Upgrade the base-builder image to include a recent version of Go toolchain and go-fuzz, so that it was possible to run go and go-fuzz-build commands from build.sh without any problems.

2. Document it :)

Action items

• Remove go-fuzz-build and build fuzz targets using the standard toolchain (available since Go 1.14, we're already on 1.14.2, as we always install the latest stable version).
• (won't fix, see Better support for fuzzing Go, remaining - code coverage #2714 (comment)) Unify the build command (will likely need some wrapper to be included in the base-builder image), migrate all existing projects to the new build interface (instead of using a copy pasted bash function in all build.sh files).
• Update documentation to reflect the new preferred way of building fuzz targets (https://google.github.io/oss-fuzz/getting-started/new-project-guide/go-lang).
• Update srcmap script to work with Go deps (Update srcmap to collect information for Go projects #3355).
• When we get more projects integrated (maybe 30-50), add support for code coverage (Code coverage support for gofuzz projects #2817).

[catenacyber]

I am not sure if it fits here, but I would like to see "LAST TESTED REVISION" on oss-fuzz.com for golang projects
The trick is, I think, that build.sh uses go get instead of git clone
Example : https://oss-fuzz.com/testcase-detail/5453012747419648

[Dor1s]

Interesting! I guess you meant Dockerfile using go get, e.g.

oss-fuzz/projects/gonids/Dockerfile

Line 19 in e82397b

RUN go get github.com/google/gonids

Yeah, looks like our srcmap script should grab revisions from the repos isntalled via go get as well. I'll upload a patch, thanks for noticing this!

[catenacyber]

Exactly

Thanks @Dor1s for patching, I did not know where to look, or I would have proposed a patch myself ;-)

[inferno-chromium]

Tracking items:

1. Switch to native go instrumentation.
2. Make projects use the native go binary for building.
3. Fix documentation - https://google.github.io/oss-fuzz/getting-started/new-project-guide/go-lang
4. Add how to write fuzz targets (examples). this section has none - https://google.github.io/oss-fuzz/getting-started/new-project-guide/go-lang/#go-fuzz-support

fyi - @lukasz-milewski @mdempsky

[Dor1s]

Thanks @inferno-chromium, I've updated the issue description with more of these.

Also added code coverage (#2817) and srcmap support (#3355).

Regarding

4. Add how to write fuzz targets (examples). this section has none - https://google.github.io/oss-fuzz/getting-started/new-project-guide/go-lang/#go-fuzz-support

I'm not sure this would belong to OSS-Fuzz repo, as OSS-Fuzz docs focus on using OSS-Fuzz and do not cover how to write fuzz targets in general (e.g. https://google.github.io/oss-fuzz/getting-started/new-project-guide/ also doesn't have any C/C++ examples).

https://github.com/google/fuzzing might be a better place, or we can point to https://godoc.org/github.com/google/gofuzz

[Dor1s]

@inferno-chromium thanks a ton for adding the new instrumentation support (#3633) and migrating all Go projects to use it (#3638). Marking the first AI as done!

If you plan to work more on this, I think we should also unify the compile_fuzzer function which is copy-pasted in all build.sh files now and make it a common script inside base-builder (+ documentation) -- this is the second action item. Otherwise, all new projects will be also copy-pasting it, and the next migration will be more annoying.

Updating the docs after that is the third item :)

[inferno-chromium]

I thought about 2) and 3). i think it will create more confusion for folks when they want to do ideal integration (which we recommend, most projects should have build.sh in their repo). They would have to look this helper script and then integrate it as well. for 2-3 build commands, it feels like a hassle. e.g. https://go.googlesource.com/protobuf/+/d8bc21f7e13fa476be55b17983bd5d43ad8c7121/internal/fuzz/oss-fuzz-build.sh. i prefer to keep it in current way.

[Dor1s]

I thought about 2) and 3). i think it will create more confusion for folks when they want to do ideal integration

Ah, right, great point!

[jonathanmetzman]

I think @catenacyber completed this by adding code coverage support for go.