proposal: cmd/go: add `-fuzzcontinue` flag to `go test`

[mknyszek]

This is a proposal to add a flag to go test that keeps fuzzing even if a crash is found. This was originally planned as part of the work to add the -fuzz functionality, but didn't make it in at the time.

There's interest in such a flag, based on the response to #48127. The use-cases for such a flag are to:

1. Accumulate many crashes in a single long-running session.
2. Ignore fuzzing crashes for crashes that cannot be fixed. (@jakebailey has such a use-case.)

I propose adding the flag as -fuzzcontinue to keep the flags grouped by prefix. (This is @rsc's suggestion in #48127 (comment).)

This was requested at least two other times: #56430 and #54641.

CC @golang/command-line

[mknyszek]

This proposal is somewhat half-baked because I don't know enough about the fuzzing engine to know how hard this would be to add. My guess is that the fuzzing engine probably needs to ignore/avoid the crashes it already found in order to meaningfully make progress.

[mknyszek]

@jakebailey It would be helpful in moving this forward if you could explain your use-case in more detail, if you're able to share.

[jakebailey]

Concretely, I was fuzzing typescript-go's JavaScript Number implementation, since it has to be reimplemented now that we can't just use the engine we're running in.

I use differential fuzzing against a script shelled out to node, which is slow. I recently found the ccgo-ified quickjs and thought I'd use it to fuzz faster, which worked (some 100x faster), but then discovered bugs in both ccgo and quickjs itself, which I could not possibly fix. Since fuzzing stops, and fuzzing won't start if any existing fuzz tests fail, I had to manually start implementing skips to ignore these cases to find more bugs (which I did).

It'd be nice to not have to do that, though I assume the challenge is when the fuzzer can experience the same bug repeatedly and generate too many crashers. I somewhat hope that the coverage-based analysis could direct it elsewhere.

Also, my understanding is that projects like OSS-Fuzz do continuous fuzzing of projects, and are also able to discover multiple crashers; I don't know what they're doing to make that work (or if it's working at all), but it seems like it'd be helpful to be able to keep finding more bugs while upstream works on fixes.

[mknyszek]

Just to set clear expectations, I do not personally plan to work on this proposal or spend time trying to move it forward. I filed this issue to make sure the feature request doesn't get lost.

If you'd like to see this proposal move forward, please hit some positive emoji buttons on the first post and reply with concrete use-cases as @jakebailey very helpfully did.

[jakebailey]

Another thing I want to do is to have our own form of CI-based fuzzing that can auto-report new bugs on a schedule.

We used to have a project called tsserverfuzzer, which effectively spawned our langauge server and sent randomish requests to it (open file, edit file, hover, do completions, edit again) looking for crashes. That worked pretty well and we'd auto-open issues for those crashes. -fuzz can do a whole lot better thanks to coverage analysis.

[gabyhelp]

Related Issues

• fuzz testing: continue after finding bugs #56430 (closed)
• proposal: testing: flag to continue testing after -failfast failure #61009
• [dev.fuzz] proposal: default timeout should be disabled #44483 (closed)
• cmd/go: -keepfuzzing needs renaming, does not exist #48127 (closed)
• proposal: cmd/go: go test should always fuzz a bit even without -fuzz #50969 (closed)
• [dev.fuzz] cmd/go: add fuzzseed flag for deterministic testing #46221 (closed)
• proposal: cmd/go: make fuzzing a first class citizen, like tests or benchmarks #19109 (closed)
• [dev.fuzz] cmd/go: add fuzz timeout flag #46220 (closed)
• cmd/go: add support for dealing with flaky tests #62244
• [dev.fuzz] cmd/go: don't run the other tests if `-fuzz` is specified #46222 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[thepudds]

My understanding is there are three main reasons why -keepfuzzing or -fuzzcontinue (or some other spelling) is not yet implemented in cmd/go:

1. There is not yet de-duplication logic for crashers. This exists in dvyukov/go-fuzz, but was not ported over. (The majority of the core fuzzing logic for the cmd/go fuzzing support is a fairly direct port from dvyukov/go-fuzz). I think not having de-dup logic for crashers was a reasonable engineering choice for v1, though I think it is important to have for a complete solution. At this point, the cmd/go fuzzing is still essentially at that first version.

2. When I talked to Bryan a while ago about me working on adding support for this, my understanding is he felt it a little unnatural to have "run forever" logic in the action graph, which is normally populated by actions that have a finite duration. At the time, my personal take was that was surmountable, either by accepting some amount of unnaturalness, or working the restart logic into one of the actions so that the action graph writ large was basically unaware that this particular action might run forever, or having something outside of the action graph, or another parent process, or some other hopefully reasonable solution, though I did not delve super deep into this piece of it at the time.

3. For a period of time, as I understand it, it was stated that there was not really any bandwidth available for reviewing fuzzing-related contributions.

And then of course, there's always the standard "someone needs to work on it".

If there is some loose agreement we are past the period of issue 3, I'd be pretty interested in working on this. (I have some experience in this area, including Dmitry was kind enough to make me a co-maintainer on dvyukov/go-fuzz several years ago, though to state the obvious -- I am nowhere close to Dmitry's league 😅).

I would start by dusting off my earlier attempt, though probably not until after the 1.26 freeze given I have some other things I am hoping to land before the freeze.

---

One other quick note is that you can somewhat emulate -keepfuzzing today with an outer loop in bash or similar (including moving the crasher out testdata; I've done a couple of ad-hoc versions of this in the past, though TBH, when I want to fuzz something important in Go, I usually reach back to using dvyukov/go-fuzz, which usually yields better results than cmd/go for a variety of reasons). All that said, some external outer loop is not nearly as good as having de-duplication logic for crashers built in.

[thepudds]

Hi @jakebailey, separately, I would definitely encourage you or one of your colleagues to hook typescript-go up to OSS-Fuzz.

OSS-Fuzz's de-duplication logic and corpus management is not constrained by the absence of a -keepfuzzing flag in cmd/go because it uses a different fuzzing engine than cmd/go's engine. (It supports the cmd/go testing.F Go fuzzing API, but does not use the cmd/go engine for the actual fuzzing).

Also, it is still coverage-based even for Go projects, so it still yields much better results than random data without coverage.

I believe TypeScript would be considered to be both "critical" and to "have a significant user base" (as long as one does not trust TIOBE 😅):

To be accepted to OSS-Fuzz, an open-source project must have a significant user base and/or be critical to the global IT infrastructure.

It is usually fairly straightforward to add a Go project. Here's one entry point to the doc:

https://google.github.io/oss-fuzz/getting-started/new-project-guide/go-lang/

(I helped a small amount in getting Go fuzzing stood up in OSS-Fuzz as an outside contributor maybe roughly 5-6 years ago, though I have not looked carefully at it recently).

[jakebailey]

I think our original compiler is a part of OSS-Fuzz, though the kinds of reports we were getting for a while were things like "if you try to parse 1000 open parens, it crashes", which are not things we generally address.

Depending, we might have stuff that is more actionable to fuzz, though up to this point, we've almost exclusively done differential fuzzing against old JS code, or slower algorithms we improve but want to verify.