[dev.fuzz] internal/fuzz: don't store corpus in memory #46669
Labels
FrozenDueToAge
fuzz
Issues related to native fuzzing support
NeedsFix
The path to resolution is known, but the work has not been done.
Comments
rolandshoemaker
added
NeedsFix
22 tasks
|
Change https://golang.org/cl/345096 mentions this issue: |
gopherbot
pushed a commit
that referenced
this issue
Sep 2, 2021
Instead of holding all corpus data/values in memory, only store seed inputs added via F.Add in memory, and only load corpus entries which are written to disk when we need them. This should significantly reduce the memory required by the coordinator process. Additionally only load the corpus in the coordinator process, since the worker has no need for it. Fixes #46669. Change-Id: Ic3b0c5e929fdb3e2877b963e6b0fa14e140c1e1d Reviewed-on: https://go-review.googlesource.com/c/go/+/345096 Trust: Roland Shoemaker <roland@golang.org> Run-TryBot: Roland Shoemaker <roland@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Jay Conrod <jayconrod@google.com>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Currently the coordinator will read all on the input corpus into memory, and store any newly discovered inputs in memory as well. For large corpora this will result in significant memory usage (see #44517).
Instead the coordinator should only read inputs from disk when necessary, storing a list of input paths that are available, and only hold them as long as is necessary to send the input to a worker.
One open question is how to handle inputs added via
f.Add, since they only ever existed in memory. We could add these to the corpus cache, so there are files for them, or store just these values in memory (in CorpusEntry have eithervalueorfilepathmaybe?) and document that if you have a lot of these kinds of inputs you should store them as files rather than adding them viaf.Add(I've seen a number of targets which decompress archives of inputs and then add them viaf.Add, so this may not be viable for everyone).The text was updated successfully, but these errors were encountered: