cmd/go/internal/cache: provide GOFUZZCACHE environment variable

[AlekSi]

Go provides GOCACHE and GOMODCACHE environment variables to control build and modules cache locations. There is no environment variable to control fuzz corpus cache location; it is hard-coded to be $GOCACHE/fuzz.

I think that fuzz cache is significantly different from the build (and test) cache:

• it is being cleaned by different go clean flags;
• it has a different lifetime (cmd/go/internal/cache: in Cache.Trim, evict fuzz files #48526);
• it is a bit more valuable than a build cache (it takes more time to find interesting inputs than "just" compilation);
• etc.

For that reason, I think it would be helpful to provide GOFUZZCACHE environment variable to control fuzz cache location. That would help people building automation around sharing that cache between machines.

(I'm also not really sure that fuzz cache directory should be a subdirectory of the build cache by default, but that might be too late to change that.)

/cc @katiehockman @jayconrod

[AlekSi]

@gopherbot fuzz

[jayconrod]

We had a GOFUZZCACHE environment variable early on, but we made a design decision to remove it. We didn't feel like it added enough value to justify the extra complexity in the CLI. I can't think of a good technical reason not to colocate it with the build cache. We wanted to store interesting inputs in a place the user has access to and somewhere that won't be indexed by desktop searching; that implies a location within the build cache or right next to it.

it is being cleaned by different go clean flags

Cached test results are also stored in the build cache and are also cleaned with a different flag.

it has a different lifetime

The cache eviction code will be able to tell the files apart. Cached fuzz inputs go in a separate subdirectory.

it is a bit more valuable than a build cache (it takes more time to find interesting inputs than "just" compilation);

We're considering a longer cache lifetime because of this. The linked issue above suggests removing fuzz inputs after 30 days without use. We do want some automatic eviction. If a fuzz target is removed or its input types are changed incompatibly, there's no reason to keep the cached files around. It's always possible to copy these files into testdata to preserve them permanently and share them with other developers.

[mvdan]

more valuable than a build cache

For what it's worth, this is the main reason I follow this feature request. If I spend a full hour burning CPU to get a nice corpus, I don't want that to just vanish because I haven't worked on the project for a few weeks.

I guess the answer should be "manually make a copy of the relevant corpus" whether GOFUZZCACHE exists or not. What I mainly want is an easy way to make a copy of the corpuses (corpora?) for a module, and I imagined that go env GOFUZZCACHE would be much saner than assuming $(go env GOCACHE)/fuzz.

[AlekSi]

that implies a location within the build cache or right next to it

I totally agree with "right next to it". But then that location should be configurable.

Cached test results are also stored in the build cache and are also cleaned with a different flag.

Well, that's not entirely true: go clean -cache removes both the build cache and the test cache; go clean -fuzzcache removes neither. From that behavior, it is clear that the build cache includes the test cache but not the fuzz cache. That's why I propose to separate them on the file system too.

We didn't feel like it added enough value to justify the extra complexity in the CLI.

Those environment variables are for advanced use-cases and users only. For example, I want to cache those caches separately on the CI. And you already expose this complexity via go clean flags as shown above; adding this environment variable adds some code to maintain, but not complexity for users.

[katiehockman]

This isn't something that we can dedicate time for in 1.18, but let's re-visit for 1.19. I'm hopeful that 1.18 will provide some good user experience stories and help us better understand which features are most important. The extra context you provided here will be helpful it making that decision too.

[ianlancetaylor]

CC @golang/fuzzing

Moving to Backlog. Please recategorize as appropriate.