Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go/internal/cache: provide GOFUZZCACHE environment variable #48901

Open
AlekSi opened this issue Oct 10, 2021 · 5 comments
Open

cmd/go/internal/cache: provide GOFUZZCACHE environment variable #48901

AlekSi opened this issue Oct 10, 2021 · 5 comments

Comments

@AlekSi
Copy link
Contributor

@AlekSi AlekSi commented Oct 10, 2021

Go provides GOCACHE and GOMODCACHE environment variables to control build and modules cache locations. There is no environment variable to control fuzz corpus cache location; it is hard-coded to be $GOCACHE/fuzz.

I think that fuzz cache is significantly different from the build (and test) cache:

  • it is being cleaned by different go clean flags;
  • it has a different lifetime (#48526);
  • it is a bit more valuable than a build cache (it takes more time to find interesting inputs than "just" compilation);
  • etc.

For that reason, I think it would be helpful to provide GOFUZZCACHE environment variable to control fuzz cache location. That would help people building automation around sharing that cache between machines.

(I'm also not really sure that fuzz cache directory should be a subdirectory of the build cache by default, but that might be too late to change that.)

/cc @katiehockman @jayconrod

@AlekSi

This comment has been hidden.

@gopherbot gopherbot added the fuzz label Oct 10, 2021
@jayconrod
Copy link
Contributor

@jayconrod jayconrod commented Oct 11, 2021

We had a GOFUZZCACHE environment variable early on, but we made a design decision to remove it. We didn't feel like it added enough value to justify the extra complexity in the CLI. I can't think of a good technical reason not to colocate it with the build cache. We wanted to store interesting inputs in a place the user has access to and somewhere that won't be indexed by desktop searching; that implies a location within the build cache or right next to it.

it is being cleaned by different go clean flags

Cached test results are also stored in the build cache and are also cleaned with a different flag.

it has a different lifetime

The cache eviction code will be able to tell the files apart. Cached fuzz inputs go in a separate subdirectory.

it is a bit more valuable than a build cache (it takes more time to find interesting inputs than "just" compilation);

We're considering a longer cache lifetime because of this. The linked issue above suggests removing fuzz inputs after 30 days without use. We do want some automatic eviction. If a fuzz target is removed or its input types are changed incompatibly, there's no reason to keep the cached files around. It's always possible to copy these files into testdata to preserve them permanently and share them with other developers.

@mvdan
Copy link
Member

@mvdan mvdan commented Oct 11, 2021

more valuable than a build cache

For what it's worth, this is the main reason I follow this feature request. If I spend a full hour burning CPU to get a nice corpus, I don't want that to just vanish because I haven't worked on the project for a few weeks.

I guess the answer should be "manually make a copy of the relevant corpus" whether GOFUZZCACHE exists or not. What I mainly want is an easy way to make a copy of the corpuses (corpora?) for a module, and I imagined that go env GOFUZZCACHE would be much saner than assuming $(go env GOCACHE)/fuzz.

@AlekSi
Copy link
Contributor Author

@AlekSi AlekSi commented Oct 11, 2021

that implies a location within the build cache or right next to it

I totally agree with "right next to it". But then that location should be configurable.

Cached test results are also stored in the build cache and are also cleaned with a different flag.

Well, that's not entirely true: go clean -cache removes both the build cache and the test cache; go clean -fuzzcache removes neither. From that behavior, it is clear that the build cache includes the test cache but not the fuzz cache. That's why I propose to separate them on the file system too.

We didn't feel like it added enough value to justify the extra complexity in the CLI.

Those environment variables are for advanced use-cases and users only. For example, I want to cache those caches separately on the CI. And you already expose this complexity via go clean flags as shown above; adding this environment variable adds some code to maintain, but not complexity for users.

@toothrot toothrot added this to the Backlog milestone Oct 12, 2021
@katiehockman katiehockman removed this from the Backlog milestone Oct 12, 2021
@katiehockman katiehockman added this to the Go1.19 milestone Oct 12, 2021
@katiehockman
Copy link
Member

@katiehockman katiehockman commented Oct 12, 2021

This isn't something that we can dedicate time for in 1.18, but let's re-visit for 1.19. I'm hopeful that 1.18 will provide some good user experience stories and help us better understand which features are most important. The extra context you provided here will be helpful it making that decision too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants