Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: auto cache clean fail #31931

Closed
josharian opened this issue May 9, 2019 · 9 comments
Closed

cmd/go: auto cache clean fail #31931

josharian opened this issue May 9, 2019 · 9 comments

Comments

@josharian
Copy link
Contributor

@josharian josharian commented May 9, 2019

I am in the process of running some toolchain experiments. (Results will probably be sent to golang-dev once it is complete.) This involves compiling executables tens of thousands of times, each time slightly differently.

It turns out that this wreaks havoc on the cache: Every single compilation results in a bunch of new cache entries, which over the course of 24 hours ate my hard drive. (And in the process, caused me to lose my results so far.) I'm several hours into go cache -clean. I have 360gb free and counting.

GOCACHE=off is no longer supported.

What should I have done (or better, what should cmd/go do) to prevent runaway cache growth? A hard size limit of 100gb for the cache would have saved me a world of trouble.

cc @bcmills @jayconrod

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented May 9, 2019

I'm several hours into go cache -clean.

That seems surprising. Could you file a separate issue for go cache -clean being slow? 🙂

@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented May 9, 2019

What should I have done (or better, what should cmd/go do) to prevent runaway cache growth?

Compiling executables tens of thousands of times with tiny variations is a very unusual mode of usage. It doesn't seem entirely unreasonable to expect you to run go cache -clean manually every so often during that experiment...

That said, if there is some portable way we can detect free space, it might be a good idea for the go command to periodically check the ratio of free space to cache usage and clean if it gets too high.

@bcmills bcmills added this to the Unplanned milestone May 9, 2019
@jayconrod

This comment has been minimized.

Copy link
Contributor

@jayconrod jayconrod commented May 9, 2019

I think this is a dup of #29561.

Definitely unfortunate behavior though. I don't think we can put an upper bound on cache size, but we ought to have a smarter eviction policy that takes cache size and free space into account.

For reference, the current cache eviction policy is hard-coded in cmd/go/internal/cache. We scan for deletable files at most once per 24 hours. We delete files that haven't been used in 5 days.

@jayconrod jayconrod closed this May 9, 2019
@josharian

This comment has been minimized.

Copy link
Contributor Author

@josharian josharian commented May 9, 2019

That seems surprising. Could you file a separate issue for go cache -clean being slow? 🙂

I'm not sure that it is slow. I think the problem is that it had to delete 400gb of smallish files. rm -rf would be similarly slow.

@josharian

This comment has been minimized.

Copy link
Contributor Author

@josharian josharian commented May 9, 2019

It doesn't seem entirely unreasonable to expect you to run go cache -clean manually every so often during that experiment...

One other sad note, just for posterity. Running go cache -clean concurrently with builds causes those builds to fail. So I'm now having to coordinate pausing and restarting all the other compilation jobs in order to clean the cache occasionally.

@jayconrod

This comment has been minimized.

Copy link
Contributor

@jayconrod jayconrod commented May 9, 2019

@josharian That doesn't seem like good behavior either. What should go clean -cache do with concurrent builds?

  • We could have a "clean" lock that would block go build and other commands until go clean -cache finishes.
  • go clean -cache could leave files created after it started. This might be hard to do reliably and portably though.
@bcmills

This comment has been minimized.

Copy link
Member

@bcmills bcmills commented May 9, 2019

I think that locking scheme is possible to implement using the existing cmd/go/internal/lockedfile API.

go build would obtain a read-lock on the file (os.O_CREATE|os.O_RDONLY)
go clean would obtain a write-lock on the file (os.O_CREATE|os.O_WRONLY)

We don't currently rely on file-locking for the cache, but since this would only prevent a build / clean race perhaps it's not so bad.

@josharian

This comment has been minimized.

Copy link
Contributor Author

@josharian josharian commented May 9, 2019

I'll file a new issue to discuss, thanks!

@josharian

This comment has been minimized.

Copy link
Contributor Author

@josharian josharian commented May 9, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.