Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal: cmd/go: add GOMODCACHE #34527

Open
mvdan opened this issue Sep 25, 2019 · 7 comments

Comments

@mvdan
Copy link
Member

commented Sep 25, 2019

Summary

Add GOMODCACHE to control where the module download cache lives. Its default can continue to be GOPATH[0]/pkg/mod, and the variable would be very similar and consistent with GOCACHE.

Description

The module download cache has lived in GOPATH[0]/pkg/mod/ since it first appeared. It's understandable why it doesn't live under GOCACHE, where the build cache is located; builds are generally fast and reliable if one has the source, but downloading a module from the internet isn't nearly as reliable.

I also understand why it was put under GOPATH; until recently, it was the only persistent directory that Go made use of. That only changed in the last release, with the addition of os.UserConfigDir()+"/go/env" for go env -w.

However, there's no way to configure where the module download cache is located. For example, this is useful in CI environments to place the build and module download caches somewhere that's persisted between builds.

The only way to store the download cache elsewhere is to move GOPATH entirely. This has several disadvantages:

  1. GOPATH contains much more. For many users, it still contains code. For almost everyone, it also contains the installed binaries, unless they've set GOBIN. It's too big of a knob to just change the location of the module download cache.

  2. Many environments explicitly set GOPATH, such as the golang:1.13 image, meaning that we can't simply go env -w GOPATH=/custom/path just like we could with GOCACHE. This makes the module download cache harder to deal with than the build cache, for no apparent reason.

  3. GOPATH's future is uncertain; it might contain more in the future, or it might go away entirely. Relying on it to set the module download cache location is not a good long-term plan.

This idea first came up in #31283, which was closed by its author. I left a comment there a while ago, but thought it would be better to open a new proposal.

@gopherbot gopherbot added this to the Proposal milestone Sep 25, 2019
@gopherbot gopherbot added the Proposal label Sep 25, 2019
@mvdan

This comment has been minimized.

Copy link
Member Author

commented Sep 25, 2019

One unfortunate side of this proposal is that GOCACHE seems to mean "directory holding all Go cached data", while GOMODCACHE would be separate. If we had a time machine, I'd argue that GOCACHE should be called GOBUILDCACHE instead.

An alternative idea is to eventually re-define GOPATH as a directory to store persistent data that doesn't belong in the short-lived GOCACHE nor in <config>/go/env. That would solve points 1 and 3, but not 2.

An alternative partial solution to point 2 is to stop explicitly declaring GOPATH in the official Docker images. That might break users who haven't switched over to $(go env GOPATH) to handle the default properly, but I hope that eventually the env var can be removed anyway.

@dmitshur

This comment has been minimized.

Copy link
Member

commented Sep 26, 2019

Thanks for making this proposal. I briefly recall conversations in the past that we might want to move the module cache out of $GOPATH/pkg eventually, but it's helpful to have something more actionable open.

I would like to see the module cache to not be in GOPATH in the future. If it's viable, I think a better outcome is if we can do this without adding configuration. That may not be viable.

@mvdan

This comment has been minimized.

Copy link
Member Author

commented Sep 26, 2019

I think we have multiple options if moving the module download cache is an option. For example:

  • Put it under GOCACHE, since it's a cache after all - just an expensive one to rebuild. One could end up with a directory structure like GOCACHE/build and GOCACHE/mod, to separate the cheap build cache.
  • Put it alongside go-build in os.UserCacheDir, for example <go-mod>. Though for consistency, you'd probably still require GOMODCACHE.

Both of these options put the expensive and cheap caches in the same place by default, though. If we want to properly separate the moduel cache from the build cache, here's another idea:

  • Put it under os.UserDataDir, which in an ideal world would be a suitable and portable place to put non-config, non-temporary data.
@bcmills

This comment has been minimized.

Copy link
Member

commented Sep 27, 2019

@jayconrod

This comment has been minimized.

Copy link
Contributor

commented Sep 27, 2019

Adding a GOMODCACHE variable to control the cache location makes total sense to me.

Moving the default location of the module cache seems more disruptive. Should we have some logic that checks if $GOPATH[0]/pkg/mod exists, then if not, some directory in os.UserCacheDir()?

@mvdan

This comment has been minimized.

Copy link
Member Author

commented Sep 27, 2019

I think if we're happy with adding an env var, we shouldn't worry about moving the default location right now. I only replied to that point to show some thoughts that might be helpful in the long term, if we do want to move it at some point.

@shoenig

This comment has been minimized.

Copy link
Contributor

commented Oct 13, 2019

I found myself wanting a GOMODCACHE variable recently for another use case:
If for example you have a local machine running an IDE talking to a remote server where your source tree lives mounted over NFS, Go modules are going to cause a bad time due to their use of flock, which NFS does not support.

go: failed to lock file at /Volumes/nfs/go/pkg/mod/cache/lock

Of course there is a workaround by creating a dummy GOPATH tree on the local machine and configuring the IDE to prepend that GOPATH ahead of the real one, so that it caches modules locally. That's a rather counter-intuitive thing to do though, and a direct environment variable for controlling the mod directory would be a nice feature.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.