Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: drop $GOPATH/pkg #4719

Open
bradfitz opened this issue Jan 28, 2013 · 42 comments

Comments

Projects
None yet
@bradfitz
Copy link
Member

commented Jan 28, 2013

(after discussion with Russ, trying to summarize here)

It's nice to be able to tell people to set $GOPATH == $HOME, which means their source
goes into $HOME/src and their binaries conveniently go into $HOME/bin.

The only wart with $GOPATH == $HOME is the $HOME/pkg directory, which is pretty ugly,
not useful for end users, and somewhat offensive to be littering in people's $HOME.

If issue #4443 and issue #3895 get fixed, $GOPATH/pkg might get even uglier.

Unlike $GOROOT/pkg, $GOPATH/pkg doesn't contain tool binaries.  It's pretty much just a
cache.

Can't cmd/go's *.a cache go into /tmp/go-cache-$USER 0700 or ~/Library/Caches with
appropriate occasional lazy cleaning?  (or letting the system clean).
@adg

This comment has been minimized.

Copy link
Contributor

commented Jan 29, 2013

Comment 1:

Do you propose to change the behavior for everyone, or make this configurable?
What about people that want to ship .a binaries only? I think the go tool currently
supports that. (ie, just mkdir the src/ path and put the binaries under pkg/ and it
works)
@minux

This comment has been minimized.

Copy link
Member

commented Jan 29, 2013

Comment 2:

FYI, for binary-only package to work, you will need to create a dummy "package xxx" Go
source
file in the directory with an older timestamp than the .a file.
This is issue #2775.
@rsc

This comment has been minimized.

Copy link
Contributor

commented Jan 29, 2013

Comment 3:

There are many facets to think about. This is just an issue to remind us to
do so.
@rsc

This comment has been minimized.

Copy link
Contributor

commented Jan 30, 2013

Comment 4:

Labels changed: added priority-later, go1.1maybe, removed priority-triage, go1.1.

Status changed to Thinking.

@robpike

This comment has been minimized.

Copy link
Contributor

commented Mar 7, 2013

Comment 5:

Labels changed: removed go1.1maybe.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Jul 30, 2013

Comment 6:

Labels changed: added go1.3.

@robpike

This comment has been minimized.

Copy link
Contributor

commented Aug 20, 2013

Comment 7:

Labels changed: removed go1.3.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Nov 27, 2013

Comment 8:

Labels changed: added go1.3maybe.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Dec 4, 2013

Comment 9:

Labels changed: added release-none, removed go1.3maybe.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Dec 4, 2013

Comment 10:

Labels changed: added repo-main.

@rsc rsc added this to the Unplanned milestone Apr 10, 2015

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Sep 29, 2015

Every release (especially Go 1.5) has been getting us closer to eliminating "pkg" and instead having a build cache directory somewhere (some system-specific cache directory)

Having a cache instead of "pkg" would also eliminate the frequent problems people run into where "go build" does too much work when they could've used "go install" instead.

/cc @adg @davecheney

@griesemer

This comment has been minimized.

Copy link
Contributor

commented Sep 29, 2015

I would love see the pkg directory go away.

I've always viewed it as an artifact of our implementation which is based on historic ideas of how a program is compiled into object files which are then linked. (In the earliest days of Go I suggested that we make this compile/link process invisible - i.e., as a programmer one just "sees" source code. But we were not ready at that time for this).

@davecheney

This comment has been minimized.

Copy link
Contributor

commented Sep 29, 2015

@bradfitz,

I view 'pkg' as this cache, and I'm fine with it living inside GOPATH
because that makes it simple to explain where to find it if people are
determined to delete it.

The change I implemented in gb was to remove the distinction between go
build, go install, and the other odd -I variants; if gb can cache
something, even while building tests, it will and this has worked out
pretty well.

For my money removing the number of ways one can tell them go tool to cache
things and just make the caching more pervasive would be a welcome
improvement.

If that cache lives inside GOPATH, or some default location relative to
HOME feels less important, but I suspect that the windows users will have a
issues with a large number of files written to their home directories,
which may not be local to their machines.

Thanks

Dave

On Wed, 30 Sep 2015 03:17 Robert Griesemer notifications@github.com wrote:

I would love see the pkg directory go away.

I've always viewed it as an artifact of our implementation which is based
on historic ideas of how a program is compiled into object files which are
then linked. (In the earliest days of Go I suggested that we make this
compile/link process invisible - i.e., as a programmer one just "sees"
source code. But we were not ready at that time for this).


Reply to this email directly or view it on GitHub
#4719 (comment).

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Sep 29, 2015

makes it simple to explain where to find it if people are
determined to delete it

If the cache is 100% correct (which isn't unimaginable; bazel does it, and Go 1.5 is basically there already), then deleting it is unnecessary. If we really wanted, we could have a go XXX subcommand to free up disk space, if that's the concern.

Great to hear that gb does caching for both build and install!

If we get to the point where the current "pkg" is just a cache and also cached go build, that's great, but then my next annoyance is that I have to look at it. I like to set my $GOPATH to $HOME, and that means I have $HOME/pkg being a useless turd right in my home directory. So I'd rather put things in the right spots:

@davecheney

This comment has been minimized.

Copy link
Contributor

commented Sep 29, 2015

I cannot explain why people want to delete GOPATH/pkg, I believe it is
unnecessary, and in every case where I have asked for more details not have
been provided, yet the lore continues to propagate itself.

I'm fine with using the LSB mandates tmp directories, that feels like the
smaller argument vs making the go tool more aggressive with its caching.

On Wed, 30 Sep 2015 07:50 Brad Fitzpatrick notifications@github.com wrote:

makes it simple to explain where to find it if people are
determined to delete it

If the cache is 100% correct (which isn't unimaginable; bazel does it, and
Go 1.5 is basically there already), then deleting it is unnecessary. If we
really wanted, we could have a go XXX subcommand to free up disk space,
if that's the concern.

Great to hear that gb does caching for both build and install!

If we get to the point where the current "pkg" is just a cache and also
cached go build, that's great, but then my next annoyance is that I have
to look at it. I like to set my $GOPATH to $HOME, and that means I have
$HOME/pkg being a useless turd right in my home directory. So I'd rather
put things in the right spots:


Reply to this email directly or view it on GitHub
#4719 (comment).

@mwhudson

This comment has been minimized.

Copy link
Contributor

commented Sep 30, 2015

If pkg is to become a cache, and possibly more hidden, it would be good to fix the ways in which GOOS + GOARCH is not quite a valid cache key: GOARM is probably the worst offender here, but also tags and things like buildmodes and the race detector (although the go tool handles the last two via installstuffix currently).

@minux

This comment has been minimized.

Copy link
Member

commented Sep 30, 2015

@ChrisHines

This comment has been minimized.

Copy link
Contributor

commented Nov 13, 2015

Forgive me if I have missed subtleties that have already been discussed.

I have not seen any discussion of how a system global pkg cache would work for people who switch between multiple GOPATHs, each of which might contain different versions of the same package(s). I understand that recent versions of the go tool will hash all the file names used to build a .a file to help keep the cache accurate. But different versions of a package in different GOPATHs could easily have the same file list but different code. It seems to me either separate caches are required per GOPATH (as the pkg directory does today) or the cache key would need to include something to differentiate between the multiple GOPATHs used.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Nov 13, 2015

Yes, it would have to be a good cache. :-)

@rsc

This comment has been minimized.

Copy link
Contributor

commented Nov 13, 2015

(That is, it would have to take all these things into consideration. It's not a trivial problem, which is why it hasn't happened.)

@rogpeppe

This comment has been minimized.

Copy link
Contributor

commented Dec 15, 2015

+1 to this, particularly making it toolchain-sensitive too. I often find myself switching between different compiler versions so I can check behaviour differences, and switching/removing $GOPATH/pkg is a pain.

@rsc rsc modified the milestones: Unplanned, Go1.12 Apr 23, 2018

@rsc

This comment has been minimized.

Copy link
Contributor

commented Apr 23, 2018

Tentatively putting in Go 1.12. Note that this means disallowing GOCACHE=off.

@JeanMertz

This comment has been minimized.

Copy link

commented May 6, 2018

Note that this means disallowing GOCACHE=off

Just to double-check, and my use-case might not be the norm, but does this mean having no way whatsoever to disable caching, outside of deleting the cache itself?

Reason I ask is that I sometimes am working with randomly-failing integration tests that I want to run in a loop to have it trigger the error, since it doesn't trigger every time.

Right now, when working with such projects, I always export GOCACHE=off before I start testing, because I know one successful run won't mean the test works the second time, and so I want to keep triggering those random failures so that I can debug and squash them.

Again, this might be an edge-case, and I can change those loops to remove the cache before running, but I just wanted to put this use-case out here for consideration.

@davecheney

This comment has been minimized.

Copy link
Contributor

commented May 6, 2018

@abitrolly

This comment has been minimized.

Copy link

commented Jul 23, 2018

I came here from #17271 (comment) and I am interested if dropping $GOPATH/pkg will allow me to have my dependency in vendor/dependency checkout without touching $GOPATH at all?

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Jul 23, 2018

@abitrolly, the bug you were at was the right place. This bug is more of a cleanup that's not very user-visible.

@abitrolly

This comment has been minimized.

Copy link

commented Jul 23, 2018

Now that the previous bug is closed, does that mean that the feature with vendor imports without $GOPATH works already in 1.12?

@rajender

This comment has been minimized.

Copy link
Contributor

commented Jul 23, 2018

@abitrolly minimal vendor support without GOPATH is being added as part of new modules feature to Go.11. Note Go.1.11 is not released yet. You can try the beta version.

@gopherbot

This comment has been minimized.

Copy link

commented Jul 30, 2018

Change https://golang.org/cl/126755 mentions this issue: cmd/go: move module cache from $GOPATH/src/mod to $GOPATH/pkg/mod

gopherbot pushed a commit that referenced this issue Aug 1, 2018

cmd/go: move module cache from $GOPATH/src/mod to $GOPATH/pkg/mod
Using $GOPATH/src/mod confuses too many tools.
$GOPATH/pkg/mod seems better for now.
It's also next to dep's cache, $GOPATH/pkg/dep.
If we do eliminate GOPATH/pkg for holding .a files (#4719)
then we could still keep it around for pkg/mod.
(Or we could move the module cache again then.)

Fixes #26401.
Fixes #26635.

Change-Id: I18f7da216ed9f490eded3c00d837fb086ae5b6a4
Reviewed-on: https://go-review.googlesource.com/126755
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Rob Pike <r@golang.org>
@gopherbot

This comment has been minimized.

Copy link

commented Aug 17, 2018

Change https://golang.org/cl/129679 mentions this issue: doc/code: drop mentions of GOPATH/pkg directory

gopherbot pushed a commit that referenced this issue Aug 17, 2018

doc/code: drop mentions of GOPATH/pkg directory
It's already half gone and later will be all gone.
It's not worth explaining in an introduction doc.

Fixes #24506
Updates #4719

Change-Id: Ie48128b3aa090d84e0e734aa45f14a4480292913
Reviewed-on: https://go-review.googlesource.com/129679
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

@bcmills bcmills modified the milestones: Go1.12, Go1.13 Oct 25, 2018

@perillo

This comment has been minimized.

Copy link

commented Mar 4, 2019

shouldn't the module cache be in os.UserCacheDir() instead of $GOPATH/pkg/mod?

@bradfitz

This comment has been minimized.

Copy link
Member Author

commented Mar 4, 2019

@perillo, we discussed that (somewhere) earlier and decided to just use $GOPATH/pkg for now while we had it. But yes, it'd have to move later when this bug is resolved.

@rsc

This comment has been minimized.

Copy link
Contributor

commented Mar 6, 2019

@perillo, or we might leave the module source cache in $GOPATH/pkg.

I think there are good reasons not to put cached module downloads in $HOME/.cache. Quoting myself on https://dave.cheney.net/2018/07/14/taking-go-modules-for-a-spin and https://groups.google.com/forum/#!msg/golang-dev/RjSj4bGSmsw/KMHhU8fmAwAJ, back when the module cache was $GOPATH/src/mod not $GOPATH/pkg/mod:

The build cache ($GOCACHE, defaulting to $HOME/.cache/go-build) is for storing recent compilation results, so that if you need to do that exact compilation again, you can just reuse the file. The build cache holds entries that are like “if you run this exact compiler on these exact inputs. this is the output you’d get.” If the answer is not in the cache, your build uses a little more CPU to run the compiler instead of reusing the output. But you are guaranteed to be able to run the compiler instead, since you have the exact inputs and the compiler binary (or else you couldn’t even look up the answer in the cache).

The module cache ($GOPATH/src/mod, defaulting to $HOME/go/src/mod) is for storing downloaded source code, so that every build does not redownload the same code and does not require the network or the original code to be available. The module cache holds entries that are like “if you need to download mymodule@v1.2.3, here are the files you’d get.” If the answer is not in the cache, you have to go out to the network. Maybe you don’t have a network right now. Maybe the code has been deleted. It’s not anywhere near guaranteed that you can redownload the sources and also get the same result. Hopefully you can, but it’s not an absolute certainty like for the build cache. (The go.sum file will detect if you get a different answer on re-download, but knowing you got the wrong bits doesn’t help you make progress on actually building your code. Also these paths end up in file-line information in binaries, so they show up in stack traces, and the like and feed into tools like text editors or debuggers that don’t necessarily know how to trigger the right cache refresh.)

I think that's still basically right. If the module cache is the only thing left in $GOPATH/pkg, we could decide to move it elsewhere, or we could decide to leave it there. Moving it to $HOME/.cache seems wrong though.

Really I'm just restating what I said in b8f42d7 on Jul 30, 2018 above:

If we do eliminate GOPATH/pkg for holding .a files (#4719)
then we could still keep it around for pkg/mod.
(Or we could move the module cache again then.)
@vith

This comment has been minimized.

Copy link

commented Mar 6, 2019

@rsc I think I understand and agree with all those points individually, but I'm missing the link between them and being opposed to using $XDG_CACHE_HOME (default $HOME/.cache) or its equivalent on other platforms.

My motivation for wanting to use a platform's standard cache location is that these caches tend to grow and never shrink, and eventually it becomes necessary to reclaim some disk space. Having a single location where all "well behaved" software stores caches that are safe to delete makes it more convenient to do that kind of maintenance.

Without standards like the XDG base directory spec, most software tends to think it deserves its own entry in your ls -a ~ output. But the more software you use, the more of a mess that becomes.

I suspect we have a different level of concern around this point:

It’s not anywhere near guaranteed that you can redownload the sources and also get the same result.

I don't think ~/.cache is meant to be limited to things that could be recomputed from locally available non-cache data. Looking inside mine, I see multiple browser caches, yarn module cache, typescript typing cache, a system package manager cache (yay), etc. All of those are caching data from the internet.

Looking at the XDG data dir spec, if a distinction must be made between the durability of the two go caches, maybe $XDG_DATA_HOME ($HOME/.local/share) should be considered.

@tv42

This comment has been minimized.

Copy link

commented Mar 6, 2019

I do get rsc's point, some of Go cache is "cheap to discard", some is "expensive to discard and potentially unavailable if lost". Note that XDG says nothing about how $XDG_CACHE_HOME should or can be cleaned, or how often etc, all it says is the directory is for "non-essential (cached) data". So I guess you could argue that any automatic tool that removes ~/.cache/go/mod or whatever in way that annoys the user is just too eager, and should learn better rules about prioritization and retainment. Maybe.

For what it's worth, rust seems to plan migrating from ~/.cargo to ~/.cache/cargo for what seem to me to be similar uses: rust-lang/cargo#148 (don't trust Github claims of "Merged").

@mathstuf

This comment has been minimized.

Copy link

commented Mar 6, 2019

The other thing a single cache directory location is for is a simple place to exclude things from backups.

@flibustenet

This comment has been minimized.

Copy link

commented Mar 7, 2019

Why will we want a long cache when there will be global modules proxies ?
I think if we want to be really safe we need a cache per project (instead of vendor ?). I'd like this cache hardlinked on linux.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.