Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: drop $GOPATH/pkg #4719

Closed
bradfitz opened this issue Jan 28, 2013 · 43 comments
Closed

cmd/go: drop $GOPATH/pkg #4719

bradfitz opened this issue Jan 28, 2013 · 43 comments

Comments

@bradfitz
Copy link
Contributor

(after discussion with Russ, trying to summarize here)

It's nice to be able to tell people to set $GOPATH == $HOME, which means their source
goes into $HOME/src and their binaries conveniently go into $HOME/bin.

The only wart with $GOPATH == $HOME is the $HOME/pkg directory, which is pretty ugly,
not useful for end users, and somewhat offensive to be littering in people's $HOME.

If issue #4443 and issue #3895 get fixed, $GOPATH/pkg might get even uglier.

Unlike $GOROOT/pkg, $GOPATH/pkg doesn't contain tool binaries.  It's pretty much just a
cache.

Can't cmd/go's *.a cache go into /tmp/go-cache-$USER 0700 or ~/Library/Caches with
appropriate occasional lazy cleaning?  (or letting the system clean).
@adg
Copy link
Contributor

adg commented Jan 29, 2013

Comment 1:

Do you propose to change the behavior for everyone, or make this configurable?
What about people that want to ship .a binaries only? I think the go tool currently
supports that. (ie, just mkdir the src/ path and put the binaries under pkg/ and it
works)

@minux
Copy link
Member

minux commented Jan 29, 2013

Comment 2:

FYI, for binary-only package to work, you will need to create a dummy "package xxx" Go
source
file in the directory with an older timestamp than the .a file.
This is issue #2775.

@rsc
Copy link
Contributor

rsc commented Jan 29, 2013

Comment 3:

There are many facets to think about. This is just an issue to remind us to
do so.

@rsc
Copy link
Contributor

rsc commented Jan 30, 2013

Comment 4:

Labels changed: added priority-later, go1.1maybe, removed priority-triage, go1.1.

Status changed to Thinking.

@robpike
Copy link
Contributor

robpike commented Mar 7, 2013

Comment 5:

Labels changed: removed go1.1maybe.

@rsc
Copy link
Contributor

rsc commented Jul 30, 2013

Comment 6:

Labels changed: added go1.3.

@robpike
Copy link
Contributor

robpike commented Aug 20, 2013

Comment 7:

Labels changed: removed go1.3.

@rsc
Copy link
Contributor

rsc commented Nov 27, 2013

Comment 8:

Labels changed: added go1.3maybe.

@rsc
Copy link
Contributor

rsc commented Dec 4, 2013

Comment 9:

Labels changed: added release-none, removed go1.3maybe.

@rsc
Copy link
Contributor

rsc commented Dec 4, 2013

Comment 10:

Labels changed: added repo-main.

@bradfitz
Copy link
Contributor Author

Every release (especially Go 1.5) has been getting us closer to eliminating "pkg" and instead having a build cache directory somewhere (some system-specific cache directory)

Having a cache instead of "pkg" would also eliminate the frequent problems people run into where "go build" does too much work when they could've used "go install" instead.

/cc @adg @davecheney

@griesemer
Copy link
Contributor

I would love see the pkg directory go away.

I've always viewed it as an artifact of our implementation which is based on historic ideas of how a program is compiled into object files which are then linked. (In the earliest days of Go I suggested that we make this compile/link process invisible - i.e., as a programmer one just "sees" source code. But we were not ready at that time for this).

@davecheney
Copy link
Contributor

@bradfitz,

I view 'pkg' as this cache, and I'm fine with it living inside GOPATH
because that makes it simple to explain where to find it if people are
determined to delete it.

The change I implemented in gb was to remove the distinction between go
build, go install, and the other odd -I variants; if gb can cache
something, even while building tests, it will and this has worked out
pretty well.

For my money removing the number of ways one can tell them go tool to cache
things and just make the caching more pervasive would be a welcome
improvement.

If that cache lives inside GOPATH, or some default location relative to
HOME feels less important, but I suspect that the windows users will have a
issues with a large number of files written to their home directories,
which may not be local to their machines.

Thanks

Dave

On Wed, 30 Sep 2015 03:17 Robert Griesemer notifications@github.com wrote:

I would love see the pkg directory go away.

I've always viewed it as an artifact of our implementation which is based
on historic ideas of how a program is compiled into object files which are
then linked. (In the earliest days of Go I suggested that we make this
compile/link process invisible - i.e., as a programmer one just "sees"
source code. But we were not ready at that time for this).


Reply to this email directly or view it on GitHub
#4719 (comment).

@bradfitz
Copy link
Contributor Author

makes it simple to explain where to find it if people are
determined to delete it

If the cache is 100% correct (which isn't unimaginable; bazel does it, and Go 1.5 is basically there already), then deleting it is unnecessary. If we really wanted, we could have a go XXX subcommand to free up disk space, if that's the concern.

Great to hear that gb does caching for both build and install!

If we get to the point where the current "pkg" is just a cache and also cached go build, that's great, but then my next annoyance is that I have to look at it. I like to set my $GOPATH to $HOME, and that means I have $HOME/pkg being a useless turd right in my home directory. So I'd rather put things in the right spots:

@davecheney
Copy link
Contributor

I cannot explain why people want to delete GOPATH/pkg, I believe it is
unnecessary, and in every case where I have asked for more details not have
been provided, yet the lore continues to propagate itself.

I'm fine with using the LSB mandates tmp directories, that feels like the
smaller argument vs making the go tool more aggressive with its caching.

On Wed, 30 Sep 2015 07:50 Brad Fitzpatrick notifications@github.com wrote:

makes it simple to explain where to find it if people are
determined to delete it

If the cache is 100% correct (which isn't unimaginable; bazel does it, and
Go 1.5 is basically there already), then deleting it is unnecessary. If we
really wanted, we could have a go XXX subcommand to free up disk space,
if that's the concern.

Great to hear that gb does caching for both build and install!

If we get to the point where the current "pkg" is just a cache and also
cached go build, that's great, but then my next annoyance is that I have
to look at it. I like to set my $GOPATH to $HOME, and that means I have
$HOME/pkg being a useless turd right in my home directory. So I'd rather
put things in the right spots:


Reply to this email directly or view it on GitHub
#4719 (comment).

@mwhudson
Copy link
Contributor

If pkg is to become a cache, and possibly more hidden, it would be good to fix the ways in which GOOS + GOARCH is not quite a valid cache key: GOARM is probably the worst offender here, but also tags and things like buildmodes and the race detector (although the go tool handles the last two via installstuffix currently).

@minux
Copy link
Member

minux commented Sep 30, 2015 via email

@ChrisHines
Copy link
Contributor

Forgive me if I have missed subtleties that have already been discussed.

I have not seen any discussion of how a system global pkg cache would work for people who switch between multiple GOPATHs, each of which might contain different versions of the same package(s). I understand that recent versions of the go tool will hash all the file names used to build a .a file to help keep the cache accurate. But different versions of a package in different GOPATHs could easily have the same file list but different code. It seems to me either separate caches are required per GOPATH (as the pkg directory does today) or the cache key would need to include something to differentiate between the multiple GOPATHs used.

@rsc
Copy link
Contributor

rsc commented Nov 13, 2015

Yes, it would have to be a good cache. :-)

@rsc
Copy link
Contributor

rsc commented Nov 13, 2015

(That is, it would have to take all these things into consideration. It's not a trivial problem, which is why it hasn't happened.)

@rogpeppe
Copy link
Contributor

+1 to this, particularly making it toolchain-sensitive too. I often find myself switching between different compiler versions so I can check behaviour differences, and switching/removing $GOPATH/pkg is a pain.

@rajender
Copy link
Contributor

@abitrolly minimal vendor support without GOPATH is being added as part of new modules feature to Go.11. Note Go.1.11 is not released yet. You can try the beta version.

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/126755 mentions this issue: cmd/go: move module cache from $GOPATH/src/mod to $GOPATH/pkg/mod

gopherbot pushed a commit that referenced this issue Aug 1, 2018
Using $GOPATH/src/mod confuses too many tools.
$GOPATH/pkg/mod seems better for now.
It's also next to dep's cache, $GOPATH/pkg/dep.
If we do eliminate GOPATH/pkg for holding .a files (#4719)
then we could still keep it around for pkg/mod.
(Or we could move the module cache again then.)

Fixes #26401.
Fixes #26635.

Change-Id: I18f7da216ed9f490eded3c00d837fb086ae5b6a4
Reviewed-on: https://go-review.googlesource.com/126755
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Rob Pike <r@golang.org>
@gopherbot
Copy link
Contributor

Change https://golang.org/cl/129679 mentions this issue: doc/code: drop mentions of GOPATH/pkg directory

gopherbot pushed a commit that referenced this issue Aug 17, 2018
It's already half gone and later will be all gone.
It's not worth explaining in an introduction doc.

Fixes #24506
Updates #4719

Change-Id: Ie48128b3aa090d84e0e734aa45f14a4480292913
Reviewed-on: https://go-review.googlesource.com/129679
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@bcmills bcmills modified the milestones: Go1.12, Go1.13 Oct 25, 2018
@perillo
Copy link
Contributor

perillo commented Mar 4, 2019

shouldn't the module cache be in os.UserCacheDir() instead of $GOPATH/pkg/mod?

@bradfitz
Copy link
Contributor Author

bradfitz commented Mar 4, 2019

@perillo, we discussed that (somewhere) earlier and decided to just use $GOPATH/pkg for now while we had it. But yes, it'd have to move later when this bug is resolved.

@rsc
Copy link
Contributor

rsc commented Mar 6, 2019

@perillo, or we might leave the module source cache in $GOPATH/pkg.

I think there are good reasons not to put cached module downloads in $HOME/.cache. Quoting myself on https://dave.cheney.net/2018/07/14/taking-go-modules-for-a-spin and https://groups.google.com/forum/#!msg/golang-dev/RjSj4bGSmsw/KMHhU8fmAwAJ, back when the module cache was $GOPATH/src/mod not $GOPATH/pkg/mod:

The build cache ($GOCACHE, defaulting to $HOME/.cache/go-build) is for storing recent compilation results, so that if you need to do that exact compilation again, you can just reuse the file. The build cache holds entries that are like “if you run this exact compiler on these exact inputs. this is the output you’d get.” If the answer is not in the cache, your build uses a little more CPU to run the compiler instead of reusing the output. But you are guaranteed to be able to run the compiler instead, since you have the exact inputs and the compiler binary (or else you couldn’t even look up the answer in the cache).

The module cache ($GOPATH/src/mod, defaulting to $HOME/go/src/mod) is for storing downloaded source code, so that every build does not redownload the same code and does not require the network or the original code to be available. The module cache holds entries that are like “if you need to download mymodule@v1.2.3, here are the files you’d get.” If the answer is not in the cache, you have to go out to the network. Maybe you don’t have a network right now. Maybe the code has been deleted. It’s not anywhere near guaranteed that you can redownload the sources and also get the same result. Hopefully you can, but it’s not an absolute certainty like for the build cache. (The go.sum file will detect if you get a different answer on re-download, but knowing you got the wrong bits doesn’t help you make progress on actually building your code. Also these paths end up in file-line information in binaries, so they show up in stack traces, and the like and feed into tools like text editors or debuggers that don’t necessarily know how to trigger the right cache refresh.)

I think that's still basically right. If the module cache is the only thing left in $GOPATH/pkg, we could decide to move it elsewhere, or we could decide to leave it there. Moving it to $HOME/.cache seems wrong though.

Really I'm just restating what I said in b8f42d7 on Jul 30, 2018 above:

If we do eliminate GOPATH/pkg for holding .a files (#4719)
then we could still keep it around for pkg/mod.
(Or we could move the module cache again then.)

@vith
Copy link

vith commented Mar 6, 2019

@rsc I think I understand and agree with all those points individually, but I'm missing the link between them and being opposed to using $XDG_CACHE_HOME (default $HOME/.cache) or its equivalent on other platforms.

My motivation for wanting to use a platform's standard cache location is that these caches tend to grow and never shrink, and eventually it becomes necessary to reclaim some disk space. Having a single location where all "well behaved" software stores caches that are safe to delete makes it more convenient to do that kind of maintenance.

Without standards like the XDG base directory spec, most software tends to think it deserves its own entry in your ls -a ~ output. But the more software you use, the more of a mess that becomes.

I suspect we have a different level of concern around this point:

It’s not anywhere near guaranteed that you can redownload the sources and also get the same result.

I don't think ~/.cache is meant to be limited to things that could be recomputed from locally available non-cache data. Looking inside mine, I see multiple browser caches, yarn module cache, typescript typing cache, a system package manager cache (yay), etc. All of those are caching data from the internet.

Looking at the XDG data dir spec, if a distinction must be made between the durability of the two go caches, maybe $XDG_DATA_HOME ($HOME/.local/share) should be considered.

@tv42
Copy link

tv42 commented Mar 6, 2019

I do get rsc's point, some of Go cache is "cheap to discard", some is "expensive to discard and potentially unavailable if lost". Note that XDG says nothing about how $XDG_CACHE_HOME should or can be cleaned, or how often etc, all it says is the directory is for "non-essential (cached) data". So I guess you could argue that any automatic tool that removes ~/.cache/go/mod or whatever in way that annoys the user is just too eager, and should learn better rules about prioritization and retainment. Maybe.

For what it's worth, rust seems to plan migrating from ~/.cargo to ~/.cache/cargo for what seem to me to be similar uses: rust-lang/cargo#148 (don't trust Github claims of "Merged").

@mathstuf
Copy link

mathstuf commented Mar 6, 2019

The other thing a single cache directory location is for is a simple place to exclude things from backups.

@flibustenet
Copy link

Why will we want a long cache when there will be global modules proxies ?
I think if we want to be really safe we need a cache per project (instead of vendor ?). I'd like this cache hardlinked on linux.

@rsc
Copy link
Contributor

rsc commented Nov 11, 2022

This issue was originally about the .a files in $GOPATH/pkg.

Those are no longer generated when using the build cache,
and as of Go 1.12, use of the build cache is required.
So it has been quite a long time since we wrote .a files to $GOPATH/pkg.
(The build cache also fixed issue #4443 and #3895 that Brad mentioned in the issue text.)

Since the issue was written, we added $GOPATH/pkg/mod and $GOPATH/pkg/sumdb.
I am comfortable with those staying as they are today.
They don't have any of the problems that the .a files had.
Now that GOPATH defaults to $HOME/go,
$HOME/go/pkg/mod and $HOME/go/pkg/sumdb
are completely reasonable paths.
(I still prefer to set GOPATH=$HOME, but that's a choice not a default.)

Closing this issue as fixed by the now-required build cache.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests