New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: drop $GOPATH/pkg #4719

Open
bradfitz opened this Issue Jan 28, 2013 · 35 comments

Comments

Projects
None yet
@bradfitz
Member

bradfitz commented Jan 28, 2013

(after discussion with Russ, trying to summarize here)

It's nice to be able to tell people to set $GOPATH == $HOME, which means their source
goes into $HOME/src and their binaries conveniently go into $HOME/bin.

The only wart with $GOPATH == $HOME is the $HOME/pkg directory, which is pretty ugly,
not useful for end users, and somewhat offensive to be littering in people's $HOME.

If issue #4443 and issue #3895 get fixed, $GOPATH/pkg might get even uglier.

Unlike $GOROOT/pkg, $GOPATH/pkg doesn't contain tool binaries.  It's pretty much just a
cache.

Can't cmd/go's *.a cache go into /tmp/go-cache-$USER 0700 or ~/Library/Caches with
appropriate occasional lazy cleaning?  (or letting the system clean).
@adg

This comment has been minimized.

Contributor

adg commented Jan 29, 2013

Comment 1:

Do you propose to change the behavior for everyone, or make this configurable?
What about people that want to ship .a binaries only? I think the go tool currently
supports that. (ie, just mkdir the src/ path and put the binaries under pkg/ and it
works)
@minux

This comment has been minimized.

Member

minux commented Jan 29, 2013

Comment 2:

FYI, for binary-only package to work, you will need to create a dummy "package xxx" Go
source
file in the directory with an older timestamp than the .a file.
This is issue #2775.
@rsc

This comment has been minimized.

Contributor

rsc commented Jan 29, 2013

Comment 3:

There are many facets to think about. This is just an issue to remind us to
do so.
@rsc

This comment has been minimized.

Contributor

rsc commented Jan 30, 2013

Comment 4:

Labels changed: added priority-later, go1.1maybe, removed priority-triage, go1.1.

Status changed to Thinking.

@robpike

This comment has been minimized.

Contributor

robpike commented Mar 7, 2013

Comment 5:

Labels changed: removed go1.1maybe.

@rsc

This comment has been minimized.

Contributor

rsc commented Jul 30, 2013

Comment 6:

Labels changed: added go1.3.

@robpike

This comment has been minimized.

Contributor

robpike commented Aug 20, 2013

Comment 7:

Labels changed: removed go1.3.

@rsc

This comment has been minimized.

Contributor

rsc commented Nov 27, 2013

Comment 8:

Labels changed: added go1.3maybe.

@rsc

This comment has been minimized.

Contributor

rsc commented Dec 4, 2013

Comment 9:

Labels changed: added release-none, removed go1.3maybe.

@rsc

This comment has been minimized.

Contributor

rsc commented Dec 4, 2013

Comment 10:

Labels changed: added repo-main.

@rsc rsc added this to the Unplanned milestone Apr 10, 2015

@bradfitz

This comment has been minimized.

Member

bradfitz commented Sep 29, 2015

Every release (especially Go 1.5) has been getting us closer to eliminating "pkg" and instead having a build cache directory somewhere (some system-specific cache directory)

Having a cache instead of "pkg" would also eliminate the frequent problems people run into where "go build" does too much work when they could've used "go install" instead.

/cc @adg @davecheney

@griesemer

This comment has been minimized.

Contributor

griesemer commented Sep 29, 2015

I would love see the pkg directory go away.

I've always viewed it as an artifact of our implementation which is based on historic ideas of how a program is compiled into object files which are then linked. (In the earliest days of Go I suggested that we make this compile/link process invisible - i.e., as a programmer one just "sees" source code. But we were not ready at that time for this).

@davecheney

This comment has been minimized.

Contributor

davecheney commented Sep 29, 2015

@bradfitz,

I view 'pkg' as this cache, and I'm fine with it living inside GOPATH
because that makes it simple to explain where to find it if people are
determined to delete it.

The change I implemented in gb was to remove the distinction between go
build, go install, and the other odd -I variants; if gb can cache
something, even while building tests, it will and this has worked out
pretty well.

For my money removing the number of ways one can tell them go tool to cache
things and just make the caching more pervasive would be a welcome
improvement.

If that cache lives inside GOPATH, or some default location relative to
HOME feels less important, but I suspect that the windows users will have a
issues with a large number of files written to their home directories,
which may not be local to their machines.

Thanks

Dave

On Wed, 30 Sep 2015 03:17 Robert Griesemer notifications@github.com wrote:

I would love see the pkg directory go away.

I've always viewed it as an artifact of our implementation which is based
on historic ideas of how a program is compiled into object files which are
then linked. (In the earliest days of Go I suggested that we make this
compile/link process invisible - i.e., as a programmer one just "sees"
source code. But we were not ready at that time for this).


Reply to this email directly or view it on GitHub
#4719 (comment).

@bradfitz

This comment has been minimized.

Member

bradfitz commented Sep 29, 2015

makes it simple to explain where to find it if people are
determined to delete it

If the cache is 100% correct (which isn't unimaginable; bazel does it, and Go 1.5 is basically there already), then deleting it is unnecessary. If we really wanted, we could have a go XXX subcommand to free up disk space, if that's the concern.

Great to hear that gb does caching for both build and install!

If we get to the point where the current "pkg" is just a cache and also cached go build, that's great, but then my next annoyance is that I have to look at it. I like to set my $GOPATH to $HOME, and that means I have $HOME/pkg being a useless turd right in my home directory. So I'd rather put things in the right spots:

@davecheney

This comment has been minimized.

Contributor

davecheney commented Sep 29, 2015

I cannot explain why people want to delete GOPATH/pkg, I believe it is
unnecessary, and in every case where I have asked for more details not have
been provided, yet the lore continues to propagate itself.

I'm fine with using the LSB mandates tmp directories, that feels like the
smaller argument vs making the go tool more aggressive with its caching.

On Wed, 30 Sep 2015 07:50 Brad Fitzpatrick notifications@github.com wrote:

makes it simple to explain where to find it if people are
determined to delete it

If the cache is 100% correct (which isn't unimaginable; bazel does it, and
Go 1.5 is basically there already), then deleting it is unnecessary. If we
really wanted, we could have a go XXX subcommand to free up disk space,
if that's the concern.

Great to hear that gb does caching for both build and install!

If we get to the point where the current "pkg" is just a cache and also
cached go build, that's great, but then my next annoyance is that I have
to look at it. I like to set my $GOPATH to $HOME, and that means I have
$HOME/pkg being a useless turd right in my home directory. So I'd rather
put things in the right spots:


Reply to this email directly or view it on GitHub
#4719 (comment).

@mwhudson

This comment has been minimized.

Contributor

mwhudson commented Sep 30, 2015

If pkg is to become a cache, and possibly more hidden, it would be good to fix the ways in which GOOS + GOARCH is not quite a valid cache key: GOARM is probably the worst offender here, but also tags and things like buildmodes and the race detector (although the go tool handles the last two via installstuffix currently).

@minux

This comment has been minimized.

Member

minux commented Sep 30, 2015

@ChrisHines

This comment has been minimized.

Contributor

ChrisHines commented Nov 13, 2015

Forgive me if I have missed subtleties that have already been discussed.

I have not seen any discussion of how a system global pkg cache would work for people who switch between multiple GOPATHs, each of which might contain different versions of the same package(s). I understand that recent versions of the go tool will hash all the file names used to build a .a file to help keep the cache accurate. But different versions of a package in different GOPATHs could easily have the same file list but different code. It seems to me either separate caches are required per GOPATH (as the pkg directory does today) or the cache key would need to include something to differentiate between the multiple GOPATHs used.

@rsc

This comment has been minimized.

Contributor

rsc commented Nov 13, 2015

Yes, it would have to be a good cache. :-)

@rsc

This comment has been minimized.

Contributor

rsc commented Nov 13, 2015

(That is, it would have to take all these things into consideration. It's not a trivial problem, which is why it hasn't happened.)

@rogpeppe

This comment has been minimized.

Contributor

rogpeppe commented Dec 15, 2015

+1 to this, particularly making it toolchain-sensitive too. I often find myself switching between different compiler versions so I can check behaviour differences, and switching/removing $GOPATH/pkg is a pain.

@gopherbot

This comment has been minimized.

gopherbot commented Nov 2, 2017

Change https://golang.org/cl/75473 mentions this issue: cmd/go: cache built packages

gopherbot pushed a commit that referenced this issue Nov 3, 2017

cmd/go: cache built packages
This CL adds caching of built package files in $GOCACHE, so that
a second build with a particular configuration will be able to reuse
the work done in the first build of that configuration, even if the
first build was only "go build" and not "go install", or even if there
was an intervening "go install" that wiped out the installed copy of
the first build.

The benchjuju benchmark runs go build on a specific revision of jujud 10 times.

Before this CL:

	102.72u 15.29s 21.98r 	 go build -o /tmp/jujud github.com/juju/juju/cmd/jujud ...
	105.99u 15.55s 22.71r 	 go build -o /tmp/jujud github.com/juju/juju/cmd/jujud ...
	106.49u 15.70s 22.82r 	 go build -o /tmp/jujud github.com/juju/juju/cmd/jujud ...
	107.09u 15.72s 22.94r 	 go build -o /tmp/jujud github.com/juju/juju/cmd/jujud ...
	108.19u 15.85s 22.78r 	 go build -o /tmp/jujud github.com/juju/juju/cmd/jujud ...
	108.92u 16.00s 23.02r 	 go build -o /tmp/jujud github.com/juju/juju/cmd/jujud ...
	109.25u 15.82s 23.05r 	 go build -o /tmp/jujud github.com/juju/juju/cmd/jujud ...
	109.57u 15.96s 23.11r 	 go build -o /tmp/jujud github.com/juju/juju/cmd/jujud ...
	109.86u 15.97s 23.17r 	 go build -o /tmp/jujud github.com/juju/juju/cmd/jujud ...
	110.50u 16.05s 23.37r 	 go build -o /tmp/jujud github.com/juju/juju/cmd/jujud ...

After this CL:

	113.66u 17.00s 24.17r 	 go build -o /tmp/jujud github.com/juju/juju/cmd/jujud ...
	3.85u 0.68s 3.49r 	 go build -o /tmp/jujud github.com/juju/juju/cmd/jujud ...
	3.98u 0.72s 3.63r 	 go build -o /tmp/jujud github.com/juju/juju/cmd/jujud ...
	4.07u 0.72s 3.57r 	 go build -o /tmp/jujud github.com/juju/juju/cmd/jujud ...
	3.98u 0.70s 3.43r 	 go build -o /tmp/jujud github.com/juju/juju/cmd/jujud ...
	4.58u 0.70s 3.58r 	 go build -o /tmp/jujud github.com/juju/juju/cmd/jujud ...
	3.90u 0.70s 3.46r 	 go build -o /tmp/jujud github.com/juju/juju/cmd/jujud ...
	3.85u 0.71s 3.52r 	 go build -o /tmp/jujud github.com/juju/juju/cmd/jujud ...
	3.70u 0.69s 3.64r 	 go build -o /tmp/jujud github.com/juju/juju/cmd/jujud ...
	3.79u 0.68s 3.41r 	 go build -o /tmp/jujud github.com/juju/juju/cmd/jujud ...

This CL reduces the overall all.bash time from 4m22s to 4m17s on my laptop.
Not much faster, but also not slower.

See also #4719, #20137, #20372.

Change-Id: I101d5363f8c55bf4825167a5f6954862739bf000
Reviewed-on: https://go-review.googlesource.com/75473
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: David Crawshaw <crawshaw@golang.org>
@gopherbot

This comment has been minimized.

gopherbot commented Nov 3, 2017

Change https://golang.org/cl/75850 mentions this issue: cmd/go: do not install dependencies during "go install"

gopherbot pushed a commit that referenced this issue Nov 3, 2017

cmd/go: do not install dependencies during "go install"
This CL makes "go install" behave the way many users expect:
install only the things named on the command line.
Future builds still run as fast, thanks to the new build cache (CL 75473).
To install dependencies as well (the old behavior), use "go install -i".

Actual definitions aside, what most users know and expect of "go install"
is that (1) it installs what you asked, and (2) it's fast, unlike "go build".
It was fast because it installed dependencies, but installing dependencies
confused users repeatedly (see for example #5065, #6424, #10998, #12329,
"go build" and "go test" so that they could be "fast" too, but that only
created new opportunities for confusion. We also had to add -installsuffix
and then -pkgdir, to allow "fast" even when dependencies could not be
installed in the usual place.

The recent introduction of precise content-based staleness logic means that
the go command detects the need for rebuilding packages more often than it
used to, with the consequence that "go install" rebuilds and reinstalls
dependencies more than it used to. This will create more new opportunities
for confusion and will certainly lead to more issues filed like the ones
listed above.

CL 75743 introduced a build cache, separate from the install locations.
That cache makes all operations equally incremental and fast, whether or
not the operation is "install" or "build", and whether or not "-i" is used.

Installing dependencies is no longer necessary for speed, it has confused
users in the past, and the more accurate rebuilds mean that it will confuse
users even more often in the future. This CL aims to end all that confusion
by not installing dependencies by default.

By analogy with "go build -i" and "go test -i", which still install
dependencies, this CL introduces "go install -i", which installs
dependencies in addition to the things named on the command line.

Fixes #5065.
Fixes #6424.
Fixes #10998.
Fixes #12329.
Fixes #18981.
Fixes #22469.

Another step toward #4719.

Change-Id: I3d7bc145c3a680e2f26416e182fa0dcf1e2a15e5
Reviewed-on: https://go-review.googlesource.com/75850
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: David Crawshaw <crawshaw@golang.org>
@rsc

This comment has been minimized.

Contributor

rsc commented Nov 4, 2017

I've been mentally translating this bug to "remove pkg from both $GOROOT and $GOPATH", which is difficult, but @crawshaw pointed out to me earlier today that it's only about $GOPATH. $GOROOT/pkg is difficult to kill because we want to keep allowing the building of programs with cgo-enabled package net from systems without a C compiler. We will probably have $GOROOT/pkg or an equivalent for quite some time. But $GOPATH/pkg doesn't have that concern.

Assuming build artifact caching goes well in Go 1.10, the only thing left in the way of removing $GOPATH/pkg would be updating source code analysis tools that want access to the .a files to know where to find them. We're working out how to do that, because we can see that proper package management is not going to fit into the usual $GOPATH/pkg structure either. I expect the answer will be in Go 1.11 and that we can use that release as a grace period in which old tools still work but people should convert to new tools that also work. If that happens, then we could possibly consider dropping $GOPATH/pkg in Go 1.12.

@rsc rsc changed the title from cmd/go: think about whether "pkg" needs to exist under $GOPATH to cmd/go: drop $GOPATH/pkg Apr 23, 2018

@rsc rsc modified the milestones: Unplanned, Go1.12 Apr 23, 2018

@rsc

This comment has been minimized.

Contributor

rsc commented Apr 23, 2018

Tentatively putting in Go 1.12. Note that this means disallowing GOCACHE=off.

@JeanMertz

This comment has been minimized.

JeanMertz commented May 6, 2018

Note that this means disallowing GOCACHE=off

Just to double-check, and my use-case might not be the norm, but does this mean having no way whatsoever to disable caching, outside of deleting the cache itself?

Reason I ask is that I sometimes am working with randomly-failing integration tests that I want to run in a loop to have it trigger the error, since it doesn't trigger every time.

Right now, when working with such projects, I always export GOCACHE=off before I start testing, because I know one successful run won't mean the test works the second time, and so I want to keep triggering those random failures so that I can debug and squash them.

Again, this might be an edge-case, and I can change those loops to remove the cache before running, but I just wanted to put this use-case out here for consideration.

@davecheney

This comment has been minimized.

Contributor

davecheney commented May 6, 2018

@abitrolly

This comment has been minimized.

abitrolly commented Jul 23, 2018

I came here from #17271 (comment) and I am interested if dropping $GOPATH/pkg will allow me to have my dependency in vendor/dependency checkout without touching $GOPATH at all?

@bradfitz

This comment has been minimized.

Member

bradfitz commented Jul 23, 2018

@abitrolly, the bug you were at was the right place. This bug is more of a cleanup that's not very user-visible.

@abitrolly

This comment has been minimized.

abitrolly commented Jul 23, 2018

Now that the previous bug is closed, does that mean that the feature with vendor imports without $GOPATH works already in 1.12?

@rajender

This comment has been minimized.

Contributor

rajender commented Jul 23, 2018

@abitrolly minimal vendor support without GOPATH is being added as part of new modules feature to Go.11. Note Go.1.11 is not released yet. You can try the beta version.

@gopherbot

This comment has been minimized.

gopherbot commented Jul 30, 2018

Change https://golang.org/cl/126755 mentions this issue: cmd/go: move module cache from $GOPATH/src/mod to $GOPATH/pkg/mod

gopherbot pushed a commit that referenced this issue Aug 1, 2018

cmd/go: move module cache from $GOPATH/src/mod to $GOPATH/pkg/mod
Using $GOPATH/src/mod confuses too many tools.
$GOPATH/pkg/mod seems better for now.
It's also next to dep's cache, $GOPATH/pkg/dep.
If we do eliminate GOPATH/pkg for holding .a files (#4719)
then we could still keep it around for pkg/mod.
(Or we could move the module cache again then.)

Fixes #26401.
Fixes #26635.

Change-Id: I18f7da216ed9f490eded3c00d837fb086ae5b6a4
Reviewed-on: https://go-review.googlesource.com/126755
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Rob Pike <r@golang.org>
@gopherbot

This comment has been minimized.

gopherbot commented Aug 17, 2018

Change https://golang.org/cl/129679 mentions this issue: doc/code: drop mentions of GOPATH/pkg directory

gopherbot pushed a commit that referenced this issue Aug 17, 2018

doc/code: drop mentions of GOPATH/pkg directory
It's already half gone and later will be all gone.
It's not worth explaining in an introduction doc.

Fixes #24506
Updates #4719

Change-Id: Ie48128b3aa090d84e0e734aa45f14a4480292913
Reviewed-on: https://go-review.googlesource.com/129679
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

@bcmills bcmills modified the milestones: Go1.12, Go1.13 Oct 25, 2018

komuw added a commit to komuw/go that referenced this issue Dec 4, 2018

doc/code: drop mentions of GOPATH/pkg directory
It's already half gone and later will be all gone.
It's not worth explaining in an introduction doc.

Fixes golang#24506
Updates golang#4719

Change-Id: Ie48128b3aa090d84e0e734aa45f14a4480292913
Reviewed-on: https://go-review.googlesource.com/129679
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment