cmd/go: drop $GOPATH/pkg

[bradfitz]

(after discussion with Russ, trying to summarize here)

It's nice to be able to tell people to set $GOPATH == $HOME, which means their source
goes into $HOME/src and their binaries conveniently go into $HOME/bin.

The only wart with $GOPATH == $HOME is the $HOME/pkg directory, which is pretty ugly,
not useful for end users, and somewhat offensive to be littering in people's $HOME.

If issue #4443 and issue #3895 get fixed, $GOPATH/pkg might get even uglier.

Unlike $GOROOT/pkg, $GOPATH/pkg doesn't contain tool binaries.  It's pretty much just a
cache.

Can't cmd/go's *.a cache go into /tmp/go-cache-$USER 0700 or ~/Library/Caches with
appropriate occasional lazy cleaning?  (or letting the system clean).

[adg]

Comment 1:

Do you propose to change the behavior for everyone, or make this configurable?
What about people that want to ship .a binaries only? I think the go tool currently
supports that. (ie, just mkdir the src/ path and put the binaries under pkg/ and it
works)

[adg]

Comment 1:

Do you propose to change the behavior for everyone, or make this configurable?
What about people that want to ship .a binaries only? I think the go tool currently
supports that. (ie, just mkdir the src/ path and put the binaries under pkg/ and it
works)

[minux]

Comment 2:

FYI, for binary-only package to work, you will need to create a dummy "package xxx" Go
source
file in the directory with an older timestamp than the .a file.
This is issue #2775.

[minux]

Comment 2:

FYI, for binary-only package to work, you will need to create a dummy "package xxx" Go
source
file in the directory with an older timestamp than the .a file.
This is issue #2775.

[rsc]

Comment 3:

There are many facets to think about. This is just an issue to remind us to
do so.

[rsc]

Comment 3:

There are many facets to think about. This is just an issue to remind us to
do so.

[rsc]

Comment 4:

Labels changed: added priority-later, go1.1maybe, removed priority-triage, go1.1.

Status changed to Thinking.

[rsc]

Comment 4:

Labels changed: added priority-later, go1.1maybe, removed priority-triage, go1.1.

Status changed to Thinking.

[robpike]

Comment 5:

Labels changed: removed go1.1maybe.

[robpike]

Comment 5:

Labels changed: removed go1.1maybe.

[rsc]

Comment 6:

Labels changed: added go1.3.

[rsc]

Comment 6:

Labels changed: added go1.3.

[robpike]

Comment 7:

Labels changed: removed go1.3.

[robpike]

Comment 7:

Labels changed: removed go1.3.

[rsc]

Comment 8:

Labels changed: added go1.3maybe.

[rsc]

Comment 8:

Labels changed: added go1.3maybe.

[rsc]

Comment 9:

Labels changed: added release-none, removed go1.3maybe.

[rsc]

Comment 9:

Labels changed: added release-none, removed go1.3maybe.

[rsc]

Comment 10:

Labels changed: added repo-main.

[rsc]

Comment 10:

Labels changed: added repo-main.

[bradfitz]

Every release (especially Go 1.5) has been getting us closer to eliminating "pkg" and instead having a build cache directory somewhere (some system-specific cache directory)

Having a cache instead of "pkg" would also eliminate the frequent problems people run into where "go build" does too much work when they could've used "go install" instead.

/cc @adg @davecheney

[bradfitz]

Every release (especially Go 1.5) has been getting us closer to eliminating "pkg" and instead having a build cache directory somewhere (some system-specific cache directory)

Having a cache instead of "pkg" would also eliminate the frequent problems people run into where "go build" does too much work when they could've used "go install" instead.

/cc @adg @davecheney

[griesemer]

I would love see the pkg directory go away.

I've always viewed it as an artifact of our implementation which is based on historic ideas of how a program is compiled into object files which are then linked. (In the earliest days of Go I suggested that we make this compile/link process invisible - i.e., as a programmer one just "sees" source code. But we were not ready at that time for this).

[griesemer]

I would love see the pkg directory go away.

I've always viewed it as an artifact of our implementation which is based on historic ideas of how a program is compiled into object files which are then linked. (In the earliest days of Go I suggested that we make this compile/link process invisible - i.e., as a programmer one just "sees" source code. But we were not ready at that time for this).

[davecheney]

@bradfitz,

I view 'pkg' as this cache, and I'm fine with it living inside GOPATH
because that makes it simple to explain where to find it if people are
determined to delete it.

The change I implemented in gb was to remove the distinction between go
build, go install, and the other odd -I variants; if gb can cache
something, even while building tests, it will and this has worked out
pretty well.

For my money removing the number of ways one can tell them go tool to cache
things and just make the caching more pervasive would be a welcome
improvement.

If that cache lives inside GOPATH, or some default location relative to
HOME feels less important, but I suspect that the windows users will have a
issues with a large number of files written to their home directories,
which may not be local to their machines.

Thanks

Dave

On Wed, 30 Sep 2015 03:17 Robert Griesemer notifications@github.com wrote:

I would love see the pkg directory go away.

I've always viewed it as an artifact of our implementation which is based
on historic ideas of how a program is compiled into object files which are
then linked. (In the earliest days of Go I suggested that we make this
compile/link process invisible - i.e., as a programmer one just "sees"
source code. But we were not ready at that time for this).

—
Reply to this email directly or view it on GitHub
#4719 (comment).

[davecheney]

@bradfitz,

I view 'pkg' as this cache, and I'm fine with it living inside GOPATH
because that makes it simple to explain where to find it if people are
determined to delete it.

The change I implemented in gb was to remove the distinction between go
build, go install, and the other odd -I variants; if gb can cache
something, even while building tests, it will and this has worked out
pretty well.

For my money removing the number of ways one can tell them go tool to cache
things and just make the caching more pervasive would be a welcome
improvement.

If that cache lives inside GOPATH, or some default location relative to
HOME feels less important, but I suspect that the windows users will have a
issues with a large number of files written to their home directories,
which may not be local to their machines.

Thanks

Dave

On Wed, 30 Sep 2015 03:17 Robert Griesemer notifications@github.com wrote:

I would love see the pkg directory go away.

I've always viewed it as an artifact of our implementation which is based
on historic ideas of how a program is compiled into object files which are
then linked. (In the earliest days of Go I suggested that we make this
compile/link process invisible - i.e., as a programmer one just "sees"
source code. But we were not ready at that time for this).

—
Reply to this email directly or view it on GitHub
#4719 (comment).

[bradfitz]

makes it simple to explain where to find it if people are
determined to delete it

If the cache is 100% correct (which isn't unimaginable; bazel does it, and Go 1.5 is basically there already), then deleting it is unnecessary. If we really wanted, we could have a go XXX subcommand to free up disk space, if that's the concern.

Great to hear that gb does caching for both build and install!

If we get to the point where the current "pkg" is just a cache and also cached go build, that's great, but then my next annoyance is that I have to look at it. I like to set my $GOPATH to $HOME, and that means I have $HOME/pkg being a useless turd right in my home directory. So I'd rather put things in the right spots:

• For Linux etc we have stuff like http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html for the user-specific cache directory.
• OS X has $HOME/Library/Caches
• Windows has %USERPROFILE%\AppData\Local\Temp (aka $TMP aka $TEMP) (from http://technet.microsoft.com/en-us/library/cc749104(v=ws.10).aspx

[bradfitz]

makes it simple to explain where to find it if people are
determined to delete it

If the cache is 100% correct (which isn't unimaginable; bazel does it, and Go 1.5 is basically there already), then deleting it is unnecessary. If we really wanted, we could have a go XXX subcommand to free up disk space, if that's the concern.

Great to hear that gb does caching for both build and install!

If we get to the point where the current "pkg" is just a cache and also cached go build, that's great, but then my next annoyance is that I have to look at it. I like to set my $GOPATH to $HOME, and that means I have $HOME/pkg being a useless turd right in my home directory. So I'd rather put things in the right spots:

• For Linux etc we have stuff like http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html for the user-specific cache directory.
• OS X has $HOME/Library/Caches
• Windows has %USERPROFILE%\AppData\Local\Temp (aka $TMP aka $TEMP) (from http://technet.microsoft.com/en-us/library/cc749104(v=ws.10).aspx

[davecheney]

I cannot explain why people want to delete GOPATH/pkg, I believe it is
unnecessary, and in every case where I have asked for more details not have
been provided, yet the lore continues to propagate itself.

I'm fine with using the LSB mandates tmp directories, that feels like the
smaller argument vs making the go tool more aggressive with its caching.

On Wed, 30 Sep 2015 07:50 Brad Fitzpatrick notifications@github.com wrote:

makes it simple to explain where to find it if people are
determined to delete it

If the cache is 100% correct (which isn't unimaginable; bazel does it, and
Go 1.5 is basically there already), then deleting it is unnecessary. If we
really wanted, we could have a go XXX subcommand to free up disk space,
if that's the concern.

Great to hear that gb does caching for both build and install!

If we get to the point where the current "pkg" is just a cache and also
cached go build, that's great, but then my next annoyance is that I have
to look at it. I like to set my $GOPATH to $HOME, and that means I have
$HOME/pkg being a useless turd right in my home directory. So I'd rather
put things in the right spots:

• For Linux etc we have stuff like
  http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html
  for the user-specific cache directory.
• OS X has $HOME/Library/Caches
• Windows has %USERPROFILE%\AppData\Local\Temp (aka $TMP aka $TEMP)
  (from http://technet.microsoft.com/en-us/library/cc749104(v=ws.10).aspx

—
Reply to this email directly or view it on GitHub
#4719 (comment).

[davecheney]

I cannot explain why people want to delete GOPATH/pkg, I believe it is
unnecessary, and in every case where I have asked for more details not have
been provided, yet the lore continues to propagate itself.

I'm fine with using the LSB mandates tmp directories, that feels like the
smaller argument vs making the go tool more aggressive with its caching.

On Wed, 30 Sep 2015 07:50 Brad Fitzpatrick notifications@github.com wrote:

makes it simple to explain where to find it if people are
determined to delete it

If the cache is 100% correct (which isn't unimaginable; bazel does it, and
Go 1.5 is basically there already), then deleting it is unnecessary. If we
really wanted, we could have a go XXX subcommand to free up disk space,
if that's the concern.

Great to hear that gb does caching for both build and install!

If we get to the point where the current "pkg" is just a cache and also
cached go build, that's great, but then my next annoyance is that I have
to look at it. I like to set my $GOPATH to $HOME, and that means I have
$HOME/pkg being a useless turd right in my home directory. So I'd rather
put things in the right spots:

• For Linux etc we have stuff like
  http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html
  for the user-specific cache directory.
• OS X has $HOME/Library/Caches
• Windows has %USERPROFILE%\AppData\Local\Temp (aka $TMP aka $TEMP)
  (from http://technet.microsoft.com/en-us/library/cc749104(v=ws.10).aspx

—
Reply to this email directly or view it on GitHub
#4719 (comment).

[mwhudson]

If pkg is to become a cache, and possibly more hidden, it would be good to fix the ways in which GOOS + GOARCH is not quite a valid cache key: GOARM is probably the worst offender here, but also tags and things like buildmodes and the race detector (although the go tool handles the last two via installstuffix currently).

[mwhudson]

If pkg is to become a cache, and possibly more hidden, it would be good to fix the ways in which GOOS + GOARCH is not quite a valid cache key: GOARM is probably the worst offender here, but also tags and things like buildmodes and the race detector (although the go tool handles the last two via installstuffix currently).

[minux]

Yeah, GOARM/GO386 and build tags (including race) should also be part of the cache key (ideally, build tag that doesn't affect the compilation of one package shouldn't be cached under the cache with the build tag.) The toolchain version should also be part of the cache key. Esp. for those who switch Go releases frequently.

[minux]

Yeah, GOARM/GO386 and build tags (including race) should also be part of the cache key (ideally, build tag that doesn't affect the compilation of one package shouldn't be cached under the cache with the build tag.) The toolchain version should also be part of the cache key. Esp. for those who switch Go releases frequently.

[ChrisHines]

Forgive me if I have missed subtleties that have already been discussed.

I have not seen any discussion of how a system global pkg cache would work for people who switch between multiple GOPATHs, each of which might contain different versions of the same package(s). I understand that recent versions of the go tool will hash all the file names used to build a .a file to help keep the cache accurate. But different versions of a package in different GOPATHs could easily have the same file list but different code. It seems to me either separate caches are required per GOPATH (as the pkg directory does today) or the cache key would need to include something to differentiate between the multiple GOPATHs used.

[ChrisHines]

Forgive me if I have missed subtleties that have already been discussed.

I have not seen any discussion of how a system global pkg cache would work for people who switch between multiple GOPATHs, each of which might contain different versions of the same package(s). I understand that recent versions of the go tool will hash all the file names used to build a .a file to help keep the cache accurate. But different versions of a package in different GOPATHs could easily have the same file list but different code. It seems to me either separate caches are required per GOPATH (as the pkg directory does today) or the cache key would need to include something to differentiate between the multiple GOPATHs used.

[rsc]

Yes, it would have to be a good cache. :-)

[rsc]

Yes, it would have to be a good cache. :-)

[rsc]

(That is, it would have to take all these things into consideration. It's not a trivial problem, which is why it hasn't happened.)

[rsc]

(That is, it would have to take all these things into consideration. It's not a trivial problem, which is why it hasn't happened.)

[rogpeppe]

+1 to this, particularly making it toolchain-sensitive too. I often find myself switching between different compiler versions so I can check behaviour differences, and switching/removing $GOPATH/pkg is a pain.

[rogpeppe]

+1 to this, particularly making it toolchain-sensitive too. I often find myself switching between different compiler versions so I can check behaviour differences, and switching/removing $GOPATH/pkg is a pain.

[smyrman]

It would be nice if binary-only packages (#2775) still live in $GOPATH/pkg, even if the remaining files where to move to OS specific cache directories.

[smyrman]

It would be nice if binary-only packages (#2775) still live in $GOPATH/pkg, even if the remaining files where to move to OS specific cache directories.

[gopherbot]

Change https://golang.org/cl/75473 mentions this issue: cmd/go: cache built packages

[gopherbot]

Change https://golang.org/cl/75473 mentions this issue: cmd/go: cache built packages

[gopherbot]

Change https://golang.org/cl/75850 mentions this issue: cmd/go: do not install dependencies during "go install"

[gopherbot]

Change https://golang.org/cl/75850 mentions this issue: cmd/go: do not install dependencies during "go install"

[rsc]

I've been mentally translating this bug to "remove pkg from both $GOROOT and $GOPATH", which is difficult, but @crawshaw pointed out to me earlier today that it's only about $GOPATH. $GOROOT/pkg is difficult to kill because we want to keep allowing the building of programs with cgo-enabled package net from systems without a C compiler. We will probably have $GOROOT/pkg or an equivalent for quite some time. But $GOPATH/pkg doesn't have that concern.

Assuming build artifact caching goes well in Go 1.10, the only thing left in the way of removing $GOPATH/pkg would be updating source code analysis tools that want access to the .a files to know where to find them. We're working out how to do that, because we can see that proper package management is not going to fit into the usual $GOPATH/pkg structure either. I expect the answer will be in Go 1.11 and that we can use that release as a grace period in which old tools still work but people should convert to new tools that also work. If that happens, then we could possibly consider dropping $GOPATH/pkg in Go 1.12.

[rsc]

I've been mentally translating this bug to "remove pkg from both $GOROOT and $GOPATH", which is difficult, but @crawshaw pointed out to me earlier today that it's only about $GOPATH. $GOROOT/pkg is difficult to kill because we want to keep allowing the building of programs with cgo-enabled package net from systems without a C compiler. We will probably have $GOROOT/pkg or an equivalent for quite some time. But $GOPATH/pkg doesn't have that concern.

Assuming build artifact caching goes well in Go 1.10, the only thing left in the way of removing $GOPATH/pkg would be updating source code analysis tools that want access to the .a files to know where to find them. We're working out how to do that, because we can see that proper package management is not going to fit into the usual $GOPATH/pkg structure either. I expect the answer will be in Go 1.11 and that we can use that release as a grace period in which old tools still work but people should convert to new tools that also work. If that happens, then we could possibly consider dropping $GOPATH/pkg in Go 1.12.

[rsc]

Tentatively putting in Go 1.12. Note that this means disallowing GOCACHE=off.

[rsc]

Tentatively putting in Go 1.12. Note that this means disallowing GOCACHE=off.

[JeanMertz]

Note that this means disallowing GOCACHE=off

Just to double-check, and my use-case might not be the norm, but does this mean having no way whatsoever to disable caching, outside of deleting the cache itself?

Reason I ask is that I sometimes am working with randomly-failing integration tests that I want to run in a loop to have it trigger the error, since it doesn't trigger every time.

Right now, when working with such projects, I always export GOCACHE=off before I start testing, because I know one successful run won't mean the test works the second time, and so I want to keep triggering those random failures so that I can debug and squash them.

Again, this might be an edge-case, and I can change those loops to remove the cache before running, but I just wanted to put this use-case out here for consideration.

[JeanMertz]

Note that this means disallowing GOCACHE=off

Just to double-check, and my use-case might not be the norm, but does this mean having no way whatsoever to disable caching, outside of deleting the cache itself?

Reason I ask is that I sometimes am working with randomly-failing integration tests that I want to run in a loop to have it trigger the error, since it doesn't trigger every time.

Right now, when working with such projects, I always export GOCACHE=off before I start testing, because I know one successful run won't mean the test works the second time, and so I want to keep triggering those random failures so that I can debug and squash them.

Again, this might be an edge-case, and I can change those loops to remove the cache before running, but I just wanted to put this use-case out here for consideration.

[davecheney]

Do you want to disable build caching or test caching? The latter can be disabled with go test -count=1

…

On 6 May 2018, at 21:54, Jean Mertz ***@***.***> wrote: Note that this means disallowing GOCACHE=off Just to double-check, and my use-case might not be the norm, but does this mean having no way whatsoever to disable caching, outside of deleting the cache itself? Reason I ask is that I sometimes am working with randomly-failing integration tests that I want to run in a loop to have it trigger the error, since it doesn't trigger every time. Right now, when working with such projects, I always export GOCACHE=off before I start testing, because I know one successful run won't mean the test works the second time, and so I want to keep triggering those random failures so that I can debug and squash them. Again, this might be an edge-case, and I can change those loops to remove the cache before running, but I just wanted to put this use-case out here for consideration. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[davecheney]

Do you want to disable build caching or test caching? The latter can be disabled with go test -count=1

…

On 6 May 2018, at 21:54, Jean Mertz ***@***.***> wrote: Note that this means disallowing GOCACHE=off Just to double-check, and my use-case might not be the norm, but does this mean having no way whatsoever to disable caching, outside of deleting the cache itself? Reason I ask is that I sometimes am working with randomly-failing integration tests that I want to run in a loop to have it trigger the error, since it doesn't trigger every time. Right now, when working with such projects, I always export GOCACHE=off before I start testing, because I know one successful run won't mean the test works the second time, and so I want to keep triggering those random failures so that I can debug and squash them. Again, this might be an edge-case, and I can change those loops to remove the cache before running, but I just wanted to put this use-case out here for consideration. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[abitrolly]

I came here from #17271 (comment) and I am interested if dropping $GOPATH/pkg will allow me to have my dependency in vendor/dependency checkout without touching $GOPATH at all?

[abitrolly]

I came here from #17271 (comment) and I am interested if dropping $GOPATH/pkg will allow me to have my dependency in vendor/dependency checkout without touching $GOPATH at all?

[bradfitz]

@abitrolly, the bug you were at was the right place. This bug is more of a cleanup that's not very user-visible.

[bradfitz]

@abitrolly, the bug you were at was the right place. This bug is more of a cleanup that's not very user-visible.

[abitrolly]

Now that the previous bug is closed, does that mean that the feature with vendor imports without $GOPATH works already in 1.12?

[abitrolly]

Now that the previous bug is closed, does that mean that the feature with vendor imports without $GOPATH works already in 1.12?

[rajender]

@abitrolly minimal vendor support without GOPATH is being added as part of new modules feature to Go.11. Note Go.1.11 is not released yet. You can try the beta version.

[rajender]

@abitrolly minimal vendor support without GOPATH is being added as part of new modules feature to Go.11. Note Go.1.11 is not released yet. You can try the beta version.

[gopherbot]

Change https://golang.org/cl/126755 mentions this issue: cmd/go: move module cache from $GOPATH/src/mod to $GOPATH/pkg/mod

[gopherbot]

Change https://golang.org/cl/126755 mentions this issue: cmd/go: move module cache from $GOPATH/src/mod to $GOPATH/pkg/mod

[gopherbot]

Change https://golang.org/cl/129679 mentions this issue: doc/code: drop mentions of GOPATH/pkg directory

[gopherbot]

Change https://golang.org/cl/129679 mentions this issue: doc/code: drop mentions of GOPATH/pkg directory