dl: golang.org/dl installer shouldn't create a new subdirectory of $HOME

[RalphCorderoy]

@andybons sent an email to golang-dev yesterday, subject 'Go 1.11 Beta 2 is Released'.
The instructions included

If you have Go installed already, the easiest way to try go1.11beta2
is by installing it using the go command:
$ go get https://golang.org/dl/go1.11beta2
$ go1.11beta2 download

These fail with Arch Linux's go 2:1.10.3-1 package.

$ go get https://golang.org/dl/go1.11beta2
package https:/golang.org/dl/go1.11beta2: "https://" not allowed in import path
$

Removing the `https://' worked. It then matches the comment in https://go.googlesource.com/dl/+/master/go1.11beta2/main.go

Please consider warning that 'go1.11beta2 download' will install to
$HOME/sdk/go1.11beta2 as some folks, me included, won't appreciate new
top-level $HOME directories. Altering $HOME for the run of 'go1.11beta2
download' avoided this on Unix, and it looks like similar is possible on other
platforms.

(Opened issue as requested by Andy in private email.)

[bcmills]

There are two separate issues here, I think. One is that the email template we used was wrong; I've updated our release docs with a fix. That leaves the install directory issue, for which I will retitle this issue.

[bcmills]

I agree that $HOME/sdk is a strange place to put the binary by default. @bradfitz chose it in https://golang.org/cl/34385, on the theory that many users already have a $HOME/sdk directory.

I do not have such a directory, and frankly I find it pretty surprising too: when I install per-language SDKs, I usually put them in ${HOME}/${NAME_OF_SDK}, like ~/go/, with symlinks in ~/bin if I don't want to include the whole install directory in $PATH.

I would argue that $GOBIN (default $GOPATH/bin) is a better default for Go users in general, since we need that directory to exist anyway and its use is already documented.¹

¹https://tip.golang.org/cmd/go/#hdr-GOPATH_environment_variable

(CC @andybons @FiloSottile @dmitshur)

[FiloSottile]

+1 on not liking $HOME/sdk, but I expect my $GOBIN to hold binaries, not whole trees. $GOPATH/sdk makes the most sense to me, even if it ends up being back to $HOME/sdk for me).

[bradfitz]

$GOPATH is going away, so let's not overload that and give it new reasons to exist.

[RalphCorderoy]

If go1.11beta2/bin/go and everything else don't care where they are, using argv[0] to hone in, then why not insist the user state the location with go1.11beta2 download $root.

[bradfitz]

@RalphCorderoy, this is supposed to be the easy way to install Go. No required arguments.

If people feel strongly that there be an optional flag to set the location, maybe. But then you have to store its location somewhere in a known location (probably in $HOME/.config, for which we don't have a helper like https://tip.golang.org/pkg/os/#UserCacheDir) so the "go1.N" binary can find it.

So I'm kinda meh on this bug. The tool is opinionated for simplicity. If you don't like its opinions, install it the traditional way wherever you like? shrug

[bcmills]

Perhaps we should use $XDG_DATA_HOME¹ instead?

That seems to be the standard place to stuff long-lived data files that the user otherwise doesn't want to think about.

¹https://specifications.freedesktop.org/basedir-spec/latest/ar01s03.html

[bradfitz]

And for Mac and Windows? $HOME/.local/share isn't idiomatic on those platforms. Mac would be ~/Library/ something. And Windows surely has something else.

But even on Unix, IMO it's kinda gross to put large files in dot directories. But I suppose we already do that for ~/.cache. I suppose I'm neutral on XDG_DATA_HOME.

[bcmills]

IMO it's kinda gross to put large files in dot directories

I would argue that it's kinda gross to put large files any place the user didn't specify explicitly, except perhaps a subdirectory of the current working directory (which go1.11beta2 download presumably can't use because that might be inside $GOPATH/src). But maybe I'm old-fashioned that way?

And Windows surely has something else.

Presumably %ProgramFiles%\go1.N, or perhaps %APPDATA%\Local\go1.N? (@alexbrainman might have a suggestion.)

[alexbrainman]

I am not really an expert in this area, but

Presumably %ProgramFiles%\go1.N,

I would say that normal users are not allow to write there. So you need to be admin to be able to do that.

or perhaps %APPDATA%\Local\go1.N

Maybe. Except I do not have %APPDATA%\Local directory on my computer:

c:\>set | grep appdata -i
APPDATA=C:\Users\Alex\AppData\Roaming
LOCALAPPDATA=C:\Users\Alex\AppData\Local
...
c:\>

and

c:\>dir %APPDATA%
 Volume in drive C has no label.
 Volume Serial Number is 9012-A870

 Directory of C:\Users\Alex\AppData\Roaming

13/05/2018  11:44 AM    <DIR>          .
13/05/2018  11:44 AM    <DIR>          ..
23/06/2017  06:43 PM    <DIR>          Adobe
11/02/2018  04:51 PM    <DIR>          Docker
02/07/2017  01:26 PM    <DIR>          Fuji Xerox
23/06/2017  06:59 PM    <DIR>          Google
23/09/2017  10:25 AM    <DIR>          Macromedia
29/08/2017  05:15 PM    <DIR>          Notepad++
23/06/2017  06:49 PM    <DIR>          Skype
...

Alex

[RalphCorderoy]

Regarding $XDG_DATA_HOME, it's analogous to /usr/share, thus its default of $HOME/.local/share, and things reading from it are meant to also continue the search through the colon-separated paths in $XDG_DATA_DIRS. My $XDG_DATA_HOME does have a few executables, typically 'contrib' scripts.

The 612 MiB go1.11beta2 dwarfs the 1-2 MiB directories in $XDG_DATA_HOME here, though size isn't a criteria in the spec. It's content doesn't change, but I suppose it's not a .cache because work is required to bring it back after delete.

The audience is small. Would an optional $GOSDK to override the use of homedir()/"sdk" in version.go be acceptable? (I'd have it set to outside the area that's backed up.)

[bradfitz]

Would an optional $GOSDK to override the use of homedir()/"sdk"

I'm neutral. I'll let others here decide.

[RalphCorderoy]

One more thought on $XDG_DATA_HOME, the user's version of /usr/share, http://www.pathname.com/fhs/pub/fhs-2.3.html#USRSHAREARCHITECTUREINDEPENDENTDATA is it seems to be for architecture-independent files, e.g. a $HOME can be mounted on various architectures, just like /usr/share, that's the reason the latter is called 'share'.

Rust describes it that way, https://docs.rs/xdg/2.0.0/xdg/struct.BaseDirectories.html and the executables under mine are scripts rather than ELFs.

[mvdan]

I too find it surprising that my home directory was polluted. I agree that we shouldn't tie it to GOPATH directly. GOCACHE (the build cache) exists so one can throw it away whenever, as rebuilding it tends to not be too expensive. So anywhere under GOCACHE is likely not a good idea either.

cmd/go already has a solution for this - the directory where the module cache is placed. Right now, that happens to be GOPATH/pkg/mod, but I presume it would get moved somewhere else once GOPATH is gone.

Why not put the release download cache there too? For example, we already have pkg/mod/cache/vcs, so one could imagine pkg/mod/cache/sdk. This might require x/dl to be able to ask cmd/go something like "where is the module cache?", but I presume we can make that work.

The only other complication is that the sdk download cache might get large, and it would be somewhat hidden. A solution could be for each goX.Y wrapper program to have a very simple goX.Y uninstall command, which would delete the cached SDK, and perhaps its own binary too.

[bradfitz]

In 2018 I wrote:

$GOPATH is going away, so let's not overload that and give it new reasons to exist.

At the time it wasn't clear whether Go 1.8's implicit GOPATH was going to stick around.

But it has stuck around (or better or worse) and still works, and it's used heavily for the module cache:

$ echo $GOPATH

$ go env GOPATH
/home/bradfitz/go
$ du -h $(go env GOPATH)/pkg/mod | tail -1
5.4G    /home/bradfitz/go/pkg/mod

So we could put it under $(go env GOPATH)/sdk or /dl or /version.

(what's another 100 MB or so drop in the bucket?)

@RalphCorderoy, @bcmills, @FiloSottile, @dmitshur, shall we move it from top-level ~/sdk/ to $(go env GOPATH)/something? We can make it still look at the old locations and use the old directory if it exists, but it won't write new directories there.

[dmitshur]

Something to consider, I often use goX.Y.Z commands at around the same time I'm doing some investigation where one of the things I do to have a clean environment for reproduction is export GOPATH=$(mktemp -d) (i.e., I don't want to reuse the module cache, etc.). It would be unfortunate not to be able to use an empty GOPATH directory for such testing.

[bcmills]

@dmitshur, I think even if we put those commands in some subdirectory of GOPATH you can still use a different GOPATH with them for testing.

(The go command looks for its own resources in GOROOT, not GOPATH, and presumably the go1.Y.Z binaries would continue to apply their own GOROOT settings like they do today.)

[RalphCorderoy]

I'd be fine with $(go env GOPATH)/something as it won't be a new $HOME directory, a user who doesn't want ~/go will already be setting $GOPATH, and it will be clear what's to blame for the space consumption.

[perillo]

My suggestion is to use ~/gosdk instead of ~/sdk as the default directory, and allowing the user to change the default directory using the GOSDK environment variable.

Please note that I'm not proposing to add support for GOSDK in the go tool.

[gopherbot]

Change https://golang.org/cl/310009 mentions this issue: dl: support user specified sdk directory

[DasSkelett]

I would highly appreciate if the downloader would put new releases in a subdirectory of the existing ~/go directory, instead of messing up my home directory with new folders that really don't fit there.

[alexdelorenzo]

I installed the Go 1.18 beta and ran into this issue. Being able to move the hardcoded ~/sdk directory would be a nice feature. Making it a subdirectory of $GOPATH feels like an intuitive solution.

[tydavis]

+1 to https://go-review.googlesource.com/c/dl/+/310009

@bradfitz @bcmills can you perform a review of the changeset from @perillo

[RalphCorderoy]

That old perillo changeset allows a GOSDK environment variable to override the existing location. It's unclear that has been decided upon as the solution. Alternatives above include $(go env GOPATH)/something being the default in the first place rather than the savvy user overriding it. Given there are folks here who have been caught out by the home directory being polluted on their first go, learning there's an environment variable to override afterwards, say by properly reading the release note, seems like a bolted horse.

Changing the default doesn't have this problem and the alternative above has, as I said, the benefit of ‘a user who doesn't want ~/go will already be setting $GOPATH, and it will be clear what's to blame for the space consumption’.

[gopherbot]

Change https://go.dev/cl/439215 mentions this issue: internal/version: download versions to gopath