proposal: cmd/go: add go env -w to set default env vars

[rsc]

Setting environment variables for go command configuration
is too difficult and system-specific.
We propose to add go env -w, to set defaults more easily.

See https://golang.org/design/30411-env.

(Note: When this was just a placeholder issue to get a number allocated for the design doc, the text here said only "mine all mine". The laughing emoji reactions are from that original text.)

[gopherbot]

Change https://golang.org/cl/164817 mentions this issue: design: add 30411-env.md

[mvdan]

(The "laugh" emojis were for the original "mine all mine" content)

Is there a reason why this has become relevant all of a sudden? I have seen a few junior developers have issues with environment variables, but I don't think the issue is severe enough to warrant adding the first per-user Go config file.

[josharian]

Two comments about fine details:

If the file contains multiple lines beginning with =, only the first has any effect.

May I suggest that later values override earlier? That’s how env vars work (see os/exec’s handling of duplicate env vars), and the implementation is clearer.

Lines with empty values set the default value to the empty string, possibly overriding a non-empty default.

I’d suggest omitting the second half of this sentence. When I first read it, in part due to context, I thought it was an exception to the “first entries” sentence immediately preceding it.

[josharian]

I don't think the issue is severe enough to warrant adding the first per-user Go config file.

FWIW, yesterday I was struggling with a Windows gomote. I am unfamiliar with windows and had to search for how to set env vars. And then was frustrated every time my ssh connection dropped, because I had to re-set my env vars. This proposal would have made my life considerably more pleasant.

[rsc]

@josharian, really duplicates should not happen, so this shouldn't matter at all, but it needs to be well-defined. The implementation of 'first wins' is quite clear: read the file until you get to the line you want, and stop. Yes, there's a clear implementation for 'last wins' too, but first seems like what most people would do when reading the file.

[rsc]

@mvdan, like the proverbial boiling frog, this has been bothering me for quite a while. It's not just junior developers. I personally have a hard time setting default environment variables. We've made go work very well out of the box but at the same time it's not uncommon to need to change a default here or there, and it's honestly just too hard. In Go 1.13 we expect to set a default GOPROXY, and we want to make sure it's easy to modify that setting. But really GOPROXY is just the latest of many straws, and the camel's back is already broken.

[josharian]

first seems like what most people would do when reading the file.

It’s not what I would do. :P I’ll take one more stab at this, and then let it drop. (I agree that either way it is not particularly critical.)

go env produces output that can be piped to a shell...in which case later overrides earlier. go env -w strikes me as having output that gets piped indirectly to a shell. So I would expect similar behavior.

[mvdan]

I agree with Josh that if the file should generally not have duplicates, it should follow the same logic as the env command and os/exec :) That's what most people would expect anyway, I think.

The reason I was asking about the urgency of this change for 1.13 was because I presumed that Go could be used by new developers without setting any env vars. Windows is a good point, I have to admit I don't know how to set an env var, though I hope I won't need to. The timing makes a lot more sense given your point about GOPROXY.

A way to summarize my concern with this proposal is that, right now, Go's behavior only changes via env vars and via PWD (modules). Adding a per-user config file adds another dimension to the problem, and I could imagine that resulting in confusion. Perhaps less confusion than setting env vars in a portable way, but still unfortunate.

[rsc]

@josharian, go env -w will never write a file with duplicate values. If a variable appears multiple times on the command line, that can just be an error.

[zombiezen]

I have a concern with go env only reading from os.UserConfigDir. The relevant part of the proposal is:

Only the go command will consult the os.UserConfigDir()/go/env file.

As discussed over on #29960, os.UserConfigDir() will likely only return a single directory. However, both XDG and macOS (and presumably other OSes) have well-established paths/mechanisms for having system-level configuration files for searching. I agree that go env -w should write to os.UserConfigDir()/go/env, but I think that go env should read from the whole set of configuration directory search paths.

This likely has overlap with @josharian's point about duplicate values. The configuration files should probably be visited in "reverse" order so that values in configuration directories for the user override configuration directories from the system.

[rsc]

@zombiezen, what problem are you trying to solve?

It is true that every problem can be solved by more indirections, but every indirection also creates new complexity (and often new problems). The problem I am trying to solve—not having a simple, portable way to set defaults for environment-based configuration—is addressed by one indirection; adding more adds complexity without making the solution any better. In fact it arguably makes it worse, because it is hard to understand, and we'd probably then need go env -system -w or whatever.

The host operating system can define whatever complexity it wants. Go is not, nor is it beholden to, the host operating system. Go is Go. It can define a single location. It's not like some general system utility is going to be working on this file on our behalf.

We have one GOCACHE, one GOBIN, one GOROOT, one GOTMPDIR, one GCCGOTOOLDIR. We do have multiple GOPATH, but that was arguably a mistake, and source code is quite different in kind from a few environment variables. Why do we need two or more places to store a list of environment variable settings? Again, what problem do you mean to solve by doing that?

[zombiezen]

Agreed: I don't want us to add complexity unnecessarily. I see this proposal as really valuable for a reason you have not cited yet: the go tool is increasingly invoked in a number of GUI contexts where it is really hard to set environment variables properly. Having the settings in a configuration file would solve these sorts of issues.

Here's the problem I'm trying to identify:

My experience in the past has been that for sysadmins or OS packagers, it is frequently useful to configure default variables at the system level (as illustrative example, these would be located at /etc/xdg/go/env for Linux or /Library/Preferences/go/env for macOS) when installing software on behalf of the users of the system. These are written/managed by hand or via configuration management like Puppet. Think of the same sort of circumstances where an /etc/profile or some such would exist.

I would not expect a go env -w -system to exist. Borrowing from XDG (which I reference the most because it is the most formal specification on the topic) emphatically says that writes should occur to os.UserConfigDir():

There is a single base directory relative to which user-specific configuration files should be written. This directory is defined by the environment variable $XDG_CONFIG_HOME.

A personal anecdote: I use the XDG_CONFIG_DIRS on my systems to have a version-controlled directory of "dot files" that is searched after the machine-specific non-version-controlled XDG_CONFIG_HOME directory. This is pretty useful for me and I believe not an uncommon setup for other developers (there's a whole Arch Wiki page on which programs support this). I feel I would be remiss if I didn't advocate for landing proper support.

[neild]

For the specific case of GOPROXY, I can easily imagine administrators at a company wanting to set a default across all developer workstations. Separate system and user configuration directories aren't the only way to achieve that, but they're a very simple one.

[rsc]

For what it's worth, admins at a company can already set default environment variables across all developer workstations. It's true that user configs then do not override those variables, but it lets them reuse an existing mechanism instead of us having to provide a separate one.

I think this proposal is worth doing with a single file. With the complexity of multiple files, it may not be.

[jimmyfrasche]

A go env -check, or similar, that reports duplicates and ignored lines in the config file would be quite useful for figuring out why things aren't working as expected when manually creating/editing the file or, for that matter, letting you know that you set something to an impossible value that's being ignored like GOMAXPROCS=bunny.

Since there are three places a setting may come from it would be useful to have something like go env -where KEY that reports default, config file, or environment.

If GOENV is not added, there should be a flag to output the location of the config file.

As for first wins vs. last wins, I'm vaguely aware that there are a number of config formats for specifying environment variables. What do they with duplicates? If there's an existing convention, it should be followed.

[davecheney]

Over the last five releases we've gotten to a point that a Go developer need not set any environment variables to develop a Go program on their machine. I don't think there needs to be a new mechanism, independent of the existing mechanisms to set environment variables on various operating systems, for operations which are now optional.

[dlsniper]

I echo the feelings of @davecheney here.

If this is needed just for how to set GOPROXY then maybe documentation should be used to teach people how to configure it in the help/documentation of GOPROXY on their operating system of choice.

[remeh]

I'm always a bit scared when we add a new feature doing things very similar to an existing one (I'm thinking of the system environment variables). It is one more layer of configuration that someone new to the language will probably have to apprehend or discover the hard way. Especially when there is subtilities such as (these are the first ones but others could appear):

The environment variables that control go libraries at runtime—for example, GODEBUG, GOMAXPROCS, and GOTRACEBACK—will not be read from go.env and will be rejected by go env command lines.

I'm maybe just worried about the fact that it looks (go env) like we're speaking about environment variables while we're in fact configuring Go. The layer is thin but it can be confusing.

[aarzilli]

If a variable is set by both the environment and go/env which one wins? I presume the environment, otherwise any program that calls go changing the environment is now broken.

There probably are quite a few bashrc out there with export GOSOMETHING lines, is it going to be weird that go env -w doesn't do anything in that case?

[ulikunitz]

As @davecheney I cannot see the actual need for the proposal and would like to see more evidence that this is a real world problem.

As @aarzilli I have a problem with the following command sequence.

$ export GOPATH=$HOME/aaa
$ go env -w GOPATH=$HOME/bbb
$ go env | grep GOPATH
GOPATH="/home/foo/aaa"

The proposal must also decide whether it is the go.env or the go/env file. Both names are used throughout the text.

[ulikunitz]

Rethinking it this looks more like changing the defaults. So following command sequence appears more logical. (I still have to see the need.)

$ go defaults -w GOPATH=$HOME/bbb
$ go defaults | grep GOPATH
GOPATH="/home/foo/bbb"
$ go env | grep GOPATH
GOPATH="/home/foo/bbb"
$ export GOPATH=$HOME/aaa
$ go env | grep GOPATH
GOPATH="/home/foo/aaa"

[thepudds]

If it would take N hours to design, discuss, implement, test, and document this feature, perhaps some subset of those N hours could instead be invested in a documentation-based approach with a mix of slightly dynamic help within the tool, coupled with something like a wiki for more details.

For example, if you are on Windows, perhaps go help env could conclude its output with something like:

On Windows, you can set a local environment variable in your command prompt
using the 'set' command, such as:

set GOPATH=C:\temp\gopath

See https://golang.org/setting-env-vars#Windows for more options, including 
how to persistently set environment variables on Windows for a specific user 
or system wide.

https://golang.org/setting-env-vars could redirect to a Wiki page. Perhaps someone from the core Go team could carve out something like 2-4 hours to put together a skeleton first version of that Wiki page, and there could be a tiny-ish outreach effort to the broader community to get other people to help populate it a bit more.

That usefulness of go help env for setting environmental variables could be briefly mentioned in a couple of additional introductory documents such as "How to Write Go Code" or "Getting Started", as well as places like the GOPROXY documentation.

For myself, if I was accessing a Plan 9 machine, rather than needing to remember that go env -w is useful, I would need to remember that go help env is useful. Perhaps that would be sufficient to help someone like @josharian when he is using gomote to access a Windows machine.

[thepudds]

If a wiki-based approach is pursued, some subset of the following could be briefly described for a given OS (either in the wiki itself, or via external reference):

1. how to set a local environment variable (e.g., export GOPATH=/tmp/gopath, set GOPATH=C:\temp\gopath)
2. how to have a local script that sets local environment variables (e.g., On Windows, something like notepad setup.bat, insert set GOPATH=C:\temp\gopath, Ctrl+S, then run setup.bat; or source setup.sh on Linux)
3. how to persistently set an environment variable via the command line
4. how to persistently set an environment in a more standard UI-based workflow

For some or maybe most OSes, only a subset of those would make sense to mention for a given OS.

For me personally, regardless of OS, I tend to favor approach 1 or 2 for a machine that is not "mine". For "my" machine, I still tend to favor 1 or 2 for project-based settings, but of course I also set some things persistently.

Taking the @josharian use case again, if he could go to a wiki page that briefly outlined approach 1, 2, and 3 for Windows, he could probably skim then pick whichever approach made sense for him at the time, including so that he doesn't have to start from scratch every time he reconnects. (E.g., I am not a Windows expert, but according to my first google hit, apparently the setx command on Windows allows making a persistent environment variable change from the Windows command line, although I will confess I have never used that command. Perhaps @josharian would pick setx if he had a short and easily accessible menu of choices).

If it is a wiki, it would probably be OK to reference external sources such as StackOverflow for more details or for anything more nuanced.

The greatest variability might be on Linux, but someone using Linux also tends to know how to sent env variables more frequently than, say, a randomly selected Windows user.

In any event, perhaps go env -w is a better approach, but wanted to at least sketch one possible alternative...

[myitcv]

In Go 1.13 we expect to set a default GOPROXY, and we want to make sure it's easy to modify that setting

Read in this context, i.e. "we need a reliable cross-platform one-liner that allows users to set a user-default value," I'm a big 👍 on this.

Imagine for one second a brand new, amazing modules proxy is announced, released etc: the usage instructions would be simple for all users on all systems. Not "please got to this wiki page and follow the hard-to-follow instructions on what to do on your OS, for your shell, etc, make sure you reload Terminal on Mac OS X etc". This is not newcomer-friendly.

Indeed I think the proposal could usefully be rewritten with GOPROXY as the main motivating example. Because as others have pointed out here, a great deal of work over the last N releases has got us to the great position where users do not need to set any environment variables (GOPATH, GOBIN etc) to get started - this is very newcomer friendly. GOPROXY however is a different sort of beast and if, as I understand it, setting GOPROXY is going to become more likely/necessary/frequent/etc independent of Go version, then having a reliable way to do so that covers a large percentage of Go users is a much better approach than regressing to asking people to follow instructions on how to set environment variables; that's the world we just moved away from.

A couple of other thoughts:

• there should be some warning/error if a user tries to set a variable via go env -w GOPROXY=https://my.amazing.proxy but the result would (in the current environment) be shadowed by an environment variable. i.e. the user has already gone to the effort of setting an environment variable. A -f "force" flag might be a step too far here (and awkward), but perhaps a warning is too weak?
• where people need, for example, per-project values of GOPROXY etc, this user-default will obviously not work and will in fact be more "racey" than updating one's .bashrc or equivalent. It might be worth documenting explicitly that environment variables are required in such a situation. Whether we then look to link to some good docs that suggest what/how to set per-project environment variables I'm unsure, because there are many permutations of setup/approach out there. I'll defer on exactly this point though. Maybe, as others have suggested, a page with the solid per-OS basics would suffice, plus a wiki with "alternatives" like smartcd, autoenv etc

[velovix]

I think this is a very good idea. Environment variables make sense in certain situations and I'm glad they're not going away, but I've found them to sometimes be a source of trouble as well. I've had experiences where certain programs that run an executable on my behalf do so in an environment other than the one I configured. I think people (including myself) set environment variables in their .bashrc or .zshrc and think they've globally configured the Go command to work this certain way. In actuality we've only configured it to work that way for our current user when the command is run with that specific shell.

I agree with @myitcv though that the killer feature here is easy cross-platform configuration.

where people need, for example, per-project values of GOPROXY etc, this user-default will obviously not work and will in fact be more "racey" than updating one's .bashrc or equivalent. It might be worth documenting explicitly that environment variables are required in such a situation.

I'm torn on this. Environment variables do well in situations where you're setting up a specific environment with a clear and small scope. For project-wide configuration, the scope becomes larger and things become less clear. An editor, the user's terminal, and any relevant scripts will all have to know about this project-specific configuration information. Where should users put it? I'm hesitant to recommend tools like smartcd because they only help in situations where you're running something in a terminal. They don't help editors unless the user "cd's" into their directory and runs the editor from the terminal with the necessary environment variables already applied.

[arschles]

@myitcv related to your comment at #30411 (comment), we haven't had much trouble with providing good docs on how to set GOPROXY for Athens. We have an end-to-end example here that includes running the server and configuring your client, and have heard seen very few issues related to misconfiguration so on.

Overall, I generally agree with @davecheney's comment that we don't need a new mechanism to set environment variables, with one small but important modification, that it would be a good idea to override a go-related environment variable in a file that is at the to level of a Go module. This was discussed in the context of a specific use case in #25530 (comment) (under "the light proposal")

[myitcv]

@arschles

we haven't had much trouble with providing good docs on how to set GOPROXY for Athens.

To my mind there is likely a significant difference in terms of experience/comfort with these sorts of things between your typical newcomer to Go and someone who is in the process of setting up Athens.

On that basis I think the point still stands: we're comparing a single command that will work for 99.9% of users with a link to a page that documents what steps to follow. It feels like a no-brainer to add go env -w to me, regardless of how easy-to-follow those steps might be, particularly if setting GOPROXY and friends is likely to become more necessary/common etc.

with one small but important modification

I totally get the use case for this, but I don't think it's worth it. There are existing mechanisms for having environment be driven by directory that are not specific to go: smartcd, direnv. Having a solution specific to Go doesn't feel like a big win to my mind, not least because those existing mechanisms could be leveraged to set GOENV to point to a file that is under version control and achieve the same result. Having the go tool search for a main module-local config file feels like another wrinkle.

[dlsniper]

Why not add -proxy=https:// flag to all the commands instead? As that will make it clear what value is used and it's easy to do for those that don't know how to set an environment variable.

[kardianos]

@rsc

1. I think this is a great idea. I think it will help many windows users I'm aware of, it will also help me when I move between shells.
2. There has been a bit of noise over the notary proposal. One possible response to that would be extend this proposal to allow setting a "go.env" file as a sibling to the "go.mod" file. That way people / groups / projects concerned about notary services could set it up their notary options at a project level, similar to how the "codereview.cfg" works for git-codereview.
3. Another potential benefit of having the PROXY and NOTARY variables set at a per proxy level, is again the project lead has to set them up and know about them (but could be in a standard company template), but individual contributors wouldn't need to know or worry about them, bringing them back into not needing to set any vars.

The obvious downside is it adds a layer of precedence. But maybe this could be avoided by some type of go env -inspect flag that lists out each location where it looks for environment variables (present or not) and lists what was set in each, then the final value. This might be nice to have regardless if this proposal is accepted (though it would be ease for a different tool to do as well).

[dr2chase]

Agreeing with @josharian, I like "last wins", just like in a shell script.

Agreeing with @zombiezen, I like the extension to at least two levels for all-users and this-user, but perhaps that can happen later. Note that supporting a system-level file will push the issue of environment-variable expansion on the RHS, then quoting, etc -- which if you are really committed to the idea of LHS=<literal string>, means that will probably not be supported. You could use $$ for literal $ and ${HOME} for environment variables without too much additional hair.

It would be nice if "go help env" or "go env -help" also printed the location of the file(s), and I don't mind hints about how to set environment variables on whatever screwy platform I happen to need help on.

[kevinburke]

Per https://stackoverflow.com/a/2821183/329700, environment variables cannot contain the equals sign character or a null byte but as far as I can tell, the spec does not define the behavior in either of those cases.

KEY=VAL1=VAL2
KEY=VAL\x00VAL2

I'm used to using e.g. envdir to manage configuration like this. A nice feature of envdir is the ability to remove variables from the environment, or keep the values of some env vars (DATABASE_URL) hidden from Git and the values of others (TZ) checked in.

[go101]

Is the env file intended to be used as a global registry for any Go programs?

[kardianos]

@go101 No. This is intended to be only used by the go command and the environment variables the go command sets.