os/exec: add Cmd.SetEnv method

[bradfitz]

I'd like to add a method to os/exec.Cmd to ease setting environment variables on commands before they're run.

Manipulating Env []string lists is tedious and error-prone, especially considering potential duplicates. (since programs would others choose either the first or last occurrence, unpredictably)

The proposed Cmd.SetEnv(key, value string) method would be documented as removing duplicates.

On Windows, environment variables are case-insensitive, so the method on Windows should de-dup case insensitively.

This method would replace a handful of copies of func mergeEnvLists that exist in various forms around the standard library, tests, and x/* repos.

/cc @ianlancetaylor @adg @rsc @griesemer @robpike

[bradfitz]

I'm not actually dead set on the exact signature. Maybe taking ...string of "key=value" strings would be better, to avoid quadratic behavior when setting a bunch of values.

We would also need to define whether the slice is mutated in-place or replaced. And the semantics when cmd.Env is nil (probably initialize it to os.Environ())

[ianlancetaylor]

SGTM. In cmd/go/go_test.go I also found it useful to have an Unsetenv.

[adg]

I'd be happy with cmd.SetEnv("KEY=val", "KEY=val", ...)

And the semantics when cmd.Env is nil (probably initialize it to os.Environ())

Makes sense to me.

[bradfitz]

I started to implement cmd.SetEnv("KEY=val", "KEY=val", ...) but found two gross things about it:

1. the os package spells it os.Setenv with a lowercase 'e'.
2. the os package signature is Setenv(key, value string), so people would assume that, and then it would silently compile with a SetEnv(...string) signature. So then what? Runtime panic? Magically work if even number of arguments and odd ones don't have = bytes in them? Gross.

So then I was back to thinking about just Setenv(key, value string) and Unsetenv(key string) (for @ianlancetaylor). But then we're back to them being quadratic if there's a bunch of them.

Maybe we shouldn't care. N is small, etc (few environment variables).

But if we did care, I was toying with ideas like:

// EnvMod represents a modification to environment variables.                                                                               
type EnvMod struct {
        key string
        val string
        set bool
}

func Unsetenv(key string) EnvMod      { return EnvMod{key: key} }
func Setenv(key, value string) EnvMod { return EnvMod{key: key, val: value, set: true} }

func (c *Cmd) ModifyEnv(mods ...EnvMod) { ... }

But call sites are then kinda ugly:

        c.ModifyEnv(exec.Setenv("GOROOT", v), exec.Unsetenv("GOBIN"))

(and calling exec.Setenv directly and ignoring its return value looks like it does something but doesn't)

So then I thought: maybe we just keep the functions simple:

    func(c *Cmd) Setenv(key, value string) {...}
    func(c *Cmd) Unsetenv(key string) {...}

... but for people who care about the quadratic behavior, they can optionally use:

    // ModifyEnv runs fn, which should contain a series of Setenv and/or Unsetenv calls.
    // Use of ModifyEnv is optional but avoids quadratic behavior eliminating duplicates
    // when modifying many environment variables.
    func (c *Cmd) ModifyEnv(fn func()) {
              // build, cache data structure on c
              fn()
              // clear cache data structure on c
    }

There's not much (any?) precedent for such a function, though. The closest is App Engine's RunInTransaction perhaps: https://godoc.org/google.golang.org/appengine/datastore#RunInTransaction

Maybe we can introduce ModifyEnv later if needed, and just accept the poor behavior of (*Cmd).Setenv and (*Cmd).Unsetenv in the meantime?

[extemporalgenome]

Perhaps we could have Setenv just append to the Env slice with custom logic: if the append would allocate, then do a pruning first (later values take precedence). Start (and Run) would also need to do a pruning as well.

[bradfitz]

We can't dedup in Start or Run. That would be a change of behavior and it's valid (even if weird) to have multi-valued keys. And if we did dedup: first or last? Different programs pick a different one.

[adg]

Why don't we just let people who really care about the quadratic behavior futz with cmd.Env manually?

[adg]

Although I also like the variadic approach panicking on cmd.Setenv("KEY", "value").

To unset you could cmd.Setenv("KEY=value", "UNSETKEY=").

[bradfitz]

UNSETKEY hack is gross. Then it's a magic environment variable you can't set.

[adg]

No no I meant that UNSETKEY is literally the key name that you want to unset. It's unset because the value component is empty. It could be "FOO=". Sorry for the confusion.

[bradfitz]

Oh. I also considered that but there's a difference between empty and unset and we wouldn't want to differ from os.Setenv semantics. That distinction and complaints is why we finally added os.Unsetenv.

[masiulaniec]

I'm not actually dead set on the exact signature. Maybe taking ...string of "key=value" strings would be better

Since it doesn't look like it's been considered: don't forget about the approach taken by https://golang.org/pkg/strings/#NewReplacer

[bradfitz]

@masiulaniec, good point. That's worth consideration. It doesn't quite work for the Unsetenv case, though. Unless we were to declare some magical value for the unset value (like "\x00", since "" is a valid value, and a NULL byte can't exist in environment variables). Little inconsistent with os.Setenv though.

I'm increasingly thinking we should just call it (*Cmd).MergeEnv(...) in any case, so we can get the case right on Env and be allowed to set our own semantics instead of conforming to expectations of os.Setenv.

[dmitshur]

I'm glad this task is being addressed and made easier, because I too had to write little internal helpers to help manage env vars for exec.Cmds. The way I've done it was:

// environ is a slice of strings representing the environment, in the form "key=value".
type environ []string

// Unset a single environment variable.
func (e *environ) Unset(key string) {
    for i := range *e {
        if strings.HasPrefix((*e)[i], key+"=") {
            (*e)[i] = (*e)[len(*e)-1]
            *e = (*e)[:len(*e)-1]
            break
        }
    }
}

And usage is:

env := environ(os.Environ())
env.Unset("FOO")
cmd.Env = env