os/exec: add Cmd.SetEnv method #12868

Open
bradfitz opened this Issue Oct 7, 2015 · 23 comments

Projects

None yet

10 participants

@bradfitz
Member
bradfitz commented Oct 7, 2015

I'd like to add a method to os/exec.Cmd to ease setting environment variables on commands before they're run.

Manipulating Env []string lists is tedious and error-prone, especially considering potential duplicates. (since programs would others choose either the first or last occurrence, unpredictably)

The proposed Cmd.SetEnv(key, value string) method would be documented as removing duplicates.

On Windows, environment variables are case-insensitive, so the method on Windows should de-dup case insensitively.

This method would replace a handful of copies of func mergeEnvLists that exist in various forms around the standard library, tests, and x/* repos.

/cc @ianlancetaylor @adg @rsc @griesemer @robpike

@bradfitz bradfitz added the Proposal label Oct 7, 2015
@bradfitz
Member
bradfitz commented Oct 7, 2015

I'm not actually dead set on the exact signature. Maybe taking ...string of "key=value" strings would be better, to avoid quadratic behavior when setting a bunch of values.

We would also need to define whether the slice is mutated in-place or replaced. And the semantics when cmd.Env is nil (probably initialize it to os.Environ())

@ianlancetaylor
Contributor

SGTM. In cmd/go/go_test.go I also found it useful to have an Unsetenv.

@adg
Contributor
adg commented Oct 8, 2015

I'd be happy with cmd.SetEnv("KEY=val", "KEY=val", ...)

And the semantics when cmd.Env is nil (probably initialize it to os.Environ())

Makes sense to me.

@bradfitz
Member
bradfitz commented Oct 8, 2015

I started to implement cmd.SetEnv("KEY=val", "KEY=val", ...) but found two gross things about it:

  1. the os package spells it os.Setenv with a lowercase 'e'.
  2. the os package signature is Setenv(key, value string), so people would assume that, and then it would silently compile with a SetEnv(...string) signature. So then what? Runtime panic? Magically work if even number of arguments and odd ones don't have = bytes in them? Gross.

So then I was back to thinking about just Setenv(key, value string) and Unsetenv(key string) (for @ianlancetaylor). But then we're back to them being quadratic if there's a bunch of them.

Maybe we shouldn't care. N is small, etc (few environment variables).

But if we did care, I was toying with ideas like:

// EnvMod represents a modification to environment variables.                                                                               
type EnvMod struct {
        key string
        val string
        set bool
}

func Unsetenv(key string) EnvMod      { return EnvMod{key: key} }
func Setenv(key, value string) EnvMod { return EnvMod{key: key, val: value, set: true} }

func (c *Cmd) ModifyEnv(mods ...EnvMod) { ... }

But call sites are then kinda ugly:

        c.ModifyEnv(exec.Setenv("GOROOT", v), exec.Unsetenv("GOBIN"))

(and calling exec.Setenv directly and ignoring its return value looks like it does something but doesn't)

So then I thought: maybe we just keep the functions simple:

    func(c *Cmd) Setenv(key, value string) {...}
    func(c *Cmd) Unsetenv(key string) {...}

... but for people who care about the quadratic behavior, they can optionally use:

    // ModifyEnv runs fn, which should contain a series of Setenv and/or Unsetenv calls.
    // Use of ModifyEnv is optional but avoids quadratic behavior eliminating duplicates
    // when modifying many environment variables.
    func (c *Cmd) ModifyEnv(fn func()) {
              // build, cache data structure on c
              fn()
              // clear cache data structure on c
    }

There's not much (any?) precedent for such a function, though. The closest is App Engine's RunInTransaction perhaps: https://godoc.org/google.golang.org/appengine/datastore#RunInTransaction

Maybe we can introduce ModifyEnv later if needed, and just accept the poor behavior of (*Cmd).Setenv and (*Cmd).Unsetenv in the meantime?

@extemporalgenome

Perhaps we could have Setenv just append to the Env slice with custom logic: if the append would allocate, then do a pruning first (later values take precedence). Start (and Run) would also need to do a pruning as well.

@bradfitz
Member
bradfitz commented Oct 8, 2015

We can't dedup in Start or Run. That would be a change of behavior and it's valid (even if weird) to have multi-valued keys. And if we did dedup: first or last? Different programs pick a different one.

@adg
Contributor
adg commented Oct 9, 2015

Why don't we just let people who really care about the quadratic behavior futz with cmd.Env manually?

@adg
Contributor
adg commented Oct 9, 2015

Although I also like the variadic approach panicking on cmd.Setenv("KEY", "value").

To unset you could cmd.Setenv("KEY=value", "UNSETKEY=").

@bradfitz
Member
bradfitz commented Oct 9, 2015

UNSETKEY hack is gross. Then it's a magic environment variable you can't set.

@adg
Contributor
adg commented Oct 9, 2015

No no I meant that UNSETKEY is literally the key name that you want to unset. It's unset because the value component is empty. It could be "FOO=". Sorry for the confusion.

@bradfitz
Member
bradfitz commented Oct 9, 2015

Oh. I also considered that but there's a difference between empty and unset and we wouldn't want to differ from os.Setenv semantics. That distinction and complaints is why we finally added os.Unsetenv.

@masiulaniec

I'm not actually dead set on the exact signature. Maybe taking ...string of "key=value" strings would be better

Since it doesn't look like it's been considered: don't forget about the approach taken by https://golang.org/pkg/strings/#NewReplacer

@bradfitz
Member

@masiulaniec, good point. That's worth consideration. It doesn't quite work for the Unsetenv case, though. Unless we were to declare some magical value for the unset value (like "\x00", since "" is a valid value, and a NULL byte can't exist in environment variables). Little inconsistent with os.Setenv though.

I'm increasingly thinking we should just call it (*Cmd).MergeEnv(...) in any case, so we can get the case right on Env and be allowed to set our own semantics instead of conforming to expectations of os.Setenv.

@shurcooL
Member

I'm glad this task is being addressed and made easier, because I too had to write little internal helpers to help manage env vars for exec.Cmds. The way I've done it was:

// environ is a slice of strings representing the environment, in the form "key=value".
type environ []string

// Unset a single environment variable.
func (e *environ) Unset(key string) {
    for i := range *e {
        if strings.HasPrefix((*e)[i], key+"=") {
            (*e)[i] = (*e)[len(*e)-1]
            *e = (*e)[:len(*e)-1]
            break
        }
    }
}

And usage is:

env := environ(os.Environ())
env.Unset("FOO")
cmd.Env = env
@adg
Contributor
adg commented Oct 12, 2015

I'm increasingly thinking we should just call it (*Cmd).MergeEnv(...)

SGTM

@rsc rsc added this to the Proposal milestone Oct 24, 2015
@rsc
Contributor
rsc commented Oct 24, 2015

A few points.

  1. I don't see what this has to do with exec.Cmd. You're talking about writing general helper functions for manipulating []string representing environment lists and then hard-coding that they only operate on cmd.Env.

  2. I don't see why methods on exec.Cmd must necessarily store their state in the Env field. The current semantics are that cmd.Env == nil means "use the current environment", while cmd.Env != nil means "use this exact environment". You could reasonably make the methods operate on behind-the-scenes state and define that cmd.Env == nil means "use the current environment as modified by calls to Setenv/Unsetenv", while if cmd.Env != nil then the calls are ignored.

  3. Along that line of thought, the Setenv and Unsetenv methods could just record the operations in an unexported field and then they could be replayed onto the effective environment at (*Cmd).Start time. Then they could apply to both nil and non-nil cmd.Env.

  4. Or maybe this really should be a separate type (like exec.Env) that helps you prepare an environment list. Internally the Env implementation would be something like a slice+map.

    var env exec.Env
    env = exec.Environ()
    env.Setenv("XXX", "YYY")
    env.Unsetenv("ZZZ")
    cmd.Env = env.List()
    
@minux
Member
minux commented Oct 24, 2015
@rsc rsc changed the title from proposal: add os/exec.Cmd.SetEnv method to proposal: os/exec: add Cmd.SetEnv method Oct 24, 2015
@bradfitz
Member

I would still like to do this. Ideally in Go 1.8.

@robpike
Contributor
robpike commented Aug 15, 2016

Accepted pending final details about the API.

@adg adg modified the milestone: Go1.8, Proposal Aug 15, 2016
@cbednarski
cbednarski commented Aug 16, 2016 edited

Slightly OT but as a history lesson, is this a slice to match os.Environ()'s type or is there some other reason? Is order important? Does some system allow dupes? Seems like a map would fit the data better (I'm aware there is a BC issue there; just curious on the original intent).

@ianlancetaylor
Contributor

@cbednarski The environment is represented as a slice mainly because in C it is a char**, both in how it is passed to the program (as the third argument to main or in the environ global variable) and in how it is passed to execve system call.

All Unix systems do support duplicate environment variables, though their effect on specific programs depends on how those programs handle the environment array.

@cbednarski

@ianlancetaylor Great explanation, thanks very much!

@rsc
Contributor
rsc commented Oct 18, 2016

Not sure whether this is still about Cmd.SetEnv or about the more general exec.Env I suggested. Either way, looks like it will wait for Go 1.9.

@rsc rsc modified the milestone: Go1.9Early, Go1.8 Oct 18, 2016
@rsc rsc changed the title from proposal: os/exec: add Cmd.SetEnv method to os/exec: add Cmd.SetEnv method Oct 18, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment