Ability to specify environment variables to be used with the run function

[fgimian]

Hey there, love invoke.

In fabric, we have with shell_env(...) so that a particular set of environment variables will be used in a particular scope, which apply to commands being run.

Is there any way to currently accomplish this with invoke?

I would really love to switch back to it, but I've reluctantly switched to paver until I can figure this out.

Thanks
Fotis

[bitprophet]

There's no explicit support for this yet, but since Invoke is primary concerned with local-to-system shell commands, you should be able to modify os.environ yourself & the subprocess ought to inherit the current state of that data structure when you call run.

So this sort of thing ought to Just Work (tm):

import os
from invoke import run

my_vars = {'FOO': 'bar'}
os.environ.update(my_vars)
run("echo $FOO") # => should result in stdout of "bar"

That's not to say we won't add a 'helper' for this (i.e. a contextmanager that modifies os.environ only in a block) or take stronger ownership of the env presented to our subprocesses (e.g. making them default to an empty env, then offering something like run("command", env={'a': 'dict'})) so I'll leave this open as a placeholder for those concerns. Thanks!

[fgimian]

Thanks so much for your reply @bitprophet, this will do for now. I'm all in favour of allowing us to pass in the env kwarg. The only con with the approach above (that modifies os.environ) is that it modifies os.environ for the entire script, one would need to save the original and restore it afterwards if they wanted to be a little neater 😄

A simple context manager would probably also do the trick, something like this (not tested yet):

def environ(env):
    original_environ = os.environ.copy()
    os.environ.update(env)
    yield
    os.environ = original_environ

Anyways, thanks a lot for all your help and have a lovely weekend!

Cheers
Fotis

[fgimian]

OK, so I thought I had a nice neat little solution here:

from contextlib import contextmanager
import os

@contextmanager
def environ(env):
    original_environ = os.environ.copy()
    os.environ.update(env)
    yield
    os.environ = original_environ

print os.environ.get('name')
with environ({'name': 'Invoke Me :o'}):
    print os.environ.get('name')
print os.environ.get('name')

Output:

➔ python testing.py 
None
Invoke Me :o
None

But ti seems that invoke is doing something which doesn't like this solution 😢

from invoke import task, run

@task
def runme():
    run('echo 1: $name')
    with environ({'name': 'Invoke Me :o'}):
        run('echo 2: $name')
    run('echo 3: $name')

Output:

➔ invoke runme
1:
2: Invoke Me :o
3: Invoke Me :o

Cheers
Fotis

[bitprophet]

Invoke isn't doing anything to os.environ right now IIRC, so something else must be afoot, though I can't see what offhand. My instinct was to be all "maybe os.environ doesn't allow itself to be outright replaced" but nope, I can do that no problem in a shell.

When I test, it looks like my initial assertion is actually false: os.environ mutations do not get picked up by the subprocess after all! To wit:

>>> os.environ['WHAT'] = 'THE HECK?'
>>> os.environ.get('WHAT')
'THE HECK?'
>>> run("env | grep LANG")
LANG=en_US.UTF-8
<invoke.runners.Result object at 0x1044510d0>
>>> run("env | grep WHAT")
<kaboom because grep exited 1 due to no matches>

So that's egg on my face :) turns out we do need a specific feature for this after all!

[bitprophet]

Hrm, I'm confused by the stdlib right now, here's the skinny:

• In the past we used subprocess to run things and I am 99% sure that honors the current environment.
• We recentlyish switched to os.execv for reasons, and while that also claims it supports modifications to os.environ (as per https://docs.python.org/2.6/library/os.html#os.environ and https://docs.python.org/2.6/library/os.html#os.putenv) I'm wondering if there is a bug or other different behavior here.
• Also of note is that we should be able to leverage os.execve and friends (https://docs.python.org/2.6/library/os.html#os.execve) to specify a "clean" env, presumably irrespective of the contents of os.environ and the current confusion about whether it's honored by default.

[bitprophet]

Running some quick tests, neither execv nor subprocess are honoring changes to os.environ on my system (OS X Yosemite, system Python 2.6), either in the Python shell or via python <file.py>. So that's interesting. One of us should dig around in google and the Python tracker to see what the heck - I'm out of time for this today, was just intending to triage briefly :) thanks!

[fgimian]

Firstly, thanks for all your efforts on this, really appreciate it! 😄

Interestingly, things work differently on my Red Hat box here:

Python 2.6.6 (r266:84292, Nov 21 2013, 10:50:32)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> from invoke import run
>>> os.environ['WHAT'] = 'THE HECK?'
>>> os.environ.get('WHAT')
'THE HECK?'
>>> run("env | grep LANG")
LANG=en_US.UTF-8
<invoke.runners.Result object at 0x1a10310>
>>> run("env | grep WHAT")
WHAT=THE HECK?
<invoke.runners.Result object at 0x1a19f10>
>>>

In my testing, both subprocess and os.execve seemed to honour the environment variables set in os.environ.

Edit: Hmm, tested on OS X with Python 2.7 and 2.6 and both got the environment variable.

Cheers
Fotis

[bitprophet]

Which OS X, and are those the system Python interpreters or e.g. installed via brew? I was on Yosemite (10.10.4) and using its system python 2.6(.9).

Also I think I never checked which Invoke version you're running on, though this still doesn't feel like anything our own code would be affecting, given my subprocess+execv test.

[fgimian]

@bitprophet sorry for the late reply on this, this is OS X 10.10 with Homebrew versions of Python, but also tested with the Python that ships and it's fine too 😄

[frol]

There is another concern here. Changing os.environ affects all threads. Basically, It is the same issue as Invoke already has with "current directory": #230

[fgimian]

@frol Very good point. This is a good argument for passing env directly to subprocess instead of manipulating os.environ. @bitprophet what are your thoughts mate? 😄

[bitprophet]

See the last bullet point here - #259 (comment) - I noted that it's an option and yea, right now I can't see a great argument for anything but the "clean env" (obviously, with option for basing that dict on the current value of os.environ) solution. So we'll do that.