Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Virtualenv cell magic #1891

Closed
wants to merge 14 commits into
from

Conversation

Projects
None yet
6 participants

fccoelho commented Jun 9, 2012

Hi,

here is my virtualenv extension. It is very simple, and has a couple of tests to it (84% coverage). It requires a couple of envs to test, but if they are not present, it raises SkipTest.

Suggestion for improvement both of the extention or its testing are very welcome!

@takluyver takluyver and 1 other commented on an outdated diff Jun 9, 2012

IPython/extensions/tests/test_virtualenvmagic.py
+def test_virtualenv_python3():
+ if not env_exists('py3'):
+ raise SkipTest
+ ip = get_ipython()
+ result = ip.run_cell_magic('virtualenv', 'py3', 'import sys;print \
+ (sys.version_info.major)')
+ nt.assert_equals('3\n', result)
+
+@nt.nottest
+def env_exists(env):
+ """
+ check if environment exists
+ """
+ if 'WORKON_HOME' not in os.environ:
+ return False
+ if os.path.exists(os.environ['WORKON_HOME'] + '/' + env):
@takluyver

takluyver Jun 9, 2012

Owner

os.path.join ;-)

@takluyver

takluyver Jun 10, 2012

Owner

Not fixed on this line, though!

@takluyver takluyver and 1 other commented on an outdated diff Jun 9, 2012

IPython/extensions/virtualenvmagic.py
+ The virtual env to be used must be created in advance.
+
+ Usage
+ =====
+ To activate this magic just write at the top of the cell:
+
+ %%virtualenv my_env
+ import sys
+ print sys.version
+
+
+ """
+ if not os.path.exists(os.environ['WORKON_HOME'] + '/' + line):
+ print >> sys.stderr, "Environment {} does not exist.".format(line)
+ return
+ env_activate_cmd = 'bash -c "source {}/{}/bin/activate && python -"'\
@takluyver

takluyver Jun 9, 2012

Owner

The plain {} in format string are new in 2.7/3.1, so while we're still supporting Python 2.6, we'll need them numbered: {0}.

http://docs.python.org/library/string.html#formatstrings

@takluyver

takluyver Jun 9, 2012

Owner

Also, we should think about how this can work in Windows. I'm sure it's possible, but it won't be quite the same.

@fccoelho

fccoelho Jun 9, 2012

I'll look into it. Need to find out how virtualenv works on windows.

@takluyver takluyver and 1 other commented on an outdated diff Jun 9, 2012

IPython/extensions/virtualenvmagic.py
+ =====
+ To activate this magic just write at the top of the cell:
+
+ %%virtualenv my_env
+ import sys
+ print sys.version
+
+
+ """
+ if not os.path.exists(os.environ['WORKON_HOME'] + '/' + line):
+ print >> sys.stderr, "Environment {} does not exist.".format(line)
+ return
+ env_activate_cmd = 'bash -c "source {}/{}/bin/activate && python -"'\
+ .format(os.environ['WORKON_HOME'], line)
+ cmd = shlex.split(env_activate_cmd)
+ p = Popen(cmd, stdout=PIPE, stderr=PIPE, stdin=PIPE)
@takluyver

takluyver Jun 9, 2012

Owner

Specify shell=True here, and we shouldn't need bash -c, I think.

@fccoelho

fccoelho Jun 9, 2012

shell= False is safer, according to the subprocess documentation, moreover, this line is not likely to stay as is, if I find a way to make this work on windows. So I am leaving it as is for now.

@takluyver

takluyver Jun 9, 2012

Owner

You're effectively recreating shell=True by using bash -c. But the vulnerability the docs mention is about untrusted input injecting shell code - which we don't need to worry about, because the notebook exists to let the user run arbitrary code.

@fccoelho

fccoelho Jun 9, 2012

I think you are right, but I am going to have to remove the dependency on
bash anyway, to make it work on windows.

cheers,

Flávio

On Sat, Jun 9, 2012 at 10:35 AM, Thomas Kluyver <
reply@reply.github.com

wrote:

  • =====
  • To activate this magic just write at the top of the cell:
  •    %%virtualenv my_env
    
  •    import sys
    
  •    print sys.version
    
  • """
  • if not os.path.exists(os.environ['WORKON_HOME'] + '/' + line):
  •    print >> sys.stderr, "Environment {} does not
    
    exist.".format(line)
  •    return
    
  • env_activate_cmd = 'bash -c "source {}/{}/bin/activate && python
    -"'\
  • .format(os.environ['WORKON_HOME'], line)
  • cmd = shlex.split(env_activate_cmd)
  • p = Popen(cmd, stdout=PIPE, stderr=PIPE, stdin=PIPE)

You're effectively recreating shell=True by using bash -c. But the
vulnerability the docs mention is about untrusted input injecting shell
code - which we don't need to worry about, because the notebook exists to
let the user run arbitrary code.


Reply to this email directly or view it on GitHub:
https://github.com/ipython/ipython/pull/1891/files#r955665

Flávio Codeço Coelho

+55(21) 3799-5567
Professor
Escola de Matemática Aplicada
Fundação Getúlio Vargas
Rio de Janeiro - RJ
Brasil

@fccoelho

fccoelho Jun 10, 2012

Thomas,

I have made some new commits which introduce the beginnings of windows
support. I decided to check sys.platform and handle env activation
differently. Haven't tested on windows, since I don't have a windows box
handy, but the Linux part is still functional and passes all existing tests.

As per your suggestion I have also added the programmatic creation of a
test env just for running the tests which gets deleted on teardown.

I'll appreciate any comments as usual.

Flávio

On Sat, Jun 9, 2012 at 12:28 PM, Flavio Coelho fccoelho@gmail.com wrote:

I think you are right, but I am going to have to remove the dependency on
bash anyway, to make it work on windows.

cheers,

Flávio

On Sat, Jun 9, 2012 at 10:35 AM, Thomas Kluyver <
reply@reply.github.com

wrote:

  • =====
  • To activate this magic just write at the top of the cell:
  •    %%virtualenv my_env
    
  •    import sys
    
  •    print sys.version
    
  • """
  • if not os.path.exists(os.environ['WORKON_HOME'] + '/' + line):
  •    print >> sys.stderr, "Environment {} does not
    
    exist.".format(line)
  •    return
    
  • env_activate_cmd = 'bash -c "source {}/{}/bin/activate && python
    -"'\
  • .format(os.environ['WORKON_HOME'], line)
  • cmd = shlex.split(env_activate_cmd)
  • p = Popen(cmd, stdout=PIPE, stderr=PIPE, stdin=PIPE)

You're effectively recreating shell=True by using bash -c. But the
vulnerability the docs mention is about untrusted input injecting shell
code - which we don't need to worry about, because the notebook exists to
let the user run arbitrary code.


Reply to this email directly or view it on GitHub:
https://github.com/ipython/ipython/pull/1891/files#r955665

Flávio Codeço Coelho

+55(21) 3799-5567
Professor
Escola de Matemática Aplicada
Fundação Getúlio Vargas
Rio de Janeiro - RJ
Brasil

Flávio Codeço Coelho

+55(21) 3799-5567
Professor
Escola de Matemática Aplicada
Fundação Getúlio Vargas
Rio de Janeiro - RJ
Brasil

Owner

takluyver commented Jun 9, 2012

For the tests: rather than relying on VEW being installed and active, we should set $WORKON_HOME to a temporary directory, make a virtualenv there and test against that. Also, rather than depending on different versions of Python, we should make the subprocess print(sys.path)*, and check that the temporary directory is in the output.

*brackets so it works on Python 2 or 3.

I have Started on support for windows but didn't finish it. it will be pretty complicated since VEW doesn't support it.

Owner

takluyver commented Jun 10, 2012

OK, that's not quite what I meant with the tests. We shouldn't mess with user visible folders in the test suite. Have a look at IPython.utils.tempdir. In the setup() function, put something like:

workon_dir = TemporaryDirectory()
os.environ['WORKON_HOME'] = workon_dir.name

Then in teardown():

del os.environ['WORKON_HOME']
workon_dir.cleanup()

workon_dir will need to be a global variable (or make the tests a class and attach it to the instance). Really, teardown() should restore $WORKON_HOME to whatever it was previously, but that's probably not a major concern.

@minrk minrk and 1 other commented on an outdated diff Jun 11, 2012

IPython/extensions/virtualenvmagic.py
+ return
+ cmd = get_activate_cmd(line)
+ p = Popen(cmd, stdout=PIPE, stderr=PIPE, stdin=PIPE)
+ out, err = p.communicate(cell)
+ p.wait()
+ if err:
+ print >> sys.stderr, err
+ return out
+
+_loaded = False
+
+def get_activate_cmd(line):
+ """
+ Returns the correct activate command given the platform
+ """
+ if sys.platform.startswith("linux"):
@minrk

minrk Jun 11, 2012

Owner

This should be the default behavior, as the most likely other platforms (OS X, freebsd, etc.) will all behave the same as Linux here, rather than "Platform not supported".

@fccoelho

fccoelho Aug 25, 2012

It's inactive for the time being but I intend to finish it in time for
release 0.14.
Em 11/06/2012 18:50, "Min RK" <
reply@reply.github.com>
escreveu:

  •    return
    
  • cmd = get_activate_cmd(line)
  • p = Popen(cmd, stdout=PIPE, stderr=PIPE, stdin=PIPE)
  • out, err = p.communicate(cell)
  • p.wait()
  • if err:
  •    print >> sys.stderr, err
    
  • return out

+_loaded = False
+
+def get_activate_cmd(line):

  • """
  • Returns the correct activate command given the platform
  • """
  • if sys.platform.startswith("linux"):

This should be the default behavior, as the most likely other platforms
(OS X, freebsd, etc.) will all behave the same as Linux here, rather than
"Platform not supported".


Reply to this email directly or view it on GitHub:
https://github.com/ipython/ipython/pull/1891/files#r963238

@takluyver takluyver and 1 other commented on an outdated diff Jun 18, 2012

IPython/extensions/tests/test_virtualenvmagic.py
+ result = ip.run_cell_magic('virtualenv', 'pypyEnv', 'import sys;print\
+ (sys.version)')
+ nt.assert_true('PyPy' in result)
+
+def test_virtualenv_python3():
+ if not env_exists('py3'):
+ raise SkipTest("Environment py3 not found, skipping test")
+ ip = get_ipython()
+ result = ip.run_cell_magic('virtualenv', 'py3', 'import sys;print \
+ (sys.version_info.major)')
+ nt.assert_equals('3\n', result)
+
+def test_virtualenv_testenv():
+ ip = get_ipython()
+ result = ip.run_cell_magic('virtualenv', 'testenv', 'print("hello")')
+ nt.assert_equals('hello\n', result)
@takluyver

takluyver Jun 18, 2012

Owner

Let's make this test do something to demonstrate that it's running in the virtualenv - like printing sys.path. Then we can get rid of the redundant test_virtualenv_pypy and test_virtualenv_python3.

@fccoelho

fccoelho Jun 18, 2012

Done.

On Sun, Jun 17, 2012 at 10:01 PM, Thomas Kluyver <
reply@reply.github.com

wrote:

  • result = ip.run_cell_magic('virtualenv', 'pypyEnv', 'import
    sys;print\
  • (sys.version)')
    
  • nt.assert_true('PyPy' in result)

+def test_virtualenv_python3():

  • if not env_exists('py3'):
  •    raise SkipTest("Environment py3 not found, skipping test")
    
  • ip = get_ipython()
  • result = ip.run_cell_magic('virtualenv', 'py3', 'import sys;print \
  • (sys.version_info.major)')
  • nt.assert_equals('3\n', result)

+def test_virtualenv_testenv():

  • ip = get_ipython()
  • result = ip.run_cell_magic('virtualenv', 'testenv',
    'print("hello")')
  • nt.assert_equals('hello\n', result)

Let's make this test do something to demonstrate that it's running in the
virtualenv - like printing sys.path. Then we can get rid of the redundant
test_virtualenv_pypy and test_virtualenv_python3.


Reply to this email directly or view it on GitHub:
https://github.com/ipython/ipython/pull/1891/files#r998914

Flávio Codeço Coelho

+55(21) 3799-5567
Professor
Escola de Matemática Aplicada
Fundação Getúlio Vargas
Praia de Botafogo, 190 sala 312
Rio de Janeiro - RJ
22250-900
Brasil

@minrk minrk commented on the diff Jun 18, 2012

IPython/extensions/virtualenvmagic.py
+ 'Scripts','activate') + "; python -"
+ else:
+ env_activate_cmd = 'bash -c "source {0}/{1}/bin/activate && python -"'\
+ .format(os.environ['WORKON_HOME'], line)
+
+ cmd = shlex.split(env_activate_cmd)
+ return cmd
+
+
+def load_ipython_extension(ip):
+ """
+ Load the extension in IPython.
+ """
+ global _loaded
+ if not _loaded:
+ if 'WORKON_HOME' in os.environ:
@minrk

minrk Jun 18, 2012

Owner

This is not an accurate test of whether virtualenvwrapper is installed. The WORKON_HOME env need not be defined (and in fact is not under default circumstances), and its default value is $HOME/.virtualenvs, not $HOME/Env as your tests would suggest.

@minrk minrk commented on the diff Jun 18, 2012

IPython/extensions/virtualenvmagic.py
+ This magic enables the execution of code in a cell on a
+ pre-existing virtual env. It Requires Virtualenv and VirtualenvWrapper
+ to be installed in the system.
+ The virtual env to be used must be created in advance.
+
+ Usage
+ =====
+ To activate this magic just write at the top of the cell:
+
+ %%virtualenv my_env
+ import sys
+ print sys.version
+
+
+ """
+ if not os.path.exists(os.path.join(os.environ['WORKON_HOME'], line)):
@minrk

minrk Jun 18, 2012

Owner

This can be safely used if workon_home is undefined, because it is normal to specify the absolute path of an env.

Something like:

def get_workon_home():
    if 'WORKON_HOME' in os.environ:
        return os.environ['WORKON_HOME']
    default = os.path.join(os.environ['HOME'], '.virtualenvs')
    if os.path.exists(default):
        return default
    # fallback on empty, absolute or cwd-relative paths must be used:
    return ''

@minrk minrk and 1 other commented on an outdated diff Jun 18, 2012

IPython/extensions/virtualenvmagic.py
+ =====
+ To activate this magic just write at the top of the cell:
+
+ %%virtualenv my_env
+ import sys
+ print sys.version
+
+
+ """
+ if not os.path.exists(os.path.join(os.environ['WORKON_HOME'], line)):
+ print >> sys.stderr, "Environment {0} does not exist.".format(line)
+ return
+ cmd = get_activate_cmd(line)
+ p = Popen(cmd, stdout=PIPE, stderr=PIPE, stdin=PIPE)
+ out, err = p.communicate(cell)
+ p.wait()
@minrk

minrk Jun 18, 2012

Owner

This wait is redundant with communicate above.

@takluyver takluyver commented on the diff Jun 18, 2012

IPython/extensions/tests/test_virtualenvmagic.py
+ #creating a testing virtualenv
+ testenv_dir = TemporaryDirectory()
+ os.environ['WORKON_HOME'] = testenv_dir.name
+ virtualenv.create_environment(os.path.join(testenv_dir.name,'testenv'))
+ ip.extension_manager.load_extension('virtualenvmagic')
+
+def teardown():
+ del os.environ['WORKON_HOME']
+ testenv_dir.cleanup()
+
+
+
+def test_virtualenv_testenv():
+ ip = get_ipython()
+ result = ip.run_cell_magic('virtualenv', 'testenv', 'import sys;print(sys.path)')
+ nt.assert_equals('tmp', os.path.split(eval(result)[1])[0].split(os.path.sep)[1])
@takluyver

takluyver Jun 18, 2012

Owner

Not a very robust test - all sorts of things can be in /tmp. And it's probably not called /tmp on Windows.

What about something like nt.assert_in(testenv_dir.name, result)?

Owner

fperez commented Jun 26, 2012

@takluyver, @minrk: what's your take on this one? It seems there's a comment from @takluyver still pending improvement; is it otherwise in mergeable state? If so, is that small missing point a deal-breaker to merge it? Given this will be probably pretty popular, I'm willing to ask @fccoelho to improve the testing post-0.13 if need be, so that we can get it in now.

Thoughts?

Owner

minrk commented Jun 26, 2012

There are several unaddressed comments here, and this is not even basically functional on any of my systems, due to the treatment of WORKON_HOME. I do not think it is ready to merge.

Owner

fperez commented Jun 26, 2012

Got it. Then let's target this for 0.14, as I simply don't see it getting ready in a day or two (and us having the bandwidth to carefully review and test it). Thanks @minrk for that one...

It's best this way. I don't have the time to address those issues right
now, but they are on my todo list.

thanks for the patience,

Flávio

On Tue, Jun 26, 2012 at 1:47 AM, Fernando Perez <
reply@reply.github.com

wrote:

Got it. Then let's target this for 0.14, as I simply don't see it getting
ready in a day or two (and us having the bandwidth to carefully review and
test it). Thanks @minrk for that one...


Reply to this email directly or view it on GitHub:
#1891 (comment)

Flávio Codeço Coelho

+55(21) 3799-5567
Professor
Escola de Matemática Aplicada
Fundação Getúlio Vargas
Praia de Botafogo, 190 sala 312
Rio de Janeiro - RJ
22250-900
Brasil

Owner

fperez commented Jun 26, 2012

No worries, @fccoelho! We're very happy to have your contributions, and 0.14 will be out before you know it. It's OK to polish after merge, but since in this case it looks like it does need real work, re-targeting seems like the best policy. We're very happy to have you contributing to IPython, so I hope this bubbles to the top of your todo list before too long :)

Owner

Carreau commented Aug 24, 2012

Hi,

This PR is unactive for 2 month now. What is it's status ?

Thanks.

It's inactive for now,

But I plan to finish it before the next Ipython's release.

Flávio

On Fri, Aug 24, 2012 at 6:31 AM, Bussonnier Matthias <
notifications@github.com> wrote:

Hi,

This PR is unactive for 2 month now. What is it's status ?

Thanks.


Reply to this email directly or view it on GitHubhttps://github.com/ipython/ipython/pull/1891#issuecomment-7996725.

Flávio Codeço Coelho

+55(21) 3799-5567
Professor
Escola de Matemática Aplicada
Fundação Getúlio Vargas
Praia de Botafogo, 190 sala 312
Rio de Janeiro - RJ
22250-900
Brasil

Owner

Carreau commented Aug 25, 2012

Ok, no problem, take your time :-)

Contributor

bfroehle commented Sep 18, 2012

@fccoelho:

I'm going to close this pull request for now. Please reopen it if and when you have time to complete the necessary changes.

Altenatively you might find it more convenient to just upload your extension to gist or similar and post it on the list of IPython exensions.

Thanks.

@bfroehle bfroehle closed this Sep 18, 2012

fine.

On Tue, Sep 18, 2012 at 5:32 PM, Bradley M. Froehle <
notifications@github.com> wrote:

@fccoelho https://github.com/fccoelho:

I'm going to close this pull request for now. Please reopen it if and when
you have time to complete the necessary changes.

Altenatively you might find it more convenient to just upload your
extension to gist https://gist.github.com or similar and post it on the
list of IPython exensions http://wiki.ipython.org/Extensions_Index.

Thanks.


Reply to this email directly or view it on GitHubhttps://github.com/ipython/ipython/pull/1891#issuecomment-8669327.

Flávio Codeço Coelho

+55(21) 3799-5567
Professor
Escola de Matemática Aplicada
Fundação Getúlio Vargas
Praia de Botafogo, 190 sala 312
Rio de Janeiro - RJ
22250-900
Brasil

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment