Recognise virtualenvs #1171

Closed
takluyver opened this Issue Dec 17, 2011 · 13 comments

Projects

None yet

4 participants

@takluyver
Member

A recurring issue, which I've just added to the FAQ - a standard installation of IPython won't recognise when you're in a virtualenv. Various workarounds are showing up, but it would be good to work out of the box.

The root of the problem is that the installation sets the shebang line of ipython to #!/usr/bin/python. If it's #!/usr/bin/env python, virtualenvs work as expected. (I don't know what the status is on Windows)

A few possible ways to deal with this:

  • Get an option added to distutils/packaging/setuptools/distribute to control the shebang line written. Long term at best.
  • For Linux distributions, the shebang could be replaced as a step in the packaging.
  • Contribute ipython-awareness to virtualenv, so if IPython is installed, it will copy the script and update the shebang line when you create a virtualenv.
  • Detect when we're in a virtualenv, and add the relevant paths to sys.path. This is suboptimal, because it only affects importable libraries, not the interpreter itself (e.g. if I make a Python 2.6 virtualenv, IPython will still start with the default Python 2.7).
@fperez
Member
fperez commented Dec 17, 2011

Mmh, how is this problem unique to us? If this is happening, then it happens to any python project that installs executable scripts. Granted, many python projects are indeed just libraries with minimal to no user-facing executable code, but still: nose, cython are two that immediately come to mind with this same situation.

How does everyone else handle this problem? I'm not claiming it's not an issue, just thinking that somebody out there must have figured this one out...

@takluyver
Member

I think for most things, it makes sense to run it with the system Python, ignoring locally installed libraries (e.g. if I run a mercurial command, a virtualenv shouldn't interfere with it). I suspect nose has the same problem in a standard installation, and I've also run into it with sphinx-build.

@fperez
Member
fperez commented Dec 18, 2011

OK, I figured it out. The solution (not just for IPython, but for installying any python package that has scripts in a virtualenv) is to do the build step separately:

./setup.py build --executable "/usr/bin/env python"
./setup.py install

This sets the shebang line correctly once the scripts get installed.

Since this is standard distutils, I think we can actually close the issue here. Feel free to put this tip up on the faq, as it's certainly not very obvious. I had an inkling of the solution because I had a vague memory of being able to control that line years ago when I was building RPMs with distutils for a lab where I had a bunch of Fedora machines. But otherwise it's not easy to just stumble upon this solution.

@fperez fperez closed this Dec 18, 2011
@minrk
Member
minrk commented Dec 18, 2011

Yes, this has never had anything to do with IPython, it's always affected all distutils-installed scripts, but IPython is often the only such script inherited into a virtualenv.

It should be considered a virtualenv bug that they don't patch the #! line of python scripts.

@takluyver
Member

So it turns out we can set this without requiring the user to pass an extra command line argument (http://stackoverflow.com/a/1719991/434217), but it only works for distutils style 'scripts', not setuptools 'entry_points'. I've got a branch for that, and I'll look into it more closely for 0.13.

I'm reopening this and assigning it to myself on the grounds that I think we can improve on the current situation.

@takluyver takluyver reopened this Dec 18, 2011
@takluyver takluyver was assigned Dec 18, 2011
@fperez
Member
fperez commented Dec 18, 2011

On Sun, Dec 18, 2011 at 4:20 AM, Thomas
reply@reply.github.com
wrote:

I'm reopening this and assigning it to myself on the grounds that I think we can improve on the current situation.

No problem.

Cheers,

f

@juliantaylor
Collaborator

fyi, the debian packages already use env python as shebang for ipython since a while [0], but for the entry point script used in py3 I have found no other way than to sed the resulting scripts [1].
what I find weird, but probably is a distutils issue, is that the script is only created on install, not on build.
This also makes testing a bit more ackward as iptest3 is only created after installation.

[0] http://anonscm.debian.org/viewvc/python-modules/packages/ipython/trunk/debian/rules?r1=15749&r2=16684
[1] http://anonscm.debian.org/viewvc/python-modules/packages/ipython/trunk/debian/rules?r1=19668&r2=19670

@takluyver
Member

Thanks, that's good to know. Is 0.12 in Debian now, then?

Virtualenv 1.7 has made no-site-packages the default behaviour. AFAIK, this
means that if we use the the env Python interpreter, IPython installed
systemwide won't be importable. We'll need to think about how best to
handle that. Off the top of my head, we might need to switch in a 'user'
sys.path when we run user code, separate from the sys.path used by IPython
itself.

@juliantaylor
Collaborator

I don't think we need special handling for the new venv behavior. That the system wide ipython is not importable with --no-site-packages is the expected (and likely wanted) behavior.
installing ipython in that venv with easy_install results in the correct shebang.

@takluyver
Member

But if we use the shebang #!/usr/bin/env ipython, IPython will refuse to
start at all with a virtualenv active until you've reinstalled it. The
ipython command will still be on the path, but it will hit an ImportError
if you try it. To my mind, that's not what people expect or want. Certainly
I'd prefer to just be able to start IPython without having to install it
again.

@takluyver
Member

Fixed, as much as it's going to be for now, by PR #1388, so I'm closing this again.

@takluyver takluyver closed this Apr 23, 2012
@ghost
ghost commented Aug 2, 2012

A helpful tip for others on Linux who find the version of IPython shipped in the distro now fails with importerror. Virtualenv changed the default behaviour from --system-site-packages to --no-site-packages. For IPython this means that when you run it from a terminal where the virtualenv is in your path, it will not pick up the ipython libs. Either install IPython in your virtualenv, or make your virtualenv use the system-site-packages.

@takluyver
Member

If your systemwide IPython is installed with a shebang line like #!/usr/bin/python (not #!/usr/bin/env python), it should basically work, even with the new default.

I like to use --system-site-packages anyway, because it's a pain to install things like numpy and matplotlib repeatedly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment