most @import fail in virtualenv

[lucasb-eyer]

Using PyCall with the globally installed python works flawlessly. But I mainly work in virtualenvs, and would like julia to interact with my virtualenv's python.

I hoped just activating the virtualenv would be enough, but @pyimport numpy (and others) won't work when a virtualenv (in the following at /home/beyer/sci-env3-dyn) is activated.

First, everything works with python in the virtualenv:

$ python
Python 3.3.3 (default, Dec 20 2013, 23:51:18) 
[GCC 4.6.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.executable
'/home/beyer/sci-env3-dyn/bin/python'
>>> sys.path
['', '/home/beyer/sci-env3-dyn/lib/python33.zip', '/home/beyer/sci-env3-dyn/lib/python3.3', '/home/beyer/sci-env3-dyn/lib/python3.3/plat-linux', '/home/beyer/sci-env3-dyn/lib/python3.3/lib-dynload', '/home/beyer/insts/lib/python3.3', '/home/beyer/insts/lib/python3.3/plat-linux', '/home/beyer/sci-env3-dyn/lib/python3.3/site-packages']
>>> import numpy
>>>

Now, through julia called while inside the virtualenv:

$ ~/insts/bin/julia-readline 
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "help()" to list help topics
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.3.0-prerelease+219 (2013-11-28 17:05 UTC)
 _/ |\__'_|_|_|\__'_|  |  Commit d114152 (0 days old master)
|__/                   |  x86_64-linux-gnu

julia> using PyCall

julia> @pyimport sys

julia> sys.executable
""

julia> sys.path
5-element Array{Any,1}:
 "/home/beyer/insts/lib/python33.zip"           
 "/home/beyer/insts/lib/python3.3"              
 "/home/beyer/insts/lib/python3.3/plat-linux"   
 "/home/beyer/insts/lib/python3.3/lib-dynload"  
 "/home/beyer/insts/lib/python3.3/site-packages"

julia> @pyimport numpy
ERROR: PyError (PyImport_ImportModule) <class 'ImportError'>
ImportError("No module named 'numpy'",)

 in pyerr_check at /home/beyer/.julia/PyCall/src/exception.jl:58
 in pyimport at /home/beyer/.julia/PyCall/src/PyCall.jl:85

julia> 

So, for some reason, when called through julia, neither sys.executable nor sys.path are correctly set. What is in sys.path (namely, stuff inside /home/beyer/insts/lib/python3.3) is the python install which has been used to create the virtualenv. I think pycall should "just work" when a virtualenv is active.

I don't really know how to debug this further, but in the following are some env-variables which might help debugging:

$ env | grep -i py
(nothing)
$ env | grep -i path
LD_LIBRARY_PATH=/home/beyer/sci-env3-dyn/lib:/opt/ros/groovy/lib
PATH=/home/beyer/sci-env3-dyn/bin:/opt/ros/groovy/bin:/usr/lib/lightdm/lightdm:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/lib/go/bin
$ env | grep -i virtual
VIRTUAL_ENV=/home/beyer/sci-env3-dyn

[lucasb-eyer]

After further inspection, it seems to me that the @windows_only handling of exec_prefix is the culprit. In a virtualenv, exec_prefix points to the virtualenv's root (in my case /home/beyer/sci-env3-dyn) and thus, if the @windows_only check is removed, sys.path is exactly the same within julia as it is within python itself, allowing me to @pyimport all the python modules I installed in the virtualenv.

Regarding the empty string value of sys.executable, this seems to be the expected behaviour when sys.argv[0] is empty, and unrelated to this issue anyways.

So the question is: why was this block made @windows_only in the first place?

• It has been introduced by 0fad204, which only talks about windows.
• It's using sys.exec_prefix (through distutils.sysconfig.EXEC_PREFIX) which, according to the documentation, takes care of virtualenvs and is not OS-specific.
• It's later been expanded by 62cf664, which, again, doesn't seem windows-specific to me.

Am I missing anything, or can we just remove that @windows_only check and have things working correctly with virtualenvs?

[stevengj]

Yes, making that @windows_only was probably an excess of caution. Probably we should set PYTHONHOME to prefix:exec_prefix, though?