Module import causes errors in julia's shell mode

[jdlangs]

I've discovered things break if I try to interface with mercurial through Julia's shell commands (using ; in the REPL). I don't know enough to know what's going on behind the scenes, but somehow invoking external python software in the shell mode is being influenced by PyCall's behavior

Example:

shell> hg identify
7fddeb3f5258+ tip

julia> import PyCall

julia> PyCall.@pyimport sys

shell> hg identify
ImportError: No module named site

I'm using the most recent version of Julia, 0.4.0-dev+719 (2014-09-21)

[stevengj]

What operating system and Python distro are you using?

Probably the problem is that PyCall is setting the PYTHONPATH environment variable (ENV["PYTHONPATH"]), but hg uses a different version of Python and wants a different path.

[jdlangs]

I'm on Ubuntu. The environment variable does seem to be the issue. I forgot I was running my Julia REPL within a python virtual environment and after doing the PyCall import, the variable PYTHONHOME gets set to the virtual env root whereas it was unset before (PYTHONPATH is always unset). Mercurial does not break if I run Julia from just the normal environment.

I guess there's not much of an option to change it unless PyCall doesn't need one of those variables set to work properly?

[stevengj]

I'm not sure if there is any clean way to fix it. The virtualenv Python you are using with PyCall and hg require different environments; the PYTHONHOME is required to tell libpython to use packages in the virtualenv path, I think.

[stevengj]

...unless there is a way to set the equivalent of PYTHONHOME in libpython without using an environment variable?

[stevengj]

Hmm, one hacky possibility could be to set PYTHONHOME, initialize Python, call Py_GetPath to force Python to compute its paths, and then unset/restore PYTHONHOME.

This should work because, looking at the source code, Python only computes its search paths once and then caches the value; it never looks at PYTHONHOME again, it seems.

[jdlangs]

I found if I remove PYTHONHOME and put the same contents in PYTHONPATH it works fine. Would it be possible to work with the path variable instead?

[jdlangs]

Yeah, it looks like if I change the PYTHONHOME references to PYTHONPATH in PyCall.jl, the directories that appear in sys.path are different.

I'll go ahead and close it, since there's no obvious change and it's easy enough to just do pop!(ENV, "PYTHONHOME") anytime system libraries need to be used in a virtual environment.

[stevengj]

No, I don't think it is safe to just set PYTHONPATH in general. The reason is that PYTHONPATH is just the absolute path of the modules, whereas Python is more clever about searching for paths relative to PYTHONHOME to find the modules. The two are not interchangeable.

I'm going to re-open this issue; I think that the best thing is probably to unset PYTHONHOME after calling Py_GetPath.

[jdlangs]

Additionally, it's not even a specific PyCall or Julia issue. If I set PYTHONHOME as an environment variable in the regular shell to the virtual environment root, mercurial stops working.

I guess a fix should be theoretically possible though because the python interpreter in the virtual environment knows to search those paths as well, and it doesn't need to set any environment variables.

[stevengj]

It looks like Py_GetPath is called by Py_InitializeEx when Python is initialized, so it should be sufficient to only set PYTHONHOME while calling this function. I'll fix this as part of #167.

[jdlangs]

Thanks for the followup!