PyCall.jl setting incorrect libpython on Julia 1.9

[MilesCranmer]

I can't seem to get PyJulia working on Julia 1.9 (beta 4), although it works fine on 1.8.5. It seems to be some issue with the Python interpreter and libpython set by PyCall.jl. It could be interference from Conda.jl's installed copy of Python and libpython?

First, I set up a completely fresh Julia install with juliaup (note that this code will delete your ~/.julia folder, only saving startup.jl):

mv ~/.julia/config/startup.jl ~/startup.jl && \
    rm -rf ~/.julia && \
    mkdir -p ~/.julia/config && \
    mv ~/startup.jl ~/.julia/config/startup.jl && \
    juliaup add 1.9 && \
    juliaup default 1.9 && \
    julia --startup-file=no -e 'using Pkg; Pkg.add(["OhMyREPL", "Revise"])'

Then, I run:

python -c 'import julia; julia.install()'

Which succeeds. However, when I try to start Julia via PyJulia:

python -c 'from julia import Main'

despite the exact same set of environment variables as used in the `install`, this fails:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 672, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 632, in _load_backward_compatible
  File "/Users/mcranmer/venvs/main/lib/python3.10/site-packages/julia/core.py", line 247, in load_module
    JuliaMainModule(self, fullname))
  File "/Users/mcranmer/venvs/main/lib/python3.10/site-packages/julia/core.py", line 149, in __init__
    self._julia = loader.julia
  File "/Users/mcranmer/venvs/main/lib/python3.10/site-packages/julia/core.py", line 239, in julia
    self.__class__.julia = julia = Julia()
  File "/Users/mcranmer/venvs/main/lib/python3.10/site-packages/julia/core.py", line 489, in __init__
    raise UnsupportedPythonError(jlinfo)
julia.core.UnsupportedPythonError: It seems your Julia and PyJulia setup are not supported.

Julia executable:
    julia
Python interpreter and libpython used by PyCall.jl:
    /Users/mcranmer/.julia/conda/3/aarch64/bin/python
    /Users/mcranmer/.julia/conda/3/aarch64/lib/libpython3.10.dylib
Python interpreter used to import PyJulia and its libpython.
    /Users/mcranmer/venvs/main/bin/python
    /opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/Python

In Julia >= 0.7, above two paths to `libpython` have to match exactly
in order for PyJulia to work out-of-the-box.  To configure PyCall.jl to use
Python interpreter "/Users/mcranmer/venvs/main/bin/python",
run the following code in the Python REPL:

    >>> import julia
    >>> julia.install()

For more information, see:

    https://pyjulia.readthedocs.io/en/latest/troubleshooting.html

In particular, the Python interpreter and libpython seem to be set incorrectly:

Python interpreter and libpython used by PyCall.jl:
    /Users/mcranmer/.julia/conda/3/aarch64/bin/python
    /Users/mcranmer/.julia/conda/3/aarch64/lib/libpython3.10.dylib
Python interpreter used to import PyJulia and its libpython.
    /Users/mcranmer/venvs/main/bin/python
    /opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/Python

I'm assuming these new python and libpython are packaged by the PyCall.jl install. But I think ideally we would want them to be set by the user from their Python/libpython, for compatibility with Python libraries, right?

I can run this entire workflow with juliaup add 1.8 && juliaup default 1.8 instead, and it works fine.

cc @mkitti @ngam @stevengj @marius311

---

edit: I should mention I'm not using conda at all. This is a homebrew-installed version of Python running in a virtualenv.

[mkitti]

What environment variables are being used?

Unless you are using this activate script from the conda-forge julia-feedstock Conda.jl will have no idea where to find Python and will default to creating its own conda based install.
https://github.com/conda-forge/julia-feedstock/blob/41f00ab1461dcc29ff22858019094c6c95e0a9bb/recipe/scripts/activate.sh#L30-L33

export CONDA_JL_HOME_BACKUP=${CONDA_JL_HOME:-}
export CONDA_JL_HOME=$CONDA_PREFIX
export CONDA_JL_CONDA_EXE_BACKUP=${CONDA_JL_CONDA_EXE:-}
export CONDA_JL_CONDA_EXE=$CONDA_EXE

[MilesCranmer]

I'm not using conda at all. This is a homebrew-installed version of Python running in a virtualenv. In other words:

> ls -lath $(which python3.10)
lrwxr-xr-x  1 mcranmer  staff    44B Nov  8 11:07 /Users/mcranmer/venvs/main/bin/python3.10 -> /opt/homebrew/opt/python@3.10/bin/python3.10

[mkitti]

One the following is needed:

1. The environment variable PYTHON needs to point to the python executable.
2. A prefs file in the depot needs to contain the location of python executable: ~/.julia/prefs/PyCall
3. which can find python or python3
   https://github.com/JuliaPy/PyCall.jl/blob/bcaba00d1e2c412b2f61d33343ef5a9ab1b9258a/deps/build.jl#L45-L47

[MilesCranmer]

I should emphasize that the exact same workflow with the exact same python environment works on Julia 1.8.5; it's only Julia 1.9.0 that generates this error.

When I check the env: which python is correct (for both 1.8.5 and 1.9.0). PYTHON is not set.

[MilesCranmer]

Weird. The contents of prefs/PyCall differ when between versions.

For Julia 1.9.0:

> cat ~/.julia/prefs/PyCall
Conda

and for Julia 1.8.5:

> cat ~/.julia/prefs/PyCall
/Users/mcranmer/venvs/main/bin/python

Again, this is the exact same Python environment and workflow. The entire .julia folder is deleted each time I try this.

[mkitti]

Try adding https://github.com/JuliaPy/Conda.jl first and see if you can detect any differences. Step through PyCall.jl's deps.jl and see if you can identify the difference.

[MilesCranmer]

FYI the output of

 py = get(ENV, "PYTHON", isfile(prefsfile) ? readchomp(prefsfile) :
                            (Sys.isunix() && !Sys.isapple()) ?
                            whichfirst("python3", "python") : "Conda")

will return "Conda" on both 1.8.5 and 1.9.0, because I'm on macOS. Not sure how the rest differs though. It couldn't be something to do with precompilation caching right??

[MilesCranmer]

Interesting. When I build PyCall.jl manually, even on 1.8.5, it uses the wrong Python library (I think this Sys.isunix() && !Sys.isapple() is a bug and should really be Sys.isunix()?). But somehow PyJulia is able to save it, maybe by passing the PYTHON variable?

But on 1.9.0, PyJulia can't save it for whatever reason.

[mkitti]

I'm guessing that at some point there was an issue on macOS with it finding a system python.

Keep working through the file. Something else must differ by the time it gets to line 114.

[MilesCranmer]

I don't see any differences if I just run in Julia though. They (1.8.5 and 1.9.0) both fail to find the system Python. Somehow by running it through PyJulia, it succeeds in finding it; maybe some env variable being passed from PyJulia?

But this stops being true on 1.9.0.

[mkitti]

Hmm, something has gone wrong here then:

pyjulia/src/julia/tools.py

Lines 55 to 134 in 1e3de7b

	def build_pycall(julia="julia", python=sys.executable, **kwargs):
	# Passing `python` to force build (OP="build")
	install(julia=julia, python=python, **kwargs)
	def install(julia="julia", color="auto", python=None, quiet=False):
	"""
	install(*, julia="julia", color="auto")
	Install Julia packages required by PyJulia in `julia`.
	This function installs and/or re-builds PyCall if necessary. It
	also makes sure to build PyCall in a way compatible with this
	Python executable (if possible).
	Keyword Arguments
	-----------------
	julia : str
	Julia executable (default: "julia")
	color : "auto", False or True
	Use colorful output if `True`. "auto" (default) to detect it
	automatically.
	"""
	if which(julia) is None:
	raise JuliaNotFound(julia, kwargname="julia")
	libpython = linked_libpython() or ""
	julia_cmd = [julia, "--startup-file=no"]
	if quiet:
	color = False
	if color == "auto":
	color = sys.stdout.isatty()
	if color:
	# `--color=auto` doesn't work?
	julia_cmd.append("--color=yes")
	"""
	if _julia_version(julia) >= (1, 1):
	julia_cmd.append("--color=auto")
	else:
	julia_cmd.append("--color=yes")
	"""
	OP = "build" if python else "install"
	install_cmd = julia_cmd + [
	os.path.join(os.path.dirname(os.path.realpath(__file__)), "install.jl"),
	"--",
	OP,
	python or sys.executable,
	libpython,
	]
	kwargs = {}
	if quiet:
	kwargs.update(
	stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True
	)
	proc = subprocess.Popen(install_cmd, **kwargs)
	output, _ = proc.communicate()
	returncode = proc.returncode
	if returncode == 113: # code_no_precompile_needed
	return
	elif returncode != 0:
	raise PyCallInstallError("Installing", output)
	if not quiet:
	print(file=sys.stderr)
	print("Precompiling PyCall...", file=sys.stderr)
	sys.stderr.flush()
	precompile_cmd = julia_cmd + ["-e", "using PyCall"]
	returncode = subprocess.call(precompile_cmd)
	if returncode != 0:
	raise PyCallInstallError("Precompiling")
	if not quiet:
	print("Precompiling PyCall... DONE", file=sys.stderr)
	print("PyCall is installed and built successfully.", file=sys.stderr)
	if julia != "julia":
	print(file=sys.stderr)
	print(_non_default_julia_warning_message(julia), file=sys.stderr)
	sys.stderr.flush()

[mkitti]

or here:
https://github.com/JuliaPy/pyjulia/blob/master/src/julia/install.jl

[MilesCranmer]

tools.py looks okay:

Julia 1.8.5:

install_cmd=['julia', '--startup-file=no', '--color=yes', '/Users/mcranmer/venvs/main/lib/python3.10/site-packages/julia-0.6.0-py3.10.egg/julia/install.jl', '--', 'install', '/Users/mcranmer/venvs/main/bin/python', '/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/Python']

1.9.0:

install_cmd=['julia', '--startup-file=no', '--color=yes', '/Users/mcranmer/venvs/main/lib/python3.10/site-packages/julia-0.6.0-py3.10.egg/julia/install.jl', '--', 'install', '/Users/mcranmer/venvs/main/bin/python', '/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/Python']

so the correct paths are being passed to install.jl.

[MilesCranmer]

Alright well that is a really nasty bug. Here's the problem:

pyjulia/src/julia/install.jl

Line 1 in 1e3de7b

OP, python, libpython = ARGS

The behavior of this line changed from Julia 1.8 to 1.9. Here's the contents of ARGS in Julia <1.8:

["install", "/Users/mcranmer/venvs/main/bin/python", "/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/Python"]

And here's the contents ARGS in 1.9+:

["--", "install", "/Users/mcranmer/venvs/main/bin/python", "/opt/homebrew/Cellar/python@3.10/3.10.9/Frameworks/Python.framework/Versions/3.10/Python"]

meaning that the following variables were loaded:

OP: --
python: install
libpython: /Users/mcranmer/venvs/main/bin/python

i.e., it looks like -- is being interpreted as separate arg whereas before it was ignored in ARGS (is that supposed to change in Julia 1.9? Seems like a breaking change...). So then PyCall.jl just ignores those incorrect paths and creates a new conda folder.

[MilesCranmer]

It looks like this change was made to Julia here: JuliaLang/julia#45335

[mkitti]

So the issue is in install.jl?

[MilesCranmer]

I wouldn't call it an issue per se, it's more a breaking change in 1.9 in how Julia parses CLI arguments. A quick fix to the first line of install.jl would be:

-OP, python, libpython = ARGS
+OP, python, libpython = if VERSION < v"1.9.0"
+    ARGS
+else
+    ARGS[2:end]
+end