Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Python 3.8 with Julia 1.3 #752

Closed
darrencl opened this issue Mar 5, 2020 · 9 comments · Fixed by #753
Closed

Python 3.8 with Julia 1.3 #752

darrencl opened this issue Mar 5, 2020 · 9 comments · Fixed by #753

Comments

@darrencl
Copy link
Contributor

darrencl commented Mar 5, 2020

I couldn't build PyCall with Python 3.8.0 and Julia 1.3.0. This happens with PyCall v1.91.4

julia> Pkg.build("PyCall")
  Building Conda ─→ `~/.julia/packages/Conda/3rPhK/deps/build.log`
  Building PyCall → `~/.julia/packages/PyCall/zqDXB/deps/build.log`
┌ Error: Error building `PyCall`: 
│ ERROR: LoadError: Couldn't find libpython; check your PYTHON environment variable.
│ 
│ The python executable we tried was /home/darren/miniconda3/bin/python3.8 (= version 3.7).
│ Re-building with
│     ENV["PYCALL_DEBUG_BUILD"] = "yes"
│ may provide extra information for why it failed.
│ 
│ Stacktrace:
│  [1] error(::String) at ./error.jl:33
│  [2] #find_libpython#3(::typeof(Libdl.dlopen), ::typeof(find_libpython), ::String) at /home/darren/.julia/packages/PyCall/zqDXB/deps/buildutils.jl:72
│  [3] find_libpython(::String) at /home/darren/.julia/packages/PyCall/zqDXB/deps/buildutils.jl:41
│  [4] top-level scope at /home/darren/.julia/packages/PyCall/zqDXB/deps/build.jl:87
│  [5] include at ./boot.jl:328 [inlined]
│  [6] include_relative(::Module, ::String) at ./loading.jl:1105
│  [7] include(::Module, ::String) at ./Base.jl:31
│  [8] include(::String) at ./client.jl:424
│  [9] top-level scope at none:5
│ in expression starting at /home/darren/.julia/packages/PyCall/zqDXB/deps/build.jl:43
└ @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Pkg/src/backwards_compatible_isolation.jl:649
false

Can confirm the Python path is correct:

(base) darren@darren-GK5CN5Z-APEX-15:~$ /home/darren/miniconda3/bin/python3.8
Python 3.8.0 (default, Oct 24 2019, 15:04:43) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

Saw this issue, so PyCall seems to support Python 3.8 already? If so, why am I getting this? Is there any workaround?

Thanks!
@tkf
Copy link
Member

tkf commented Mar 6, 2020

I think this line indicates there was something wrong (python3.8 vs version 3.7):

The python executable we tried was /home/darren/miniconda3/bin/python3.8 (= version 3.7).

Did you try

│ Re-building with
│     ENV["PYCALL_DEBUG_BUILD"] = "yes"
│ may provide extra information for why it failed.

? What was in the build log? The path of the log file is printed as Building PyCall → `~/.julia/packages/PyCall/zqDXB/deps/build.log`.

@darrencl
Copy link
Contributor Author

darrencl commented Mar 6, 2020 •
edited

@tkf It seems looking for libpython

julia> Pkg.build("PyCall")
  Building Conda ─→ `~/.julia/packages/Conda/3rPhK/deps/build.log`
  Building PyCall → `~/.julia/packages/PyCall/zqDXB/deps/build.log`
┌ Error: Error building `PyCall`: 
│ DEBUG is_windows = False
│ DEBUG is_apple = False
│ DEBUG Candidate: None
│ DEBUG Not found.
│ DEBUG Candidate: /home/darren/miniconda3/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7m.so
│ DEBUG Not found.
│ DEBUG Candidate: /home/darren/miniconda3/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.7.so
│ DEBUG Not found.
│ DEBUG Candidate: /home/darren/miniconda3/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython3.so
│ DEBUG Not found.
│ DEBUG Candidate: /home/darren/miniconda3/lib/python3.7/config-3.7m-x86_64-linux-gnu/libpython.so
│ DEBUG Not found.
│ DEBUG Candidate: /home/darren/miniconda3/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.7m.so
│ DEBUG Not found.
│ DEBUG Candidate: /home/darren/miniconda3/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.7.so
│ DEBUG Not found.
│ DEBUG Candidate: /home/darren/miniconda3/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.so
│ DEBUG Not found.
│ DEBUG Candidate: /home/darren/miniconda3/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython.so
│ DEBUG Not found.
│ DEBUG Candidate: /home/darren/miniconda3/lib/libpython3.7m.so
│ DEBUG Not found.
│ DEBUG Candidate: /home/darren/miniconda3/lib/libpython3.7.so
│ DEBUG Not found.
│ DEBUG Candidate: /home/darren/miniconda3/lib/libpython3.so
│ DEBUG Not found.
│ DEBUG Candidate: /home/darren/miniconda3/lib/libpython.so
│ DEBUG Not found.
│ DEBUG Candidate: /home/darren/miniconda3/libpython3.7m.so
│ DEBUG Not found.
│ DEBUG Candidate: /home/darren/miniconda3/libpython3.7.so
│ DEBUG Not found.
│ DEBUG Candidate: /home/darren/miniconda3/libpython3.so
│ DEBUG Not found.
│ DEBUG Candidate: /home/darren/miniconda3/libpython.so
│ DEBUG Not found.
│ DEBUG Candidate: libpython3.7m.so.1.0
│ DEBUG Not found.
│ dlopen(libpython3.7m) ==> ErrorException("could not load library \"libpython3.7m\"\nlibpython3.7m.so: cannot open shared object file: No such file or directory")
│ dlopen(libpython3.7) ==> ErrorException("could not load library \"libpython3.7\"\nlibpython3.7.so: cannot open shared object file: No such file or directory")
│ dlopen(libpython3) ==> ErrorException("could not load library \"libpython3\"\nlibpython3.so: cannot open shared object file: No such file or directory")
│ dlopen(libpython) ==> ErrorException("could not load library \"libpython\"\nlibpython.so: cannot open shared object file: No such file or directory")
│ ERROR: LoadError: Couldn't find libpython; check your PYTHON environment variable.
│ 
│ The python executable we tried was /home/darren/miniconda3/bin/python3.8 (= version 3.7).
│ Re-building with
│     ENV["PYCALL_DEBUG_BUILD"] = "yes"
│ may provide extra information for why it failed.
│ 
│ Stacktrace:
│  [1] error(::String) at ./error.jl:33
│  [2] #find_libpython#3(::typeof(Libdl.dlopen), ::typeof(find_libpython), ::String) at /home/darren/.julia/packages/PyCall/zqDXB/deps/buildutils.jl:72
│  [3] find_libpython(::String) at /home/darren/.julia/packages/PyCall/zqDXB/deps/buildutils.jl:41
│  [4] top-level scope at /home/darren/.julia/packages/PyCall/zqDXB/deps/build.jl:87
│  [5] include at ./boot.jl:328 [inlined]
│  [6] include_relative(::Module, ::String) at ./loading.jl:1105
│  [7] include(::Module, ::String) at ./Base.jl:31
│  [8] include(::String) at ./client.jl:424
│  [9] top-level scope at none:5
│ in expression starting at /home/darren/.julia/packages/PyCall/zqDXB/deps/build.jl:43
└ @ Pkg.Operations /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Pkg/src/backwards_compatible_isolation.jl:649
false

I think this is same issue as #565, which means the libpython finder needs to be updated.

FYI, this is my libpython paths

(base) darren@darren-GK5CN5Z-APEX-15:~$ find /home/darren/miniconda3/ -type f -name "*libpython*"
/home/darren/miniconda3/pkgs/python-3.8.0-h0371630_0/lib/libpython3.8.so.1.0
/home/darren/miniconda3/pkgs/python-3.8.0-h0371630_0/lib/libpython3.8.a
/home/darren/miniconda3/pkgs/python-3.7.4-h265db76_1/lib/libpython3.7m.so.1.0
/home/darren/miniconda3/pkgs/python-3.7.4-h265db76_1/lib/libpython3.7m.a
/home/darren/miniconda3/lib/libpython3.8.so.1.0
/home/darren/miniconda3/lib/libpython3.8.a

@darrencl
Copy link
Contributor Author

darrencl commented Mar 6, 2020

And of course, the hacky workaround for this is to create a symlink to one of those expected path, e.g.

sudo ln -s /home/darren/miniconda3/lib/libpython3.8.so /home/darren/miniconda3/libpython.so

@tkf
Copy link
Member

tkf commented Mar 7, 2020

This is puzzling. It's trying to find libpython3.7m.so under lib/python3.8...:

│ DEBUG Candidate: /home/darren/miniconda3/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.7m.so
│ DEBUG Not found.
│ DEBUG Candidate: /home/darren/miniconda3/lib/python3.8/config-3.8-x86_64-linux-gnu/libpython3.7.so
│ DEBUG Not found.

Did you use Python 3.7 before? Maybe there is a path to Python 3.7 in ~/.julia/prefs/PyCall?

@darrencl
Copy link
Contributor Author

darrencl commented Mar 8, 2020

@tkf Never used Python 3.7 in this machine and there is also no path to Python 3.7.

(base) darren@darren-GK5CN5Z-APEX-15:~$ cat ~/.julia/prefs/PyCall 
/home/darren/miniconda3/bin/python3.8(base) darren@darren-GK5CN5Z-APEX-15:~$

@tkf
Copy link
Member

tkf commented Mar 8, 2020

You have ~/miniconda3/pkgs/python-3.7.4-h265db76_1/ so miniconda definitely downloaded Python 3.7 in your machine. Though I don't know how PyCall's build script would be confused by it.

@darrencl
Copy link
Contributor Author

darrencl commented Mar 9, 2020

@tkf You're right! Hmm, anything I can help to investigate this?

@tkf
Copy link
Member

tkf commented Mar 9, 2020

This part of the build is just a Python script: https://github.com/JuliaPy/PyCall.jl/blob/master/deps/find_libpython.py

So if you want to debug it, you can run it with python find_libpython.py. See --help for what it does. Passing --verbose --list-all may print something useful.

@darrencl darrencl mentioned this issue Mar 9, 2020
Merged
@darrencl
Copy link
Contributor Author

darrencl commented Mar 9, 2020 •
edited

@tkf Found the issue, it seems that for some reason the python version in config var sysconfig.get_config_var("VERSION") is not updated (More info in the PR). This might happen to someone else too, so I think using config var only as last resort will fix this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Improving find libpython darrencl/PyCall.jl
2 participants
@tkf @darrencl