Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Can't re-build PyCall.jl after upgrading Python #222

Closed
MilesCranmer opened this issue Nov 8, 2022 · 9 comments
Closed

[BUG] Can't re-build PyCall.jl after upgrading Python #222

MilesCranmer opened this issue Nov 8, 2022 · 9 comments
Labels
bug Something isn't working

Comments

@MilesCranmer
Copy link
Owner

MilesCranmer commented Nov 8, 2022

Just documenting this in case others encounter it.

This is a weird bug often reported on PyCall.jl where PyCall will somehow cache the location of your old Python binary, and re-building will fail.

I downgraded my Python version from 3.11 to 3.10, and tried to do python -c 'import pysr; pysr.install()'. This gave me the error:

┌ Info: PyCall is already installed but not compatible with this Python
└ executable.  Re-building PyCall...
[ Info: Run `Pkg.build("PyCall"; verbose=true)`
ERROR: LoadError: The following package names could not be resolved:
 * PyCall (not found in project or manifest)
Stacktrace:
 [1] pkgerror(msg::String)
   @ Pkg.Types ~/.julia/juliaup/julia-1.8.2+0.aarch64/share/julia/stdlib/v1.8/Pkg/src/Types.jl:67
 [2] ensure_resolved(ctx::Pkg.Types.Context, manifest::Pkg.Types.Manifest, pkgs::Vector{Pkg.Types.PackageSpec}; registry::Bool)
   @ Pkg.Types ~/.julia/juliaup/julia-1.8.2+0.aarch64/share/julia/stdlib/v1.8/Pkg/src/Types.jl:952
 [3] ensure_resolved
   @ ~/.julia/juliaup/julia-1.8.2+0.aarch64/share/julia/stdlib/v1.8/Pkg/src/Types.jl:904 [inlined]
 [4] build(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; verbose::Bool, kwargs::Base.Pairs{Symbol, Base.TTY, Tuple{Symbol}, NamedTuple{(:io,), Tuple{Base.TTY}}})
   @ Pkg.API ~/.julia/juliaup/julia-1.8.2+0.aarch64/share/julia/stdlib/v1.8/Pkg/src/API.jl:1028
 [5] build(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:verbose,), Tuple{Bool}}})
   @ Pkg.API ~/.julia/juliaup/julia-1.8.2+0.aarch64/share/julia/stdlib/v1.8/Pkg/src/API.jl:156
 [6] #build#99
   @ ~/.julia/juliaup/julia-1.8.2+0.aarch64/share/julia/stdlib/v1.8/Pkg/src/API.jl:144 [inlined]
 [7] #build#98
   @ ~/.julia/juliaup/julia-1.8.2+0.aarch64/share/julia/stdlib/v1.8/Pkg/src/API.jl:143 [inlined]
 [8] build_pycall()
   @ Main ~/venvs/main/lib/python3.10/site-packages/julia/install.jl:54
 [9] top-level scope
   @ ~/venvs/main/lib/python3.10/site-packages/julia/install.jl:108
in expression starting at /Users/mcranmer/venvs/main/lib/python3.10/site-packages/julia/install.jl:73
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/mcranmer/Documents/PySR/pysr/julia_helpers.py", line 79, in install
    julia.install(quiet=quiet)
  File "/Users/mcranmer/venvs/main/lib/python3.10/site-packages/julia/tools.py", line 118, in install
    raise PyCallInstallError("Installing", output)
julia.tools.PyCallInstallError: Installing PyCall failed.

** Important information from Julia may be printed before Python's Traceback **

Some useful information may also be stored in the build log file
`~/.julia/packages/PyCall/*/deps/build.log`.

If I open up Julia, install PyCall.jl into any environment, and write:

julia> using PyCall

julia> PyCall.python
"/Users/mcranmer/venvs/main/bin/python3.11"

it gives me the old location of the Python binary! The only one in that folder is now python3.10 - the python3.11 was removed. This is even true if I uninstall and re-install PyCall.jl. It will somehow remember the old location, and re-building it will fail. Maybe it expects PyCall.python to be unchanged between Python versions?

The way I ended up fixing this is to run (inside Julia)

using Pkg
ENV["PYTHON"] = "/Users/mcranmer/venvs/main/bin/python3.10"
pkg"build PyCall"

this successfully re-builds PyCall.jl. But I think ideally this shouldn't be necessary, to lower friction for users. To do this, I think PyJulia should check what Python binary was used to launch it, and, if necessary, reset ENV and rebuild.

FYI @mkitti @stevengj

@MilesCranmer MilesCranmer added the bug Something isn't working label Nov 8, 2022
@MilesCranmer MilesCranmer self-assigned this Nov 8, 2022
@mkitti
Copy link
Contributor

mkitti commented Nov 8, 2022

This is probably due to a stale deps.jl file. We should probably move to a Preferences.jl mechanism for this or again take inspiration from PythonCall.jl.

@mkitti
Copy link
Contributor

mkitti commented Nov 8, 2022

You should be able to locate the deps file via this code.

using PyCall # repeat if this errors
readdir(joinpath(dirname(pathof(PyCall)), "..", "deps"))

@MilesCranmer
Copy link
Owner Author

Maybe this could be checked at julia.install(), and, if deps.jl's python key doesn't match the calling binary, it gets reset?

@mkitti
Copy link
Contributor

mkitti commented Nov 8, 2022

This would be much easier to handle via Preferences.jl. We should use that.

@MilesCranmer MilesCranmer removed their assignment Apr 20, 2023
@dbl001
Copy link

dbl001 commented May 5, 2023

Will 'PySr' run with Julia 1.9?

% ipython
Python 3.11.0 | packaged by conda-forge | (main, Oct 25 2022, 06:24:51) [Clang 14.0.4 ]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.7.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import pysr

In [2]: pysr.install()
[ Info: Julia version info
Julia Version 1.9.0-rc3
Commit 1853b903282 (2023-04-26 15:51 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin21.4.0)
  uname: Darwin 22.4.0 Darwin Kernel Version 22.4.0: Mon Mar  6 21:00:17 PST 2023; root:xnu-8796.101.5~3/RELEASE_X86_64 x86_64 i386
  CPU: Intel(R) Core(TM) i7-10700K CPU @ 3.80GHz: 
                 speed         user         nice          sys         idle          irq
       #1-16  3800 MHz    4265439 s          0 s     764858 s   16874798 s          0 s
  Memory: 128.0 GB (4417.48828125 MB free)
  Uptime: 274568.0 sec
  Load Avg:  7.181640625  7.73876953125  12.20849609375
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, skylake)
  Threads: 1 on 16 virtual cores
Environment:
  JULIA_DEPOT_PATH_BACKUP = 
  JULIA_PROJECT_BACKUP = 
  JULIA_LOAD_PATH_BACKUP = 
  JULIA_DEPOT_PATH = /Users/davidlaxer/anaconda3/envs/pysr_test/share/julia:/Users/davidlaxer/anaconda3/envs/pysr_test/share/pysr/depot:
  JULIA_PROJECT = @pysr-0.12.3
  JULIA_LOAD_PATH = @:@pysr_test:@stdlib
  JULIA_SSL_CA_ROOTS_PATH_BACKUP = 
  JULIA_CONDAPKG_BACKEND_BACKUP = 
  JULIA_CONDAPKG_BACKEND = System
  JULIA_CONDAPKG_EXE_BACKUP = 
  JULIA_CONDAPKG_EXE = /Users/davidlaxer/anaconda3/bin/conda
  JULIA_DEPOT_PATH_PYSR_BACKUP = /Users/davidlaxer/anaconda3/envs/pysr_test/share/julia:
  TERM = xterm-256color
  PATH = /Users/davidlaxer/.opam/_coq-platform_.2021.02.1/bin:/Users/davidlaxer/.juliaup/bin:/Users/davidlaxer/.cabal/bin:/Users/davidlaxer/.ghcup/bin:/Users/davidlaxer/anaconda3/envs/pysr_test/bin:/Users/davidlaxer/anaconda3/condabin:/opt/local/bin:/opt/local/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Users/davidlaxer/.cargo/bin:/Users/jetbrains/.local/bin
  XPC_FLAGS = 0x0
  HOME = /Users/davidlaxer
  JAVA_HOME = :-
  JAVA_LD_LIBRARY_PATH = :-
  CAML_LD_LIBRARY_PATH = /Users/davidlaxer/.opam/_coq-platform_.2021.02.1/lib/stublibs:/Users/davidlaxer/.opam/_coq-platform_.2021.02.1/lib/ocaml/stublibs:/Users/davidlaxer/.opam/_coq-platform_.2021.02.1/lib/ocaml
  OCAML_TOPLEVEL_PATH = /Users/davidlaxer/.opam/_coq-platform_.2021.02.1/lib/toplevel
  PKG_CONFIG_PATH = /Users/davidlaxer/.opam/_coq-platform_.2021.02.1/lib/pkgconfig:
  CONDA_BACKUP_CPPFLAGS = -D_FORTIFY_SOURCE=2 -isystem /Users/davidlaxer/anaconda3/include
  CONDA_BACKUP_CFLAGS = -march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -isystem /Users/davidlaxer/anaconda3/include
  CONDA_BACKUP_LDFLAGS = -Wl,-pie -Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,/Users/davidlaxer/anaconda3/lib -L/Users/davidlaxer/anaconda3/lib
  CONDA_BACKUP_LDFLAGS_LD = -pie -headerpad_max_install_names -dead_strip_dylibs -rpath /Users/davidlaxer/anaconda3/lib -L/Users/davidlaxer/anaconda3/lib
  CONDA_BACKUP_DEBUG_CFLAGS = -march=core2 -mtune=haswell -mssse3 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -Og -g -Wall -Wextra -isystem /Users/davidlaxer/anaconda3/include
  CONDA_BACKUP_CMAKE_PREFIX_PATH = :/Users/davidlaxer/anaconda3
  CONDA_BACKUP_FFLAGS = -march=core2 -mtune=haswell -ftree-vectorize -fPIC -fstack-protector -O2 -pipe -isystem /Users/davidlaxer/anaconda3/include
  CONDA_BACKUP_FORTRANFLAGS = -march=core2 -mtune=haswell -ftree-vectorize -fPIC -fstack-protector -O2 -pipe -isystem /Users/davidlaxer/anaconda3/include
  CONDA_BACKUP_DEBUG_FFLAGS = -march=core2 -mtune=haswell -ftree-vectorize -fPIC -fstack-protector -O2 -pipe -isystem /Users/davidlaxer/anaconda3/include
  CONDA_BACKUP_DEBUG_FORTRANFLAGS = -march=core2 -mtune=haswell -ftree-vectorize -fPIC -fstack-protector -O2 -pipe -isystem /Users/davidlaxer/anaconda3/include
  CONDA_JL_HOME_BACKUP = 
  CONDA_JL_HOME = /Users/davidlaxer/anaconda3/envs/pysr_test
[ Info: Julia executable: /Users/davidlaxer/anaconda3/envs/pysr_test/share/julia/juliaup/julia-1.9.0-rc3+0.x64.apple.darwin14/bin/julia
[ Info: Trying to import PyCall...
┌ Info: PyCall is already installed but not compatible with this Python
└ executable.  Re-building PyCall...
[ Info: Run `Pkg.build("PyCall"; verbose=true)`
ERROR: LoadError: The following package names could not be resolved:
 * PyCall (not found in project or manifest)
Stacktrace:
  [1] pkgerror(msg::String)
    @ Pkg.Types ~/anaconda3/envs/pysr_test/share/julia/juliaup/julia-1.9.0-rc3+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/Types.jl:69
  [2] ensure_resolved(ctx::Pkg.Types.Context, manifest::Pkg.Types.Manifest, pkgs::Vector{Pkg.Types.PackageSpec}; registry::Bool)
    @ Pkg.Types ~/anaconda3/envs/pysr_test/share/julia/juliaup/julia-1.9.0-rc3+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/Types.jl:995
  [3] ensure_resolved
    @ ~/anaconda3/envs/pysr_test/share/julia/juliaup/julia-1.9.0-rc3+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/Types.jl:946 [inlined]
  [4] build(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; verbose::Bool, kwargs::Base.Pairs{Symbol, Base.TTY, Tuple{Symbol}, NamedTuple{(:io,), Tuple{Base.TTY}}})
    @ Pkg.API ~/anaconda3/envs/pysr_test/share/julia/juliaup/julia-1.9.0-rc3+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/API.jl:1052
  [5] build(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Pairs{Symbol, Bool, Tuple{Symbol}, NamedTuple{(:verbose,), Tuple{Bool}}})
    @ Pkg.API ~/anaconda3/envs/pysr_test/share/julia/juliaup/julia-1.9.0-rc3+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/API.jl:156
  [6] #build#85
    @ ~/anaconda3/envs/pysr_test/share/julia/juliaup/julia-1.9.0-rc3+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/API.jl:144 [inlined]
  [7] build
    @ ~/anaconda3/envs/pysr_test/share/julia/juliaup/julia-1.9.0-rc3+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/API.jl:144 [inlined]
  [8] #build#84
    @ ~/anaconda3/envs/pysr_test/share/julia/juliaup/julia-1.9.0-rc3+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/API.jl:143 [inlined]
  [9] build
    @ ~/anaconda3/envs/pysr_test/share/julia/juliaup/julia-1.9.0-rc3+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/API.jl:143 [inlined]
 [10] build_pycall()
    @ Main ~/anaconda3/envs/pysr_test/lib/python3.11/site-packages/julia/install.jl:54
 [11] top-level scope
    @ ~/anaconda3/envs/pysr_test/lib/python3.11/site-packages/julia/install.jl:108
in expression starting at /Users/davidlaxer/anaconda3/envs/pysr_test/lib/python3.11/site-packages/julia/install.jl:73
---------------------------------------------------------------------------
PyCallInstallError                        Traceback (most recent call last)
Cell In[2], line 1
----> 1 pysr.install()

File ~/PySR/pysr/julia_helpers.py:84, in install(julia_project, quiet, precompile)
     81 if precompile == False:
     82     os.environ["JULIA_PKG_PRECOMPILE_AUTO"] = "0"
---> 84 julia.install(quiet=quiet)
     85 Main, init_log = init_julia(julia_project, quiet=quiet, return_aux=True)
     86 io_arg = _get_io_arg(quiet)

File ~/anaconda3/envs/pysr_test/lib/python3.11/site-packages/julia/tools.py:118, in install(julia, color, python, quiet)
    116     return
    117 elif returncode != 0:
--> 118     raise PyCallInstallError("Installing", output)
    120 if not quiet:
    121     print(file=sys.stderr)

PyCallInstallError: Installing PyCall failed.

** Important information from Julia may be printed before Python's Traceback **

Some useful information may also be stored in the build log file
`~/.julia/packages/PyCall/*/deps/build.log`.


In [3]: quit

@MilesCranmer
Copy link
Owner Author

Yes, it works for me. Just follow the instructions here: #257

@dbl001
Copy link

dbl001 commented May 5, 2023

Works for me now. Thanks!

@MilesCranmer
Copy link
Owner Author

Cool. Yes I would really like to have that bug automatically solved by PySR, but I'm not sure how to do it so far.

@MilesCranmer
Copy link
Owner Author

Self-repair added by #363, so closing this. (Will re-open if any other issues)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants