[BUG] Errors when building PyCall.jl #257
Comments
|
It might be good to automatically reset |
|
I think PySR should catch I think another couple of changes would simplify the install process for the user:
|
|
Couldn't we just check for a mismatch between PyCall.python and whatever the current Python executable is? How do you find the latter? |
|
I'm not sure, because the Python executable could have the same name. So ideally we would want to also store the version of Python and any other meta information that might affect things. |
|
We need more diagnostics here. Can you catch this error and do one of the following? Somehow either Project.toml or Manifest.toml had been corrupted or the active environment is not what we think it is. If there is something to delete, it might be the deps folder within thr package. |
|
I'm trying to get a docker container that can consistently reproduce this. FYI I've only seen the error when building via PyJulia; I'm not sure how to trigger it with PyCall directly. |
|
My current attempt to generate this error. No error reported yet... I wonder if somehow the latest PyCall has made this error go away. Dockerfile:FROM debian:bullseye-slim
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y curl git build-essential libssl-dev zlib1g-dev libbz2-dev \
libreadline-dev libsqlite3-dev libncurses5-dev libncursesw5-dev \
xz-utils libffi-dev liblzma-dev
# Install juliaup:
RUN curl -fsSL https://install.julialang.org | sh -s -- -y
# Install pyenv:
RUN curl -fsSL curl https://pyenv.run | sh && \
echo 'export PATH="/root/.pyenv/bin:$PATH"' >> ~/.bashrc && \
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc && \
echo 'eval "$(pyenv init -)"' >> ~/.bashrc && \
echo 'eval "$(pyenv virtualenv-init -)"' >> ~/.bashrc
# Default to using bash -l:
SHELL ["/bin/bash", "-l", "-c"]
RUN juliaup add 1.8 && juliaup default 1.8
RUN pyenv install 3.9.2 && pyenv global 3.9.2
RUN python3 -m pip install --upgrade pip
# Try to install pysr:
RUN python3 -m pip install pysr
RUN python3 -m pysr install
# Change Julia and Python versions:
RUN juliaup add 1.9 && juliaup default 1.9 && juliaup remove 1.8
RUN pyenv install 3.10 && pyenv global 3.10
RUN python3 -m pip install --upgrade pip
# Try to re-install pysr:
RUN python3 -m pip install pysr
RUN python3 -m pysr installOne potential solution if I can end up producing the bug: make the PySR environment equal to |
|
Okay I have something to reliably generate an error. If you install PySR with a specific Python version, and then change the Python version, remove Dockerfile:FROM debian:bullseye-slim
ENV DEBIAN_FRONTEND=noninteractive
# Install juliaup and pyenv:
RUN apt-get update && apt-get install -y curl
# Install juliaup:
RUN curl -fsSL https://install.julialang.org | sh -s -- -y
RUN apt-get install -y git build-essential libssl-dev zlib1g-dev libbz2-dev \
libreadline-dev libsqlite3-dev libncurses5-dev libncursesw5-dev \
xz-utils libffi-dev liblzma-dev
# Install pyenv:
RUN curl -fsSL curl https://pyenv.run | sh && \
echo 'export PATH="/root/.pyenv/bin:$PATH"' >> ~/.bashrc && \
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc && \
echo 'eval "$(pyenv init -)"' >> ~/.bashrc && \
echo 'eval "$(pyenv virtualenv-init -)"' >> ~/.bashrc
# Default to using bash -l:
SHELL ["/bin/bash", "-l", "-c"]
RUN juliaup add 1.8 && juliaup default 1.8
RUN pyenv install 3.9.2 && pyenv global 3.9.2
RUN python3 -m pip install --upgrade pip
# Try to install pysr:
RUN python3 -m pip install pysr==0.14.2
RUN python3 -m pysr install
# Change Python version:
RUN pyenv install 3.10 && pyenv global 3.10 && pyenv uninstall -f 3.9.2
RUN python3 -m pip install --upgrade pip
# Try to use PySR:
RUN python3 -m pip install pysr==0.14.2
RUN rm -r ~/.julia/environments/pysr-0.14.2
RUN python3 -m pysr installThis generates the following error during the `python -m pysr install` step:There should definitely be a way we can automatically fix this, right? Is it I note that even doing |
|
It seems like PyCall.jl is storing the const libpython = "/root/.pyenv/versions/3.9.2/lib/libpython3.9.so.1.0"Maybe that's the issue? Perhaps a new package directory is not being created when the Python binary changes? |
|
Run a Julia process that executes this before trying to connect Julia and Python. That will force the following file to run again. https://github.com/JuliaPy/PyCall.jl/blob/master/deps/build.jl |
|
That's a good idea. Let me try that. |
|
I implemented your suggestion in #363 so it automatically updates PyCall. Seems to work locally! I added a unittest too. |
|
Okay I'm temporarily going to close this with #363 merged. Fingers crossed. I'll leave it pinned though. |
|
Fixed by #535 |
tldr, delete the folders:
~/.julia/packages/PyCall~/.julia/prefs/PyCall~/.julia/environments/pysr-{your-pysr-version}and re-run
python -m pysr install.I am documenting this error because it has shown up a lot for people, and I want to document the solution.
The error is basically that after upgrading Python or Julia or both, and running PySR or re-installing PySR, you get the following error:
The solution to this is to delete your
~/.julia/packages/PyCallfolder. Then, simply re-runpython -m pysr install.If this gives you further errors in building, you should also delete
~/.julia/prefs/PyCall, and re-run the install step. For further issues, you could delete~/.julia/environmentsfolder (or more specifically, the folders~/.julia/environments/pysr-{version}and~/.julia/environments/{julia_version}, if you want to keep your other environments).This should fix the problem. This issue is basically that the old installed version of PyCall.jl is attached to a specific combination of Julia and Python. And, for whatever reason, re-running
julia.install()doesn't update the built version of PyCall.jl.This is a temporary manual solution. In the future PySR should handle this automatically.
The text was updated successfully, but these errors were encountered: