BUG: Compilation of scipy 1.10.1 and 1.11.1 fails with Python 3.11 #19026
Comments
lmmx
added
the
defect
lmmx
changed the title
|
With the same source but scipy version changed to 1.10.1, I get an error, so this is not just a scipy 1.11.1 bug: I suspect it's a Python 3.11 bug. Click to show Dockerfile source for failing Python 3.11 build of scipy 1.10.1FROM mlupin/docker-lambda:python3.11-build AS build
USER root
WORKDIR /var/task
# https://towardsdatascience.com/how-to-shrink-numpy-scipy-pandas-and-matplotlib-for-your-data-product-4ec8d7e86ee4
ENV CFLAGS "-g0 -Wl,--strip-all -DNDEBUG -Os -I/usr/include:/usr/local/include -L/usr/lib64:/usr/local/lib64:/usr/lib:/usr/local/lib"
RUN yum install -y wget curl git nasm openblas-devel.x86_64 lapack-devel.x86_64 python-dev file-devel make Cython libgfortran10.x86_64 openssl-devel
# Download and install CMake
WORKDIR /tmp
ENV CMAKE_VERSION=3.26.4
# Download and install CMake
RUN wget https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}.tar.gz && \
tar -xvzf cmake-${CMAKE_VERSION}.tar.gz && \
cd cmake-${CMAKE_VERSION} && ./bootstrap && make -j4 && make install
# Clean up temporary files
RUN rm -rf /tmp/cmake-${CMAKE_VERSION} && \
rm /tmp/cmake-${CMAKE_VERSION}.tar.gz
WORKDIR /var/task
RUN /usr/bin/cmake --version
# Upgrade GCC to version 8 for SciPy Meson build system
RUN wget https://ftp.gnu.org/gnu/gcc/gcc-8.4.0/gcc-8.4.0.tar.gz && \
tar xf gcc-8.4.0.tar.gz && \
rm gcc-8.4.0.tar.gz && \
cd gcc-8.4.0 && \
./contrib/download_prerequisites && \
mkdir build && \
cd build && \
../configure --disable-multilib && \
make -j$(nproc) && \
make install && \
cd / && \
rm -rf gcc-8.4.0
# Set environment variables
ENV CC=/usr/local/bin/gcc
ENV CXX=/usr/local/bin/g++
ENV FC=/usr/local/bin/gfortran
# Verify GCC version
RUN gcc --version && \
/usr/local/bin/gfortran --version
# ------------------- END OF SYSTEM BUILD DEPENDENCY SETUP ------------------
RUN mkdir -p /var/task/np_scipy_pd_layer/python && \
mkdir -p /var/task/np_scipy_pd_layer/lib
# Install build dependencies for the wheels
RUN python3.11 -m pip install --upgrade pip && \
python3.11 -m pip --version && \
python3.11 -m pip install Cython pybind11 pythran
# ------------------- END OF PACKAGE BUILD DEPENDENCY SETUP ------------------
# Specify the version to use for numpy and scipy
ENV NUMPY_VERSION=1.25.2
ENV SCIPY_VERSION=1.10.1
ENV PANDAS_VERSION=1.5.3
# Download numpy and scipy source distributions
RUN python3.11 -m pip download --no-binary=:all: numpy==$NUMPY_VERSION
# Extract the numpy package and build the wheel
RUN ls && tar xzf numpy-$NUMPY_VERSION.tar.gz
RUN ls && cd numpy-$NUMPY_VERSION && python3.11 setup.py bdist_wheel build_ext -j 4
ENV BUILT_NUMPY_WHEEL=numpy-$NUMPY_VERSION/dist/numpy-$NUMPY_VERSION-*.whl
RUN ls $BUILT_NUMPY_WHEEL
# Don't install NumPy from the built wheel but use same version (it's a SciPy dependency)
RUN python3.11 -m pip install numpy==$NUMPY_VERSION
RUN python3.11 -c "import numpy"
# ------------------------ END OF NUMPY BUILD --------------------------------
# Extract the SciPy package and build the wheel
RUN git clone -b "v$SCIPY_VERSION" --depth 1 --recursive \
https://github.com/scipy/scipy.git scipy-$SCIPY_VERSION && \
cd scipy-$SCIPY_VERSION && \
git submodule update --init
RUN cd scipy-$SCIPY_VERSION && python3.11 setup.py bdist_wheel build_ext -j 4
ENV BUILT_SCIPY_WHEEL=scipy-$SCIPY_VERSION/dist/SciPy-*.whl
RUN ls $BUILT_SCIPY_WHEEL
# ------------------------ END OF SCIPY BUILD --------------------------------
# Download pandas source from git tree (pip download failed due to bad metadata)
RUN git clone -b "v$PANDAS_VERSION" --depth 1 \
https://github.com/pandas-dev/pandas.git pandas-$PANDAS_VERSION
# Extract the pandas package and build the wheel
RUN cd pandas-$PANDAS_VERSION && python3.11 setup.py bdist_wheel build_ext -j 4
ENV BUILT_PANDAS_WHEEL=pandas-$PANDAS_VERSION/dist/pandas-$PANDAS_VERSION-*.whl
RUN ls $BUILT_PANDAS_WHEEL
# ------------------------ END OF PANDAS BUILD --------------------------------
# Install the wheels with pip
# (Note: previously this used --compile but now we already did the wheel compilation)
RUN python3.11 -m pip install --no-compile --no-cache-dir \
-t /var/task/np_scipy_pd_layer/python \
$BUILT_NUMPY_WHEEL \
$BUILT_SCIPY_WHEEL \
$BUILT_PANDAS_WHEEL && \
ls /var/task/np_scipy_pd_layer/python
# -------------------- END OF PACKAGE INSTALLATION -----------------------------
# Clean up the sdists and wheels, uninstall non-built numpy after building SciPy wheel with it
RUN rm numpy-$NUMPY_VERSION.tar.gz && \
rm -r numpy-$NUMPY_VERSION scipy-$SCIPY_VERSION pandas-$PANDAS_VERSION && \
python3.11 -m pip uninstall numpy -y
RUN cp /usr/lib64/libblas.so.3.4.2 /var/task/np_scipy_pd_layer/lib/libblas.so.3 \
&& cp /usr/lib64/libgfortran.so.4.0.0 /var/task/np_scipy_pd_layer/lib/libgfortran.so.4 \
&& cp /usr/lib64/libgfortran.so.5.0.0 /var/task/np_scipy_pd_layer/lib/libgfortran.so.5 \
&& cp /usr/lib64/liblapack.so.3.4.2 /var/task/np_scipy_pd_layer/lib/liblapack.so.3 \
&& cp /usr/lib64/libquadmath.so.0.0.0 /var/task/np_scipy_pd_layer/lib/libquadmath.so.0 \
&& cd /var/task/np_scipy_pd_layer \
&& zip -r9 np_scipy_pd_layer.zip python \
&& zip -r9 np_scipy_pd_layer.zip lib
FROM mlupin/docker-lambda:python3.11-build
COPY --from=build /var/task/np_scipy_pd_layer /opt
COPY --from=build /var/task /var/task
RUN PYTHONPATH=/opt/python python3 -c 'import numpy as np'
RUN PYTHONPATH=/opt/python python3 -c 'from scipy import spatial'
RUN PYTHONPATH=/opt/python python3 -c 'import pandas as pd' |
|
I have a few thoughts on what you're doing here, but my first one is if you really need bleeding edge Cython? Can you not use I can build SciPy from source and pass the full test suite locally with the Hopefully that information is helpful to you. Let me check with @rgommers re: priority on backporting support for Cython |
|
Ah yes it may well be Cython, I see version 3 came out mid July. I wasn't intentionally going "bleeding edge" no! I ran our previous known-good build overnight and indeed that too now fails (where it previously ran) as it was last done mid-June pre-v3 release (Cython version was not pinned). I will try to follow the guidance on meson in future. As for reasons for not doing that now, originally the distutils build from source approach was to reduce packaged size for cloud deployment but I'm not sure if this is an issue now or merely requires more time investment to migrate. Please feel free to close if Cython backport support is not a priority, thanks for all your work here. |
rgommers
added
Build issues
Do you do things like strip out tests and unused components to fit into AWS Lambda or some such thing? If so, I can suggest a replacement probably, and it'd be good to know if people still need that kind of thing.
That wasn't one simple fix, so I don't think we want to add Cython 3.0 support retroactively. So I'll close this. Also, I'll open a PR to remove building with |
…e used See the added code comment in this commit for rationale of keeping `_setup.py` around. We got another two bug reports that used `python setup.py install` in the last day (scipygh-19022 and scipygh-19026), so it's time to make that impossible. For those who really really need it, they can still manually rename `_setup.py` to `setup.py` for a while longer, but this should make it much clearer that we no longer support this install method (beyond the one conda-forge on Windows case). [skip circle] [skip cirrus]
…e used See the added code comment in this commit for rationale of keeping `_setup.py` around. We got another two bug reports that used `python setup.py install` in the last day (scipygh-19022 and scipygh-19026), so it's time to make that impossible. For those who really really need it, they can still manually rename `_setup.py` to `setup.py` for a while longer, but this should make it much clearer that we no longer support this install method (beyond the one conda-forge on Windows case). [skip circle] [skip cirrus]
|
No solution: pip doesn't work correctly for my system, even for matplotlib which I can compile with setup.py. |
|
I don't know how to answer that question. |
|
with alias env312 × pip subprocess to install build dependencies did not run successfully. note: This error originates from a subprocess, and is likely not a problem with pip. × pip subprocess to install build dependencies did not run successfully. note: This error originates from a subprocess, and is likely not a problem with pip. Hope this helps. |
|
It's failing to build Basically, you are too early in trying to use Python 3.12 release candidates. Just give it a few weeks, because this is just asking for problems. |
I don't think so, but I would definitely be keen to hear what this is!
Ah, not quite the impact I was expecting but impact nonetheless haha! 🤣 |
SciPy contains 17 submodules which your application may not all need, a full test suite (of quite of few MBs), docstrings that take up a fair amount of space, etc. If you're really space-constrained, you may want to go through the trouble of stripping all that out, for example by downloading a wheel, unpacking it, I'll also note that conda-forge split the test suite off into a separate |
…e used See the added code comment in this commit for rationale of keeping `_setup.py` around. We got another two bug reports that used `python setup.py install` in the last day (scipygh-19022 and scipygh-19026), so it's time to make that impossible. For those who really really need it, they can still manually rename `_setup.py` to `setup.py` for a while longer, but this should make it much clearer that we no longer support this install method (beyond the one conda-forge on Windows case). [skip circle] [skip cirrus]
Describe your issue.
I previously built a Docker image with Python 3.10, numpy 1.24.3, scipy 1.10.1, pandas 1.5.3.
Today I tried to build one identically but with different versions: Python 3.11 numpy 1.25.2, scipy 1.11.1, pandas 1.5.3
I can't say for sure yet which version change is at fault but I suspect it's not numpy, leaving either:
Edit I tried building version 1.10 again with Python 3.11 and got a different error but same Cython failure: I'll add a separate comment in thread
Similar to:
Perhaps going back further to:
Reproducing Code Example
Error message
SciPy/NumPy/Python version and system information
Docker image of Python 3.11 built for deployment to AWS Lambda: https://hub.docker.com/layers/mlupin/docker-lambda/python3.11-build/images/sha256-7ec5685e7361cfbc09d11427cd949bf98396cdac19f56dff98ebd7daf0d4e773)
The text was updated successfully, but these errors were encountered: