cvxpy 1.1.7 no longer compatible with numpy<1.20?

[roberthuisman]

I use python 3.7.9 with cvxpy 1.1.7 and numpy 1.18.1.
However as of Sunday we get an error message that the module is compiled with a newer numpy version. Before the weekend everything was still working.

What happened on sunday 31 january that this throws an error now?
Issue is resolved by upgrading numpy to 1.20, however isnt cvxpy supposed to support numpy>1.15? I want to stay on 1.18.1 still.

import cvxpy as cp
/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/cvxpy/init.py:18: in
from cvxpy.atoms import *
/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/cvxpy/atoms/init.py:20: in
from cvxpy.atoms.geo_mean import geo_mean
/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/cvxpy/atoms/geo_mean.py:20: in
from cvxpy.utilities.power_tools import (fracify, decompose, approx_error, lower_bound,
/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/cvxpy/utilities/power_tools.py:18: in
from cvxpy.atoms.affine.reshape import reshape
/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/cvxpy/atoms/affine/reshape.py:18: in
from cvxpy.atoms.affine.hstack import hstack
/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/cvxpy/atoms/affine/hstack.py:18: in
from cvxpy.atoms.affine.affine_atom import AffAtom
/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/cvxpy/atoms/affine/affine_atom.py:22: in
from cvxpy.cvxcore.python import canonInterface
/opt/hostedtoolcache/Python/3.7.9/x64/lib/python3.7/site-packages/cvxpy/cvxcore/python/init.py:3: in
import _cvxcore
E ImportError: numpy.core.multiarray failed to import
------------------------------- Captured stderr --------------------------------
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd

[BastiaanvG]

Experiencing a similar issue. When installing cvxpy it builds with the latest version of numpy (1.20 released this weekend), which makes cvxpy incompatible with the existing numpy installation < 1.20.

Fixed it by locally by changing the pyroject.toml build requirements to "numpy>=1.15,<1.20". But have not been able to figure out a more permanent fix.

[rileyjmurray]

I don't know for sure what's happened. If you're installing via pip, then the source distribution of CVXPY hasn't been touched since October 2020. There are some additional wheels that went up on December 31, for macOS and Windows.

When any CVXPY wheel is built, our C++ backend needs to be compatible with NumPy's API as it exists on the user's machine. Our approach here has been to build against the earliest version of NumPy possible, with the expectation that future versions wouldn't create problems. Perhaps that assumption isn't valid with NumPy 1.20. In any case, I'm not sure why a pip install cvxpy would try to update numpy to the latest release.

Pinging @SteveDiamond @akshayka.

Background info on NumPy in CVXPY's continuous integration

CVXPY's NumPy requirements are no longer as simple as they used to be. Because we have several low-level dependencies, our continuous integration testing has had to tie the NumPy version to the python version (#1202). From that PR:
https://github.com/cvxgrp/cvxpy/blob/8ac508a3214a0f7045253e27cf41189a8c9062dc/continuous_integration/Travis/install_dependencies.sh#L29-L47.

Those build configurations are used for macOS and linux. If you look at our Travis builds you'll see they only fail because our CI scripts try to upload wheels on all platforms, which results in PyPI rejecting uploads when we send them a "linux" (not manylinux) wheel. This funniness with linux wheels is bad form, but it's definitely not the source of this problem.

I just checked our AppVeyor builds (which upload Windows wheels) and they don't restrict the numpy version. The December 31 deployment built the Windows wheels against NumPy 1.19.4 and the current AppVeyor runs are building against NumPy 1.19.5 (these wheels haven't been deployed, however).

[akshayka]

I replicated the issue locally, by installing via pip (using the latest version of pip), on Linux, in an environment that already had NumPy 1.19.0 installed.

Some related issues.

freqtrade/freqtrade#4281

which describes the same issue with CVXPY.

pypa/pip#9542

which was opened in response to the above issue.

The issue seems to be the following:

pip isolates builds, meaning it uses a separate temporary environment to install/process build dependencies (declared in pyproject.toml). In our case, NumPy. For whatever reason, pip installs NumPy 1.20, instead of the version of NumPy that the user already has. This means that our C++ extension is built with numpy 1.20, even though the user might have 1.18.1 installed.

A possible workaround is to use

https://pypi.org/project/oldest-supported-numpy/

in our pyproject.toml, which would make sure that the extension we build used the oldest possible version of NumPy:

The reason to use the oldest available Numpy version as a build-time dependency is because of ABI compatibility. Binaries compiled with old Numpy versions are binary compatible with newer Numpy versions, but not vice versa. This meta-package exists to make dealing with this more convenient, without having to duplicate the same list manually in all packages requiring it.

[akshayka]

So: the issue is not that CVXPY is incompatible with older versions of NumPy, but rather installation via pip is building the cvxcore extension against the latest version of NumPy.

[darose]

We just ran into this issue as well. Is there a fix or workaround?