Import error with latest version of scipy installed

[ehtec]

Describe the bug
When using this together with the latest version 1.11.0 of scipy (from pypi), you get an error when importing cvxpy:

File "/usr/local/lib/python3.10/site-packages/cvxpy/__init__.py", line 18, in <module>
    import cvxpy.interface.scipy_wrapper
  File "/usr/local/lib/python3.10/site-packages/cvxpy/interface/scipy_wrapper.py", line 37, in <module>
    method = getattr(spmatrix, method_name)
AttributeError: type object 'spmatrix' has no attribute '__div__'

Version

• OS: Debian 11
• CVXPY Version: 1.3.1

Additional context
I encountered this because my CI pipeline is failing since the latest version of scipy was released a few hours ago. It works with scipy 1.10.1 (the second-latest version).

[phschiele]

@ehtec Thanks for reporting this! I was able to replicate the issue and found the following:

• As stated in the message, the error originates from scipy_wrapper.py
• Adding a try ... except around the loop in the file fixes all tests
• Indeed, so does commenting out all of the contents in the file
• I used the below snippet to understand the intended behavior under SciPy 1.10:

import cvxpy as cp
import numpy as np
import scipy.sparse as sp

a = cp.Variable((1,3))
b = sp.coo_matrix(np.arange(3))
a + b
>>> Expression(AFFINE, UNKNOWN, (1, 3))
b + a
>>> Expression(AFFINE, UNKNOWN, (1, 3))
a.__add__(b)
>>> Expression(AFFINE, UNKNOWN, (1, 3))
b.__add__(a)
>>> NotImplemented

My thoughts are:

1. At the very least, we had some missing test coverage on the intended effect of these operations
2. If anything, we should probably raise NotImplementedError instead of return NotImplemented
3. The operation b + a does not fail because it is wrapped again by

   cvxpy/cvxpy/expressions/expression.py

   Lines 46 to 51 in 8f3e2b9

   	def cast_op(self, other):
   	"""A wrapped binary operator that can handle non-Expression arguments.
   	"""
   	other = self.cast_to_const(other)
   	return binary_op(self, other)
   	return cast_op

   , so the combination of the wrappers is likely not intended behavior
4. Another fix could be to replace __div__ with __truediv__ in the failing file.

The files have been there for quite a long time, so maybe @cvxpy/maintainers can give some context.

[niklasvm]

I can confirm we are experiencing the same issue since a few hours ago

[Gazzzzyy]

Hi all, we're also experiencing the same issues on our end! Will keep an eye on this thread :)

[detrin]

I can confirm we are facing the same issue here

[phschiele]

Thanks, everyone for the heads-ups! It looks like everyone who is installing cvxpy with updated SciPy is getting these errors. Anyone else finding this issue, just give a 👍 to the first post and subscribe to the issue :)

[detrin]

There are changes in API

this release
last release

I have a sense that it is not going to be one line fix.

[phschiele]

Linking @h-vetinari's PR in the SciPy repo: scipy/scipy#18749

[SteveDiamond]

The proposal here seems fine to me:
scipy/scipy#18749 (comment)

The purpose of scipy_wrapper.py is to enable operator overloading with SciPy sparse matrices. It's not needed for NumPy ndarrays because NumPy has better native support for operator overloading.

[SteveDiamond]

I started a PR to fix this:
#2159
@phschiele feel free to add try/catch and/or more tests to the PR.

[phschiele]

Closed via #2159, patch is in the works.

[niklasvm]

Closed via #2159, patch is in the works.

When will this fix be released to pypi?

[phschiele]

The patch is now available on PyPI: https://pypi.org/project/cvxpy/1.3.2/
Patches for 1.2 and 1.1 will follow shortly.

[detrin]

Much appreciated, thanks