Surprise ImportError

[patwanjau]

Description

I am working on a recommender systems project and Surprise has been used as the preferred package in my examples. However, I'm experiencing problems when I use Surprise. There results in an import error attributed to the cpython library. Below is the error encountered:

Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "/home/user/.local/lib/python3.10/site-packages/surprise/__init__.py", line 3, in <module>
    from .prediction_algorithms import AlgoBase
  File "/home/user/.local/lib/python3.10/site-packages/surprise/prediction_algorithms/__init__.py", line 30, in <module>
    from .matrix_factorization import SVD
ImportError: /home/user/.local/lib/python3.10/site-packages/surprise/prediction_algorithms/matrix_factorization.cpython-310-x86_64-linux-gnu.so: undefined symbol: _PyGen_Send

Steps/Code to Reproduce

python -c 'import surprise; print("surprise", surprise.__version__)'

Expected Results

1.1.1

Actual Results

Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "/home/user/.local/lib/python3.10/site-packages/surprise/__init__.py", line 3, in <module>
    from .prediction_algorithms import AlgoBase
  File "/home/user/.local/lib/python3.10/site-packages/surprise/prediction_algorithms/__init__.py", line 30, in <module>
    from .matrix_factorization import SVD
ImportError: /home/user/.local/lib/python3.10/site-packages/surprise/prediction_algorithms/matrix_factorization.cpython-310-x86_64-linux-gnu.so: undefined symbol: _PyGen_Send

Versions

Linux-5.15.12-200.fc35.x86_64-x86_64-with-glibc2.34
Python 3.10.1 (main, Dec 9 2021, 00:00:00) [GCC 11.2.1 20211203 (Red Hat 11.2.1-7)]

Getting the version for surprise failed when executed in the code for "Steps to reproduce"

[ProfHercules]

What version of Cython are you running?

Seems that the issue stems from a slightly older version of Cython being incompatible with Python 3.10.1, I found reference to it in the Cython repo issues.

If there is an update to your version of Cython available, install that. Otherwise, there are some instructions in the linked issue around building the latest release of Cython from source.

Alternatively, if possible, try using Python < 3.10.1 (maybe 3.9)

[patwanjau]

What version of Cython are you running?

Seems that the issue stems from a slightly older version of Cython being incompatible with Python 3.10.1, I found reference to it in the Cython repo issues.

If there is an update to your version of Cython available, install that. Otherwise, there are some instructions in the linked issue around building the latest release of Cython from source.

Alternatively, if possible, try using Python < 3.10.1 (maybe 3.9)

1. I had not installed any Cython libraries, therefore, no Cython updates were available.
2. After your response, I installed the latest version of Cython via pip (version 0.29.26) and the same issue arose

I will attempt to install from source as downgrading Python is not an option at the moment.

[ProfHercules]

I just realised that what you're downloading when you install Surprise may be a pre-compiled wheel, which would have been compiled with an older version of Cython.

Perhaps try this:

For the latest version, you can also clone the repo and build the source (you'll first need Cython and numpy):

pip install numpy cython
git clone https://github.com/NicolasHug/surprise.git
cd surprise
python setup.py install

(from the README)

It should compile Surprise with the new version of Cython you have installed.

[mskiptr]

I hit the same issue.
Python: python-3.10.2-1 from Arch Linux

scikit-surprise from pip within a venv:

(.venv) [piotr@andrad ...]$ python -c 'import surprise; print("surprise", surprise.__version__)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File ".../.venv/lib/python3.10/site-packages/surprise/__init__.py", line 3, in <module>
    from .prediction_algorithms import AlgoBase
  File ".../.venv/lib/python3.10/site-packages/surprise/prediction_algorithms/__init__.py", line 30, in <module>
    from .matrix_factorization import SVD
ImportError: .../.venv/lib/python3.10/site-packages/surprise/prediction_algorithms/matrix_factorization.cpython-310-x86_64-linux-gnu.so:
undefined symbol: _PyGen_Send

Then I did:

(.venv) [piotr@andrad ...]$ pip install numpy cython
(.venv) [piotr@andrad ...]$ git clone https://github.com/NicolasHug/surprise.git
(.venv) [piotr@andrad ...]$ cd surprise
(.venv) [piotr@andrad .../surprise]$ python setup.py install

End of the build log

creating dist
creating 'dist/scikit_surprise-1.1.1-py3.10-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing scikit_surprise-1.1.1-py3.10-linux-x86_64.egg
creating .../.venv/lib/python3.10/site-packages/scikit_surprise-1.1.1-py3.10-linux-x86_64.egg
Extracting scikit_surprise-1.1.1-py3.10-linux-x86_64.egg to .../.venv/lib/python3.10/site-packages
Adding scikit-surprise 1.1.1 to easy-install.pth file
Installing surprise script to .../.venv/bin

Installed .../.venv/lib/python3.10/site-packages/scikit_surprise-1.1.1-py3.10-linux-x86_64.egg
Processing dependencies for scikit-surprise==1.1.1
Searching for six==1.16.0
Best match: six 1.16.0
Adding six 1.16.0 to easy-install.pth file

Using .../.venv/lib/python3.10/site-packages
Searching for scipy==1.8.0
Best match: scipy 1.8.0
Adding scipy 1.8.0 to easy-install.pth file

Using .../.venv/lib/python3.10/site-packages
Searching for numpy==1.22.3
Best match: numpy 1.22.3
Adding numpy 1.22.3 to easy-install.pth file
Installing f2py script to .../.venv/bin
Installing f2py3 script to .../.venv/bin
Installing f2py3.10 script to .../.venv/bin

Using .../.venv/lib/python3.10/site-packages
Searching for joblib==1.1.0
Best match: joblib 1.1.0
Adding joblib 1.1.0 to easy-install.pth file

Using .../.venv/lib/python3.10/site-packages
Finished processing dependencies for scikit-surprise==1.1.1

After that I tried again:

(.venv) [piotr@andrad .../surprise]$ python -c 'import surprise; print("surprise", surprise.__version__)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File ".../surprise/surprise/__init__.py", line 3, in <module>
    from .prediction_algorithms import AlgoBase
  File ".../surprise/surprise/prediction_algorithms/__init__.py", line 23, in <module>
    from .algo_base import AlgoBase
  File ".../surprise/surprise/prediction_algorithms/algo_base.py", line 10, in <module>
    from .. import similarities as sims
ImportError: cannot import name 'similarities' from partially initialized module 'surprise' (most likely due to a circular import) (.../surprise/surprise/__init__.py)

(.venv) [piotr@andrad .../surprise]$ cd ..
(.venv) [piotr@andrad ...]$ python -c 'import surprise; print("surprise", surprise.__version__)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File ".../.venv/lib/python3.10/site-packages/surprise/__init__.py", line 3, in <module>
    from .prediction_algorithms import AlgoBase
  File ".../.venv/lib/python3.10/site-packages/surprise/prediction_algorithms/__init__.py", line 30, in <module>
    from .matrix_factorization import SVD
ImportError: .../.venv/lib/python3.10/site-packages/surprise/prediction_algorithms/matrix_factorization.cpython-310-x86_64-linux-gnu.so:
undefined symbol: _PyGen_Send

I don't think I understand the problem and I can't really dig deeper into it right now, but maybe someone more knowledgeable can make something out of it.
Btw, this issue seems to be quite similar.

Recreating the venv with Python 3.9 (python39-3.9.11-3 from AUR) alleviated the issue.

(.venv) [piotr@andrad ...]$ python -c 'import surprise; print("surprise", surprise.__version__)'
surprise 1.1.1

[jaklan]

We also had a problem with importing surprise, but we discovered the issue was with numpy version:

• 1.20.3 failed
• 1.22.4 succeeded

So it seems the numpy requirement:

Surprise/requirements.txt

Line 2 in 46b9914

numpy>=1.11.2

is too loose and it should be more like numpy>=1.22 (or at least it should exclude version range around 1.20)