Cython CompileError: expected_mutual_info_fast.pyx

[brobr]

Description

Hi, with installed:
python-2.7.14
numpy-1.13.1
scipy-0.18.0
Cython-0.28.1

compiling scikiyt-learn-0.19.1 on Slackware64-current: gives this error:
Cython.Compiler.Errors.CompileError: sklearn/metrics/cluster/expected_mutual_info_fast.pyx

[16/40] Cythonizing sklearn/manifold/_utils.pyx
[17/40] Cythonizing sklearn/metrics/cluster/expected_mutual_info_fast.pyx

Error compiling Cython file:
------------------------------------------------------------
...
# cython: cdivision=True
^
------------------------------------------------------------

sklearn/metrics/cluster/expected_mutual_info_fast.pyx:1:0: 'sklearn.metrics/cluster.expected_mutual_info_fast' is not a valid module name
Traceback (most recent call last):
File "setup.py", line 269, in
setup_package()
File "setup.py", line 265, in setup_package
setup(**metadata)
File "/usr/lib64/python2.7/site-packages/numpy/distutils/core.py", line 135, in setup
config = configuration()
File "setup.py", line 135, in configuration
config.add_subpackage('sklearn')
File "/usr/lib64/python2.7/site-packages/numpy/distutils/misc_util.py", line 1029, in add_subpackage
caller_level = 2)
File "/usr/lib64/python2.7/site-packages/numpy/distutils/misc_util.py", line 998, in get_subpackage
caller_level = caller_level + 1)
File "/usr/lib64/python2.7/site-packages/numpy/distutils/misc_util.py", line 935, in _get_configuration_from_setup_py
config = setup_module.configuration(*args)
File "sklearn/setup.py", line 82, in configuration
maybe_cythonize_extensions(top_path, config)
File "/tmp/SBo/scikit-learn-0.19.1/sklearn/_build_utils/init.py", line 84, in maybe_cythonize_extensions
config.ext_modules = cythonize(config.ext_modules)
File "/usr/lib64/python2.7/site-packages/Cython/Build/Dependencies.py", line 1026, in cythonize
cythonize_one(*args)
File "/usr/lib64/python2.7/site-packages/Cython/Build/Dependencies.py", line 1146, in cythonize_one
raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: sklearn/metrics/cluster/expected_mutual_info_fast.pyx

Steps/Code to Reproduce

Expected Results

Actual Results

Versions

[rth]

Thanks for reporting it @brobr .

It's a duplicate of #10342

Either using Cython 0.27.x with scikit-learn 0.19.1 or building the latest dev version with Cython 0.28.x should fix it. Please comment if it's not the case.

[brobr]

I noticed #10342 after googling with the observed error-message I got with compiling 0.19.1;

Then downloaded the sklearn-master from github and tried to compile sklearn again. Same error.

So, why does it not now behave??

[rth]

Cython 0.28.x???

I mean any minor version of Cython 0.28 (including 0.28.0 and 0.28.1).

Could you try scikit-learn 0.19.1 for instance with Cython 0.27.3, please?

[rth]

FWIW, I can reproduce this on Linux with conda, when trying to build the v0.19.1 from source with Cython 0.28.1,

pip install git+https://github.com/scikit-learn/scikit-learn.git@0.19.1

So it looks like scikit-learn 0.19.1 is not compatible with Cython >=0.28 cc @lesteve

@brobr I imagine you are building this version of scikit-learn as part of your package manager on Slackware? This will affect Slackware, Gentoo etc, there is not much we can do about it on the scikit-learn side: the dependencies should be updated accordingly in the corresponding package managers.

Meanwhile, you can just downgrade Cython or use the development scikit-learn version.

[brobr]

That is not so easy; Cython-0.28.1 is part of the current system and the previous version of Cython available to me is 0.23.4 on Slackware-14.2. On pip I only see the latest Cython. Further, at the moment I am running projects in Jupiter Notebook for which I rather not recompile everything, unless that would be helpful.

[rth]

On pip I only see the latest Cython

You can get any of the previous versions on PyPi from https://pypi.org/project/Cython/#history
for instance, pip install Cython==0.27.3.

Also, if you just install scikit-learn with pip, you will get a binary wheel wich will avoid this issue (but it might be less optimized for your system)..

[brobr]

Also, this was in the output far before the error, would this be relevant?

..
scikit-learn-0.19.1/sklearn/utils/weight_vector.pyx
Partial import of sklearn during the build process.
blas_opt_info:
blas_mkl_info:
libraries mkl_rt not found in ['/usr/local/lib64', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/']
NOT AVAILABLE

blis_info:
libraries blis not found in ['/usr/local/lib64', '/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/']
NOT AVAILABLE

openblas_info:
FOUND:
libraries = ['openblas', 'openblas']
library_dirs = ['/usr/lib64']
language = c
define_macros = [('HAVE_CBLAS', None)]

FOUND:
libraries = ['openblas', 'openblas']
library_dirs = ['/usr/lib64']
language = c
define_macros = [('HAVE_CBLAS', None)]

sklearn/manifold/_barnes_hut_tsne.pyx: cannot find cimported module 'sklearn.neighbors'
Compiling sklearn/metrics/pairwise_fast.pyx because it changed.
...

[rth]

Also, this was in the output far before the error, would this be relevant?

Not particularly, this seems fine apart for the cannot find cimported module 'sklearn.neighbors'.

But I was saying there isn't much we can do about this issue on our side.

[brobr]

Well, the pip-way I try to evade in order to keep track of installed software using the package management that comes with the system. But doing it, it works:

bash-4.4# pip install sklearn
Collecting sklearn
Downloading sklearn-0.0.tar.gz
Collecting scikit-learn (from sklearn)
Downloading scikit_learn-0.19.1-cp27-cp27mu-manylinux1_x86_64.whl (12.2MB)
100% |████████████████████████████████| 12.2MB 129kB/s
Installing collected packages: scikit-learn, sklearn
Running setup.py install for sklearn ... done
Successfully installed scikit-learn-0.19.1 sklearn-0.0

and it works ok in my notebook excercise .....
how come?

[brobr]

But definitely thanks for the program; now I have to learn to use it....

[lesteve]

So it looks like scikit-learn 0.19.1 is not compatible with Cython >=0.28 cc @lesteve

Any failure in scipy-dev-wheels is something that will break once the dependency is released, this is something to bear in mind each time we fix the cython-dev-wheels build. Is it worth thinking about 0.19.2 for a reason like this I am not sure ... I would argue that is is not very often that someone will try to compile scikit-learn 0.19.1 with cython >= 0.28.

Ping @jnothman in case you have some spare bandwidth and a opinion!

[rth]

Is it worth thinking about 0.19.2 for a reason like this I am not sure ... I would argue that is is not very often that someone will try to compile scikit-learn 0.19.1 with cython >= 0.28.

+1

Downloading scikit_learn-0.19.1-cp27-cp27mu-manylinux1_x86_64.whl (12.2MB)
and it works ok in my notebook excercise ..... how come?

It installs a binary manylinux wheel that doesn't need to be compiled.

[jnothman]

So am I right to think that the issues here are: can/should we pin cython support to a particular version? And if we don't, and our most recent release breaks with the most recent cython, do we issue a minor release to fix? Are these the issues?

[lesteve]

And if we don't, and our most recent release breaks with the most recent cython, do we issue a minor release to fix?

Yeah that's the main question. I guess if it was numpy or scipy we would probably release a minor version, but cython seems less critical.

So am I right to think that the issues here are: can/should we pin cython support to a particular version?

I guess I did not think of that but I am not a big fan.

[jnothman]

what are the disadvantages of pinning cython? does pinning cython create incompatibilities? I've missed in this conversation what constrains a user to a particular version of cython.

[lesteve]

I've missed in this conversation what constrains a user to a particular version of cython.

scikit-learn 0.19.1 does not compile with cython >= 0.28 (cython 0.28 was released on March 13 2018)

what are the disadvantages of pinning cython?

By pinning cython, do you mean requiring an exact cython version for a given scikit-learn version, e.g. cython==0.28.1 for scikit-learn==0.20? In general there is some range of cython versions for which scikit-learn version n and version n+1 compiles fine (with cython >= 0.23 I am pretty sure that you can go back 2 or 3 scikit-learn versions). I may be missing something but I don't really see what problems we solve by requiring an exact cython version. I see many disadvantages like needing to switch from one cython version to another when compiling scikit-learn across versions (e.g. when git bisecting).