`pip install` fails (compile error) with Python 3.7rc1

[pgdr]

Description

Unable to pip install(compile error: PyThreadState has no member ...) with Python 3.7rc1.

My apologies if this has been reported, but I couldn't find it in the issues (closed/open) nor any pull request for it.

There have been similar issues with pyyaml (yaml/pyyaml#126), numpy (numpy/numpy#10500), cython (cython/cython#1978), and pygame (pygame/pygame#382).

Steps/Code to Reproduce

Install Python 3.7rc1.
pip install scikit-learn

Expected Results

pip succeeds.

Actual Results

Compilation errors:

...
sklearn/cluster/_dbscan_inner.cpp: In function ‘int
__Pyx__GetException(PyThreadState*, PyObject**, PyObject**, PyObject**)’:

sklearn/cluster/_dbscan_inner.cpp:6029:24: error: ‘PyThreadState’ has no member named ‘exc_type’
     tmp_type = tstate->exc_type;
                        ^
sklearn/cluster/_dbscan_inner.cpp:6030:25: error: ‘PyThreadState’ has no member named ‘exc_value’
     tmp_value = tstate->exc_value;
                         ^
sklearn/cluster/_dbscan_inner.cpp:6031:22: error: ‘PyThreadState’ has no member named ‘exc_traceback’
     tmp_tb = tstate->exc_traceback;
                      ^
sklearn/cluster/_dbscan_inner.cpp:6032:13: error: ‘PyThreadState’ has no member named ‘exc_type’
     tstate->exc_type = local_type;
             ^
sklearn/cluster/_dbscan_inner.cpp:6033:13: error: ‘PyThreadState’ has no member named ‘exc_value’
     tstate->exc_value = local_value;
             ^
sklearn/cluster/_dbscan_inner.cpp:6034:13: error: ‘PyThreadState’ has no member named ‘exc_traceback’
     tstate->exc_traceback = local_tb;
             ^
error: Command "g++ -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g
-fwrapv -O3 -Wall -fPIC
-I/data/install/lib/python3.7/site-packages/numpy/core/include
-I/data/install/lib/python3.7/site-packages/numpy/core/include
-I/data/install/include/python3.7m -c sklearn/cluster/_dbscan_inner.cpp -o
build/temp.linux-x86_64-3.7/sklearn/cluster/_dbscan_inner.o -MMD -MF
build/temp.linux-x86_64-3.7/sklearn/cluster/_dbscan_inner.o.d" failed with exit
status 1

Versions

Linux-3.10.0-862.el7.x86_64-x86_64-with-redhat-7.5-Maipo
Python 3.7.0rc1 (default, Jun 19 2018, 10:54:58) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]
NumPy 1.14.5
SciPy 1.1.0

[rth]

Thanks for reporting this!

Does installation of the latest dev version works?

pip install https://github.com/scikit-learn/scikit-learn/archive/master.zip

Note that you would need Cython >=0.27.3 to compile it..

(Linking to #11224 for future reference)

[jnothman]

It sounds like this might be something Cython needs to fix and release. We are dependent on Cython's support for Py3.7 here.​

[rth]

Yes, according to cython/cython#1955 the minimum requirements under Python 3.7 is Cython 0.27.3 where this issue was fixed. In #11224 I can confirm that the build works fine with Cython 0.28.

The minimal Cython version in doc/developers/advanced_installation.rst should probably be updated accordingly for Python 3.7

[jnothman]

The easiest way for us to move towards Python 3.7 support is for us to use it in our daily "latest dependencies" job. But this is easiest when NumPy and SciPy are supporting Python 3.7. NumPy is producing wheels for 3.7. At the moment, SciPy isn't making builds on Python 3.7, but is working on it (scipy/scipy#8988). We will try to follow suit.

IMO, we should not worry about Python 3.7 support in v0.20.0, but deal with it in a bug fix release (0.20.1 or 0.20.2), given the limited support in SciPy and Six (benjaminp/six#241).

[rth]

IMO, we should not worry about Python 3.7 support in v0.20.0, but deal with it in a bug fix release (0.20.1 or 0.20.2),

To summarize the situation for Python 3.7,

• with master (future 0.20.0), building scikit-learn requires Cython >= 0.27.3 and the test suite passes on Linux Compatibility with Python 3.7.0b5 #11224, but there may be other issues on plaforms that were not tested (Windows, Mac OS). What remains is,
  • to add CI
  • fix DeprecationWarning: Using or importing the ABCs from 'collections' in Python 3.7 #11121
  • build wheels Add python 3.7 support MacPython/scikit-learn-wheels#5
• with v0.19.1, building scikit-learn requires Cython >=0.27.3, <0.28 (I think). I can confirm @selasley 's report in scikit-learn fails to install with python 3.7.0 on macOS 12.13.5 #11378 (comment) that compilation fails with 0.28.3 due to Build error with cython dev #10342 . When using Cython 0.27.3, there are 75 test failures with the setup from Compatibility with Python 3.7.0b5 #11224 (see full log in sklearn-py37.log) most can be ignored (white space issues), but a few are genuine.

[cbrnr]

SciPy now has binary wheels for Python 3.7 (all platforms).

[jnothman]

FWIW, #11409 identifies that we can build with the development versions of numpy, scipy and cython, and all tests pass (albeit with PendingDeprecationWarning and DeprecationWarning). It does not try to run the example gallery, though. Nor does it try building in Windows (perhaps it should).

[sam-s]

homebrew upgraded me to python 3.7.0 and I am now left without scikit-learn.
when do you think I will be able to do pip3 install scikit-learn? a week? a month? a year?
I need this information to decide whether to wait or to downgrade to 3.6.*.
Thank you very much.

[rth]

when do you think I will be able to do pip3 install scikit-learn? a week? a month? a year?

A month very roughly for the next release. Meanwhile you can either install the development version #11320 (comment) or downgrade to python 3.6.

[jnothman]

We could consider releasing 0.19.2 in a hurry just for Py3.7 support??​