Incompatibility with Python 3.7.0

[ductng]

While installing on Mac OSX 10.13.5 with Python 3.7.0

brew list --versions | grep python
python 3.7.0

pip3 list | grep Cython
Cython                             0.28.3

pip3 install root_numpy
following errors occur:

 cython_utility:186:65: error: too many arguments to function call, expected 3, have 4
      return (*((__Pyx_PyCFunctionFast)meth)) (self, args, nargs, NULL);
             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~                     ^~~~
  /Library/Developer/CommandLineTools/usr/lib/clang/10.0.0/include/stddef.h:100:18: note: expanded from macro 'NULL'
  #    define NULL __null
                   ^~~~~~
  cython_utility:1331:21: error: no member named 'exc_type' in '_ts'
      *type = tstate->exc_type;
              ~~~~~~  ^
  cython_utility:1332:22: error: no member named 'exc_value' in '_ts'; did you mean 'curexc_value'?
      *value = tstate->exc_value;
                       ^~~~~~~~~
                       curexc_value
  /usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/include/python3.7m/pystate.h:237:15: note: 'curexc_value' declared here
      PyObject *curexc_value;
                ^
  cython_utility:1333:19: error: no member named 'exc_traceback' in '_ts'; did you mean 'curexc_traceback'?
      *tb = tstate->exc_traceback;
                    ^~~~~~~~~~~~~
                    curexc_traceback
  /usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/include/python3.7m/pystate.h:238:15: note: 'curexc_traceback' declared here
      PyObject *curexc_traceback;
                ^
  cython_utility:1340:24: error: no member named 'exc_type' in '_ts'
      tmp_type = tstate->exc_type;
                 ~~~~~~  ^
  cython_utility:1341:25: error: no member named 'exc_value' in '_ts'; did you mean 'curexc_value'?
      tmp_value = tstate->exc_value;
                          ^~~~~~~~~
                          curexc_value
  /usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/include/python3.7m/pystate.h:237:15: note: 'curexc_value' declared here
      PyObject *curexc_value;
                ^
  cython_utility:1342:22: error: no member named 'exc_traceback' in '_ts'; did you mean 'curexc_traceback'?
      tmp_tb = tstate->exc_traceback;
                       ^~~~~~~~~~~~~
                       curexc_traceback
  /usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/include/python3.7m/pystate.h:238:15: note: 'curexc_traceback' declared here
      PyObject *curexc_traceback;
                ^
  cython_utility:1343:13: error: no member named 'exc_type' in '_ts'
      tstate->exc_type = type;
      ~~~~~~  ^
  cython_utility:1344:13: error: no member named 'exc_value' in '_ts'; did you mean 'curexc_value'?
      tstate->exc_value = value;
              ^~~~~~~~~
              curexc_value
  /usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/include/python3.7m/pystate.h:237:15: note: 'curexc_value' declared here
      PyObject *curexc_value;
                ^
  cython_utility:1345:13: error: no member named 'exc_traceback' in '_ts'; did you mean 'curexc_traceback'?
      tstate->exc_traceback = tb;
              ^~~~~~~~~~~~~
              curexc_traceback
  /usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/include/python3.7m/pystate.h:238:15: note: 'curexc_traceback' declared here
      PyObject *curexc_traceback;
                ^
  cython_utility:1400:24: error: no member named 'exc_type' in '_ts'
      tmp_type = tstate->exc_type;
                 ~~~~~~  ^
  cython_utility:1401:25: error: no member named 'exc_value' in '_ts'; did you mean 'curexc_value'?
      tmp_value = tstate->exc_value;
                          ^~~~~~~~~
                          curexc_value
  /usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/include/python3.7m/pystate.h:237:15: note: 'curexc_value' declared here
      PyObject *curexc_value;
                ^
  cython_utility:1402:22: error: no member named 'exc_traceback' in '_ts'; did you mean 'curexc_traceback'?
      tmp_tb = tstate->exc_traceback;
                       ^~~~~~~~~~~~~
                       curexc_traceback
  /usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/include/python3.7m/pystate.h:238:15: note: 'curexc_traceback' declared here
      PyObject *curexc_traceback;
                ^
  cython_utility:1403:13: error: no member named 'exc_type' in '_ts'
      tstate->exc_type = local_type;
      ~~~~~~  ^
  cython_utility:1404:13: error: no member named 'exc_value' in '_ts'; did you mean 'curexc_value'?
      tstate->exc_value = local_value;
              ^~~~~~~~~
              curexc_value
  /usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/include/python3.7m/pystate.h:237:15: note: 'curexc_value' declared here
      PyObject *curexc_value;
                ^
  cython_utility:1405:13: error: no member named 'exc_traceback' in '_ts'; did you mean 'curexc_traceback'?
      tstate->exc_traceback = local_tb;
              ^~~~~~~~~~~~~
              curexc_traceback
  /usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/include/python3.7m/pystate.h:238:15: note: 'curexc_traceback' declared here
      PyObject *curexc_traceback;
                ^
  cython_utility:1427:24: error: no member named 'exc_type' in '_ts'
      tmp_type = tstate->exc_type;
                 ~~~~~~  ^
  cython_utility:1428:25: error: no member named 'exc_value' in '_ts'; did you mean 'curexc_value'?
      tmp_value = tstate->exc_value;
                          ^~~~~~~~~
                          curexc_value
  /usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/include/python3.7m/pystate.h:237:15: note: 'curexc_value' declared here
      PyObject *curexc_value;
                ^
  cython_utility:1429:22: error: no member named 'exc_traceback' in '_ts'; did you mean 'curexc_traceback'?
      tmp_tb = tstate->exc_traceback;
                       ^~~~~~~~~~~~~
                       curexc_traceback
  /usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/include/python3.7m/pystate.h:238:15: note: 'curexc_traceback' declared here
      PyObject *curexc_traceback;
                ^
  fatal error: too many errors emitted, stopping now [-ferror-limit=]
  1 warning and 20 errors generated.
  error: command 'clang' failed with exit status 1

It seems a common problem with Python 3.7.0. I have seen several fixes in several projects. I started with _librootnumpy.cpp like:

-    return (*((__Pyx_PyCFunctionFast)meth)) (self, args, nargs, NULL);
+    return (*((__Pyx_PyCFunctionFast)meth)) (self, args, nargs);
 
-    *type = tstate->exc_type;
-    *value = tstate->exc_value;
-    *tb = tstate->exc_traceback;
+    *type = tstate->exc_state.exc_type;
+    *value = tstate->exc_state.exc_value;
+    *tb = tstate->exc_state.exc_traceback;
 
-            return PyUnicode_AsUTF8(o);
+            return const_cast <char*> (PyUnicode_AsUTF8(o));

If I have confirmed that these were all the changes necessary, I will post a commit.

[ductng]

Also needed to apply the same changes to _libtmvanumpy.cpp. Now all the changes are in
ductng@4e920b7

Just one comment, a more knowledgeable developer should have a look, if the changes are fine and that it does not break any functionality. Especially the "const_cast" change should be reviewed.

This version might break compatibility with Python 3.6.x. So what is still missing are some preprocessor commands that switch between old and new code depending on the python version.

Maybe someone knows how to implement this most elegantly?

[serhiy-storchaka]

Seems these files are generated by Cython. Just upgrade Cython to the version that supports Python 3.7 and regenerate files.

[raymondEhlers]

Following @serhiy-storchaka advice, I regenerated the cython files with

$ make clean-cython
$ make cython

I noticed that there are a number of local paths in the generated files - I'm not sure if this is normal for cython, but it is causing a rather large number of small changes. In any case, all tests pass with python 3.7 except for one (root_numpy.tmva.tests.test_tmva_multiclass), but I think it failed before these changes. I also can't trivially change python and ROOT versions right now, so I haven't been able to test how this impacts other python or ROOT versions.

Although the caveats above should be kept in mind, I opened PR #368 with the changes.