Compilation failures with Cython 3.0

[mgorny]

Describe the bug

When building against Cython 3.0.0b2, I get the following errors:

Error output

Error compiling Cython file:
------------------------------------------------------------
...
    # smooth
    for i in range(n):
        prev = (n+i-1) % n

        # s[t-m] = xhat[prev, 2+m-1]
        yhat[i] = (xhat[prev, 0] * xhat[prev, 1]**phi) + xhat[prev, 2+m-1]
                                                       ^
------------------------------------------------------------

statsmodels/tsa/exponential_smoothing/_ets_smooth.pyx:156:55: Cannot assign type 'npy_float64 complex' to 'float64_t'

Error compiling Cython file:
------------------------------------------------------------
...

        # s[t-m] = xhat[prev, 2+m-1]
        yhat[i] = (xhat[prev, 0] * xhat[prev, 1]**phi) + xhat[prev, 2+m-1]
        # l_t = a * (y_t - s_t-m) + (1-a) * (l_t-1 * b_t-1**phi)
        xhat[i, 0] = (alpha * (y[i] - xhat[prev, 2+m-1])
                      + (1 - alpha) * (xhat[prev, 0] * xhat[prev, 1]**phi))
                      ^
------------------------------------------------------------

statsmodels/tsa/exponential_smoothing/_ets_smooth.pyx:159:22: Cannot assign type 'npy_float64 complex' to 'float64_t'

Error compiling Cython file:
------------------------------------------------------------
...
        # l_t = a * (y_t - s_t-m) + (1-a) * (l_t-1 * b_t-1**phi)
        xhat[i, 0] = (alpha * (y[i] - xhat[prev, 2+m-1])
                      + (1 - alpha) * (xhat[prev, 0] * xhat[prev, 1]**phi))
        # b_t = (b*) * (l_t / l_t-1) + (1 - (b*)) * b_t-1**phi
        xhat[i, 1] = (beta_star * (xhat[i, 0] / xhat[prev, 0])
                      + (1 - beta_star) * xhat[prev, 1]**phi)
                      ^
------------------------------------------------------------

statsmodels/tsa/exponential_smoothing/_ets_smooth.pyx:162:22: Cannot assign type 'npy_float64 complex' to 'float64_t'

Error compiling Cython file:
------------------------------------------------------------
...
    # smooth
    for i in range(n):
        prev = (n+i-1) % n

        # s[t-m] = xhat[prev, 2+m-1]
        yhat[i] = (xhat[prev, 0] * xhat[prev, 1]**phi) * xhat[prev, 2+m-1]
                                                       ^
------------------------------------------------------------

statsmodels/tsa/exponential_smoothing/_ets_smooth.pyx:191:55: Cannot assign type 'npy_float64 complex' to 'float64_t'

Error compiling Cython file:
------------------------------------------------------------
...

        # s[t-m] = xhat[prev, 2+m-1]
        yhat[i] = (xhat[prev, 0] * xhat[prev, 1]**phi) * xhat[prev, 2+m-1]
        # l_t = a * (y_t / s_t-m) + (1-a) * (l_t-1 * b_t-1**phi)
        xhat[i, 0] = (alpha * (y[i] / xhat[prev, 2+m-1])
                      + (1 - alpha) * (xhat[prev, 0] * xhat[prev, 1]**phi))
                      ^
------------------------------------------------------------

statsmodels/tsa/exponential_smoothing/_ets_smooth.pyx:194:22: Cannot assign type 'npy_float64 complex' to 'float64_t'

Error compiling Cython file:
------------------------------------------------------------
...
        # l_t = a * (y_t / s_t-m) + (1-a) * (l_t-1 * b_t-1**phi)
        xhat[i, 0] = (alpha * (y[i] / xhat[prev, 2+m-1])
                      + (1 - alpha) * (xhat[prev, 0] * xhat[prev, 1]**phi))
        # b_t = (b*) * (l_t / l_t-1) + (1 - (b*)) * b_t-1**phi
        xhat[i, 1] = (beta_star * (xhat[i, 0] / xhat[prev, 0])
                      + (1 - beta_star) * xhat[prev, 1]**phi)
                      ^
------------------------------------------------------------

statsmodels/tsa/exponential_smoothing/_ets_smooth.pyx:197:22: Cannot assign type 'npy_float64 complex' to 'float64_t'

While this is still a beta, it means that the release is just around the corner and we're trying to report problems early. Furthermore, some packages already decided to require Cython-3 which makes all incompatibilities a true PITA for Linux distributions.

Output of import statsmodels.api as sm; sm.show_versions()

INSTALLED VERSIONS

Python: 3.10.11.final.0
OS: Linux 6.2.14-gentoo-dist #1 SMP PREEMPT_DYNAMIC Mon May 1 15:03:02 -00 2023 x86_64
byteorder: little
LC_ALL: None
LANG: C.UTF8

statsmodels

Installed: 0.14.0 (/usr/lib/python3.10/site-packages/statsmodels)

Required Dependencies

cython: 0.29.34 (/usr/lib/python3.10/site-packages/Cython)
numpy: 1.24.3 (/usr/lib/python3.10/site-packages/numpy)
scipy: 1.10.1 (/usr/lib/python3.10/site-packages/scipy)
pandas: 2.0.1 (/usr/lib/python3.10/site-packages/pandas)
dateutil: 2.8.2 (/usr/lib/python3.10/site-packages/dateutil)
patsy: 0.5.3 (/usr/lib/python3.10/site-packages/patsy)

Optional Dependencies

matplotlib: 3.7.1 (/usr/lib/python3.10/site-packages/matplotlib)
backend: agg
cvxopt: Not installed
joblib: 1.1.0 (/usr/lib/python3.10/site-packages/joblib)

Developer Tools

IPython: 8.13.2 (/usr/lib/python3.10/site-packages/IPython)
jinja2: 3.1.2 (/usr/lib/python3.10/site-packages/jinja2)
sphinx: 6.2.1 (/usr/lib/python3.10/site-packages/sphinx)
pygments: 2.15.1 (/usr/lib/python3.10/site-packages/pygments)
pytest: 7.3.1 (/usr/lib/python3.10/site-packages/pytest)
virtualenv: 20.23.0 (/usr/lib/python3.10/site-packages/virtualenv)

[bashtage]

This would be someday and likely not until statsmodels 0.15 which is about a year away.

[bashtage]

These appear to be Cython bugs. Have they been reported to Cython?

[mgorny]

These appear to be Cython bugs. Have they been reported to Cython?

Why would they? FWICS it's Cython becoming more strict and rejecting addition of incompatible types.

[bashtage]

Because they are deciding to use a power function that returns a complex value rather than the double function double pow(double x, double y). If we wanted a complex, would would have specifically asked for complex.

[bashtage]

np.float64(-1.0) ** np.float64(0.5)

does not return a complex. It warns and returns np.float64

[mgorny]

Because they are deciding to use a power function that returns a complex value rather than the double function double pow(double x, double y). If we wanted a complex, would would have specifically asked for complex.

Apparently it's a deliberate change (but FWIU can be trivially changed back): https://docs.cython.org/en/latest/src/userguide/migrating_to_cy30.html#power-operator

[bashtage]

Thanks for noting the change. I still think cython 3 won't have any support until 0.15 since changing Cython seems like a big deal.

[mgorny]

My preference would be to keep both versions supported. I'm not sure how feasible that is with a large code base though.

[josef-pkt]

Will there be a problem if packages are compiled with different major versions of cython?
e.g. the cython interface to scipy.linalg BLAS/LAPACK

[bashtage]

Cython is not working well. It is casing c floats to doubles, then performing double complex ops, and then won't cast back to C float.