ENH: special: add spherical Bessel functions to cython_special

[person142]

Reference issue

Closes gh-11029.

What does this implement/fix?

The spherical Bessel functions aren't currently available in Cython
special because they aren't ufuncs: they have an optional derivative
argument. But the underlying kernels are written in Cython and Cython
supports optional arguments in cdef mode, so add custom wrappers
that make them available.

Additional information

Most things in special are ufuncs, so I think it's reasonable to craft a
few custom wrappers for the remaining functions people care about.
The only other important one is probably zeta.

[Kai-Striega]

@person142 the changes look good to me. As a follow-up question, do you remember why the CI failed?

[pv]

Merge conflicts, otherwise LGTM.

I guess the typedef number_t is also suitable to be added permanently.

[rgommers]

There's one failure here that's real, on 32-bit Linux on Azure:

=================================== FAILURES ===================================
________________________ test_cython_api[spherical_kn] _________________________
[gw0] linux -- Python 3.6.9 /usr/bin/python3.6
scipy/special/tests/test_cython_special.py:340: in test_cython_api
    assert_allclose(cyval, pyval, err_msg="{} {} {}".format(pt, typecodes, signature))
E   AssertionError: 
E   Not equal to tolerance rtol=1e-07, atol=0
E   (1, (-1-1j), True) lDb ['double complex']
E   Mismatched elements: 1 / 1 (100%)
E   Max absolute difference: 6.57098077e-17
E   Max relative difference: 0.11837227
E    x: array(6.548581e-17+5.496905e-16j)
E    y: array(0.+5.551115e-16j)
        cy_spec_func = <cyfunction __pyx_fuse_0spherical_kn at 0xec13fb8c>
        cyfunc     = <cyfunction spherical_kn at 0xec13fc6c>
        cyval      = (6.548581121812447e-17+5.496905014501507e-16j)
        is_fused_code = [False, True, False]
        j          = 2
        knownfailure = None
        max_params = 3
        param      = (<function spherical_kn at 0xef40138c>, <cyfunction spherical_kn at 0xec13fc6c>, ('ld', 'ldb', 'lD', 'lDb'), None)
        pt         = (1, (-1-1j), True)
        pts        = [(-10, (-10-10j), True), (-10, (-10-10j), False), (-10, (-10-1j), True), (-10, (-10-1j), False), (-10, (-10+1j), True), (-10, (-10+1j), False), ...]
        pyfunc     = <function spherical_kn at 0xef40138c>
        pyval      = 5.551115123125783e-16j
        seen       = {('D', 'd'), ('b',), ('l',)}
        signature  = ['double complex']
        specializations = ('ld', 'ldb', 'lD', 'lDb')
        sup        = <numpy.testing._private.utils.suppress_warnings object at 0xdcf222cc>
        typecodes  = 'lDb'
        v          = {'b'}
        values     = [{'l'}, {'D', 'd'}, {'b'}]
        vv         = ('b',)
- generated xml file: /scipy/build/testenv/lib/python3.6/site-packages/junit/test-results.xml -
=========================== short test summary info ============================
FAILED scipy/special/tests/test_cython_special.py::test_cython_api[spherical_kn]
= 1 failed, 36213 passed, 1329 skipped, 131 xfailed, 6 xpassed in 831.64s (0:13:51) =

[person142]

There's one failure here that's real, on 32-bit Linux on Azure:

Yeah I need to track that down... I think I saw that the failure was 32 bit only and was like "boy that's going to be a PITA to deal with" and keep pushing it off...

[tylerjereddy]

@person142 @rgommers I used my 32-bit SciPy Docker setup to look into this--I can reproduce the failure with cython==0.29.14, but not with cython==0.29.19 (latest stable release).

We have historically bumped the minimum required Cython version pretty frequently without too much hesitation.

Perhaps we don't have to go quite as far as 0.29.19 from our current Cython>=0.29.14, but I'm just trying to help move this forward since the reviews are positive and one of us could "bisect" the minimum required Cython version if really needed.

On the other hand, perhaps the version susceptibility will clue you in to the nature of the original issue, but I'd probably suggest just bumping, esp. if just one or two minor versions would suffice.

[tylerjereddy]

Ok, I did the bisection myself: 0.29.18 is the minimum Cython version for this feature branch to pass full special test suite in 32-bit Docker container.

[person142]

Hm thanks for looking into that @tylerjereddy. Nothing in the relase notes leaps out at me immediately:

https://cython.readthedocs.io/en/latest/src/changes.html#id13

If we're uncomfortable just bumping I can look into it, though since it's a patch version bump we should probably bump anyway.

[tylerjereddy]

I went ahead and bumped to Cython 0.29.18 as minimum required in this PR, and rebased on latest master while I was at it.

I used a separate commit so you can always adjust/revert/whatever.

I tried to catch the other places where the Cython version needed to be documented as well, but probably good to check that.

The CI run should hopefully reveal any issues with the bump.

[tylerjereddy]

CI looks "ok" apart from the ARM64 timeout.

In theory adding more tests might contribute to that and having the low-level contribution tested on ARM might be useful, but I'd also like to make sure that the Cython version bump gets some time in CI, so I think I'll merge given positive reviewer feedback/CI and we'll keep an eye on the ARM test.

[person142]

Thanks for pushing this one through @tylerjereddy.