Added missing np.import_array()

[da-woods]

In principle this should always be called before using cimported
Numpy from Cython files. It becomes more important on recent
versions of Cython (since they use the Numpy API rather than
directly looking into the objects)

Reference issue

#14732

What does this implement/fix?

Ensures that np.import_array() is always called any-time that numpy is cimported

[tylerjereddy]

Do we want to remove the Cython version pin from gh-14801 here so we can see the CI pass with latest Cython alpha? Is this meant to guard against the issue on the latest alpha?

IIUC the related NumPy PR will not solve the alpha release problem: numpy/numpy#20042

[da-woods]

Do we want to remove the Cython version pin from gh-14801 here so we can see the CI pass with latest Cython alpha? Is this meant to guard against the issue on the latest alpha?

Yes - that's probably a good idea to show that it genuinely has fixed what I believe it has.

IIUC the related NumPy PR will not solve the alpha release problem: numpy/numpy#20042

I think it would solve it if merged but it's probably best fixed here and in Cython.

[tirthasheshpatel]

Here are some more files in stats that I think need np.import_array():

• scipy/stats/_qmc_cy.pyx
• scipy/stats/biasedurn.pyx.templ
• scipy/stats/_unuran/unuran_wrapper.pyx.templ

[da-woods]

scipy/stats/_qmc_cy.pyx has it already

scipy/scipy/stats/_qmc_cy.pyx

Line 12 in 0785d6f

np.import_array()

I missed templ files (and a few other auto-generated files) when I did the search so I've added them now - thanks

[ev-br]

So, am I understanding this correctly: from now on, any cimport numpy as np will need to be followed by np.import_array() ?

A user in me believes there are reasons why Cython does not do it automatically, but then is it documented somewhere? It should be --- this can possibly trip quite a bit of one-off small-script downstream user code.

[da-woods]

So, am I understanding this correctly: from now on, any cimport numpy as np will need to be followed by np.import_array() ?

In principle this was always the case. Cython 3 changes to how things like .shape are accessed makes it much more sensitive to this though (since they now use the imported Numpy API instead of looking directly in the C struct)

A user in me believes there are reasons why Cython does not do it automatically, but then is it documented somewhere? It should be --- this can possibly trip quite a bit of one-off small-script downstream user code.

Cython 3 actually does try to do it automatically if the user forgets and emits a reminder warning. A recent change in Numpy broke that (although it should now be fixed in Cython).

I can't find it in the Cython documentation though, which looks like an omission.

Essentially the Cython view is that users should call import_array and probably should always have been calling it. However, we do make a best-effort attempt to do it ourselves.

[ev-br]

Thanks @da-woods , this is helpful. So a user recommendation used to be "import_array when using PyArray_... functions directly", now it's "always import_array".

While this is OT for this PR (and the PR LGTM), I'd ask to document this as thorough as realistic on the Cython side.

I'm on to updating my teaching materials and small projects where I contributed Cython bits and pieces :-).

[tupui]

Thanks @da-woods. So if I got this correctly, if we want to use NumPy in Cython, the canonical way should be to add

import numpy as np
cimport numpy as np  # and sometimes I see this as cnp. I have no clue what I should do
np.import_array()

Is that right? I just got to play with Cython while working on the QMC module so I am really new to these.

[da-woods]

Thanks @da-woods. So if I got this correctly, if we want to use NumPy in Cython, the canonical way should be to add

import numpy as np
cimport numpy as np  # and sometimes I see this as cnp. I have no clue what I should do
np.import_array()

Is that right? I just got to play with Cython while working on the QMC module so I am really new to these.

Yes that's right. np vs cnp is fine either way - it's just personal choice. I think some people find it clearer to use a different name for the runtime and compile time definitions.

You only need cimport numpy (and np.import_array()) if you're planning to use "Cython features" with Numpy (for example, typing something as np.ndarray). If the code would work unchanged in Python then you only need the import

[tylerjereddy]

Sounds like @ev-br is ok with this PR as well, and the other queries here are just us learning about Cython I think.

Squash-merged and marked for backport, thanks for engaging from the Cython side @da-woods.

[ev-br]

Followed up on the cython-users mailing list, here's a link for completeness https://groups.google.com/g/cython-users/c/6qGmzvIv7Io