`numexpr` still uses `NPY_ARRAY_UPDATEIFCOPY`

[seberg]

We deprecated NPY_ARRAY_UPDATEIFCOPY a pretty long time ago, but with a Python warning. There is one (probably almost never used) branch that still has an NPY_ARRAY_UPDATEIFCOPY . NumPy main currently removes the symbol entirely, so this will not work. And if compiled on older NumPy versions, it would just do the wrong thing (not copy back).

It is in an n_inputs == 0 branch, is that branch meaningless enough, or should we fix this here, and delay the deprecation for one release in NumPy for the sake of numexpr?

Here is the code:

numexpr/numexpr/interpreter.cpp

Line 1271 in d745933

a = (PyArrayObject *)PyArray_FromArray(operands[0], dtypes[0],

[robbmcleod]

I've managed to figure out how to replicate this case:

import numpy as np
import numexpr as ne
x = np.empty(1, dtype=int)
ne.evaluate('1', out=x)

Just ne.evaluate('1') doesn't go into this branch. My concern (as always) is whether it could effect Pandas or PyTables as they are downstream.

I'm not really sure what you mean about a Python warning as I don't get one when running the above code snippet? Maybe because it's in the C-API?

If I just remove the NPY_ARRAY_UPDATEIFCOPY flag it seems to compile and run just fine.

[seberg]

It should give a warning, try running with python -Wall or something. It was a DeprecationWarning, so it may be hidden from you as an "end-user".

If you are unsure/worried, I guess we should probably revert this, and add a deprecation macro for at least one version to flush out untested code paths.

[seberg]

If I just remove the NPY_ARRAY_UPDATEIFCOPY flag it seems to compile and run just fine.

The result should be incorrect though? I.e. return out unmodified? This has the potential of silent error, if it was just an exception, I probably would not bother.

[robbmcleod]

No, it definitely works as intended, or as I assume it was to be, since I didn't write any of the code we're talking about:

x = np.zeros(1, dtype=int)
print(f"pre-eval, x = {x} at {x.__array_interface__['data'][0]}")
ne.evaluate('3', out=x)
print(f"post-eval, x = {x} at {x.__array_interface__['data'][0]}")

outputs:

pre-eval, x = [0] at 1866944975104
n_inputs = 0
operands[0] = 000001B2AF069930, retsig = i
Case with NPY_ARRAY_UPDATEIFCOPY
post-eval, x = [3] at 1866944975104

In which version of NumPy will this flag be removed, and when will the release be?

[seberg]

As of now, it would be the next version, i.e. 1.23.0 (or maybe 2.0 if that happens), in about 6 months.

Sorry for the noise. Looking closer, the bug is just that the flag is used at all :). The flag only makes sense if you also set the base object (or better call PyArray_SetUpdateIfCopyBase), which you do not do.

So nothing wrong will happy, because the flag should just be ignored I think.

[robbmcleod]

Ok, don't worry about it then. I'll just remove it from the code.

[robbmcleod]

There's no issue with NPY_ITER_UPDATEIFCOPY? I think not from the NumPy issues tracker, but wanted to ask to be sure.

Closing otherwise.

[seberg]

Wait, no, arrrggg. NumPy does this for you, so I think it should be used, but here, probably only in the crazy unlikely case that the input is an unaligned array.

EDIT: The mistake here may be that we gave a warning only when deleting such an array. But that means that no warning was ever given when the flag was passed here, but ended up not having an effect.

[seberg]

NPY_ITER_UPDATEIFCOPY is fine, it uses the newer WRITEBACK_IF_COPY internally for the arrays. A bit confusing, but for the iterator it was possible to keep things as they were, for UPDATEIFCOPY it was not.

[robbmcleod]

Is there any way to make NumPy generate an unaligned array? In NumPy parlance aligned means on a 16-bit basis right? Maybe with np.frombuffer?

[seberg]

It is a bit tricky, the easiest way is this, IMO: np.empty(5, dtype=np.uint8)[1:].view(np.int32).

[robbmcleod]

Yes this fails:

x = np.empty(5, dtype=np.uint8)[1:].view(np.int32)
print(f"pre-eval, x = {x} at {x.__array_interface__['data'][0]}")
ne.evaluate('3', out=x)
print(f"post-eval, x = {x} at {x.__array_interface__['data'][0]}")

outputs:

pre-eval, x = [217964907] at 2254816281185
n_inputs = 0
operands[0] = 0000020CFDE699F0, retsig = i
Case with NPY_ARRAY_UPDATEIFCOPY
post-eval, x = [217964907] at 2254816281185

So it is no longer setting x to 3. Any suggestions on possible fixes? I am pretty familiar with the NumPy C-API but not the flag issues here.