CI: Troubleshoot CI

[jbrockmendel]

• closes #xxxx
• tests added / passed
• Ensure all linting tests pass, see here for how to run them
• whatsnew entry

Tacked some CI-troubleshooting stuff onto the just-started FIXMES21 branch.

[jreback]

great lmk when ready

[jbrockmendel]

Looks like npdev changed ndarray.mean s.t. np.arange(10).view('m8[ns]').mean() returns a float instead of a td64 obj cc @seberg

[seberg]

Wow, yet another weird case :(. Our mean code dares to force cast to "f8" for timedeltas, so we effectively call:

arr.sum(dtype="f8")

but, happy coincidence apparently the old datetime resolution is broken and just ignores the dtype request... Although, I am still surprised why it doesn't just fail in the new code.

Well, I guess, mean is arguably broken, if I get luck reverting numpy/numpy#20544 (comment) will "fix" this too (i.e. revert to the old borked up state).

[jreback]

@jbrockmendel if you can fix this up

[jbrockmendel]

waiting for a fix in npdev

[seberg]

Should be fixed, not sure if the nightly is build though.

[jreback]

waiting for a fix in npdev

ok if not fixed, then can you disable tests?

[jbrockmendel]

will re-push and find out

[jbrockmendel]

Looks like the td64.mean may have gotten fixed but not we're seeing a bunch of new warnings (guessing division by zero related). will dig into this in a bit

[seberg]

:(, sorry about this, its been a confusing can of worms. One thing that could still be changed that if you provide out= to a reduction, you may end up with a more precise loop (which I suppose could mean float if the array is an integer, but the out is a float).

But I guess this is something awfully gone wrong, or just a different error?:

numpy.core._exceptions._UFuncBinaryResolutionError: ufunc 'divide' cannot use operands with types dtype('int8') and dtype('<m8[ns]')

Where apparently the type resolution is somehow failing (which confuses me a lot, because I am not sure why this would have been any better with the previous version.)

[jbrockmendel]

But I guess this is something awfully gone wrong, or just a different error?:

Yah in these cases we're currently testing that we get a TypeError, so getting a numpy.core._exceptions._UFuncBinaryResolutionError instead isn't the worst thing in the world; if it can't/won't be changed upstream then we can try to catch and re-raise

[seberg]

Ah OK, puh, I was seriously worried it was not an error before... That error should be inheriting from TypeError, so I think I would prefer to just ignore it, at least from the NumPy side. But please complain if you think that is annoying.

[jbrockmendel]

I think the exception is fine. The failing tests look like a new warning which im assuming is division by zero, which is also pretty reasonable so we can adjust to handle on our end.

[seberg]

Cool, thanks. Let me know if something is looking shady there, so I can have a closer look.

[jreback]

@jbrockmendel if you have a chance to fix up here (i don't care if you xfail these tests) but cannot merge anything else until we are back to green.

[jbrockmendel]

its gonna take a couple more rounds on the CI bc i cant get npdev working locally

[jbrockmendel]

It looks very tentatively like a lot of np_obj.__op__(pd_obj) calls are going through pd_obj__array_ufunc__ instead of returning NotImplemented, which we want so they can use pd_obj.__rop__(np_obj)

[jreback]

maybe worthile to simply fix the numpy dev wheel to a working one while this is fixed?

[jbrockmendel]

maybe worthile to simply fix the numpy dev wheel to a working one while this is fixed?

first should confirm with @seberg whether the guess about __array_ufunc__ is correct, and if so if it is something they intend to fix

[jreback]

sorry what i mean is pin the wheel to a working one from 2 days ago

[jbrockmendel]

i guess. no idea how to do that

[lithomas1]

numpy==1.23.0.dev0+101.ga81535a36 in the yaml files?

[jbrockmendel]

@lithomas1 can you pinch-hit on this one?

[lithomas1]

Sure, #44834 is my attempt. We should still keep this PR open to revert the pin in there and come up with a long-term fix.

[jbrockmendel]

sounds good. ive confirmed locally that the td64 object is not respecting __array_priority__ on numpy:main

[seberg]

@jbrockmendel sorry, only catching up now/tomorrow. I cannot think of immediate changes related to __array_priority__ or binary op's behaviour. This is for numpy's timedelta64 scalars?

[seberg]

(I guess one thing I am confused, is because if __array_ufunc__ is defined, I believe the design is that the binary op's go through __array_ufunc__ and never return NotImplemented, unless __array_ufunc__ is None?)

[seberg]

Ah, I had looked at the wrong CI run yesterday, I think. But I am still confused what the issue is? Do you have an example?

Btw. assuming you are on linux, one quick hack that may work to get numpy dev, is:

python3 runtests.py --shell

to drop in a shell with numpy dev version injected into the PYTHONPATH.

[jbrockmendel]

I got npdev working locally by installing from source instead of using the wheel. So I can confirm that a) the timedelta64.__truediv__ method is not returning NotImplemented as I think it should and b) that actually isn't new (numpy/numpy#17017), so isn't the explanation here.

[jbrockmendel]

Found a likely candidate for a bug in our __array_ufunc__ implementation. just pushed.

[jbrockmendel]

Numpy pin reverted + green

[jreback]

great

going to merge but is there anything we should backport to 1.3.5?

[jbrockmendel]

if we want 1.3.5 to work with npdev then the one line in ops_dispatch.pyx would need to be backported i think

[jreback]

ok let's try to backport this and then

[jreback]

@meeseeksdev backport 1.3.x

[lumberbot-app]

Something went wrong ... Please have a look at my logs.