ci: deactivate a randomly failing answer test

[neutrinoceros]

PR Summary

This one answer test has been randomly failing in all PRs since the release of numpy 0.20.0 a week ago.
I suggest to deactivate it at least temporarily (I added a comment to explain the situation).

[brittonsmith]

I'm worried that this will result in this never getting fixed. There should probably be at least an issue (maybe even a blocking one) so it doesn't get forgotten.

[neutrinoceros]

Fair point; here you go #3069

[munkm]

Ok I've added the 4.0 label to that issue so we make sure to address it before the release.

I do worry about deactivating this test -- we might introduce other errors. That said, we might still attribute it to the random failure pre-fixing so I'm really not sure of the best approach here.

[neutrinoceros]

Right now it seems to me the test is mostly catching external errors out of our control 🤷🏻‍♂️, but I agree that I'm not happy about the solution either.

[cphyc]

All my PRs are failing randomly because of this bug... I am definitely in favour of having it silenced for the moment until we figure out a solution. Given an issue has been filled, I think it is safe to merge this one.

[cphyc]

Also @munkm the bug is actually a pixel-wise difference in an answer test, and both the old and new answer seem sensible to me. I guess the issue is that we can't have both answers at the same time, except if we pin a particular version of numpy, can we?

[neutrinoceros]

we could maybe adjust the rms tolerance but I think it's not a priority now

[munkm]

I guess the issue is that we can't have both answers at the same time, except if we pin a particular version of numpy, can we?

Yeah I don't think we can have both? Or, I don't think it's the hassle of figuring out answers for both versions right now. I will merge this and hopefully we can reactivate the test soon.