Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MNT: more compatibility with NumPy 2.0 #15235

Merged
merged 7 commits into from Aug 29, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion astropy/io/ascii/cparser.pyx
Expand Up @@ -733,7 +733,7 @@ cdef class CParser:
if nrows != -1:
num_rows = nrows

cdef np.ndarray col = np.empty(num_rows, dtype=np.float_)
cdef np.ndarray col = np.empty(num_rows, dtype=np.float64)
cdef double converted
cdef int row = 0
cdef double *data = <double *> col.data
Expand Down
5 changes: 3 additions & 2 deletions astropy/io/ascii/tests/test_c_reader.py
Expand Up @@ -4,6 +4,7 @@
import io
import os
import re
import sys
from contextlib import nullcontext
from io import BytesIO
from textwrap import dedent
Expand Down Expand Up @@ -1243,7 +1244,7 @@ def test_data_out_of_range(parallel, fast_reader, guess):
fast_reader["parallel"] = parallel
if fast_reader.get("use_fast_converter"):
rtol = 1.0e-15
elif np.iinfo(np.int_).dtype == np.dtype(np.int32):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could imagine that the windows failures are related to this change. I just got the sys.maxsize replacement from stackoverflow. But I guess on 32bit one could also test if np.intp == np.int32.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is obviously not triggering the pytest.xfail, but it is also not failing – just missing 1 expected warning, so maybe that part should instead be changed below.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pytest refuses to run locally on Windows for other reasons and I don't have time to fix it. I think to match existing behavior, it is best to just xfail if win32 is detected. I added a commit to do this.

elif (sys.maxsize < 2**32) or (sys.platform == "win32"):
# On 32bit the standard C parser (strtod) returns strings for these
pytest.xfail("C parser cannot handle float64 on 32bit systems")

Expand Down Expand Up @@ -1358,7 +1359,7 @@ def test_data_at_range_limit(parallel, fast_reader, guess):
fast_reader["parallel"] = parallel
if fast_reader.get("use_fast_converter"):
rtol = 1.0e-15
elif np.iinfo(np.int_).dtype == np.dtype(np.int32):
elif (sys.maxsize < 2**32) or (sys.platform == "win32"):
# On 32bit the standard C parser (strtod) returns strings for these
pytest.xfail("C parser cannot handle float64 on 32bit systems")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another case where we don't even seem to need the xfail but are instead missing a warning. Strangely the failure is

  with pytest.warns() as warning_lines:

E Failed: DID NOT WARN. No warnings of type (<class 'Warning'>,) were emitted.
E The list of emitted warnings is: [].

while the test below in L1410 actually expects len(warning_lines) in (0, 1) – looks like that part has already been broken for 0 warnings before, probably with some pytest update.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But there isn't any new pytest release this past weekend.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, this would have broken already when changing this from pytest.warns(None) to pytest.warns() in #12823 (comment) – this overlooked that there was a case where 0 warnings were expected, my reading of this is that only 32bit systems do not raise any warning here and thus the error was never discovered while the xfail was in effect.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the Windows job really 32-bit though? The log says MSC v.1929 64 bit (AMD64) even though it is win32. 🤯

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is passing all the reads up to e.g. https://github.com/astropy/astropy/actions/runs/5997690440/job/16264575442#step:10:1868
just like any 64-bit system, so is not failing the precision test, just not emitting an expected warning in 2 cases.


Expand Down
1 change: 0 additions & 1 deletion astropy/io/misc/tests/test_yaml.py
Expand Up @@ -41,7 +41,6 @@
2.0,
np.float64(),
3 + 4j,
np.complex_(3 + 4j),
np.complex64(3 + 4j),
np.complex128(1.0 - 2**-52 + 1j * (1.0 - 2**-52)),
],
Expand Down
5 changes: 2 additions & 3 deletions astropy/io/misc/yaml.py
Expand Up @@ -252,7 +252,6 @@ def _represent_tuple(self, data):
# Numpy dtypes
AstropyDumper.add_representer(np.bool_, yaml.representer.SafeRepresenter.represent_bool)
for np_type in [
np.int_,
np.intc,
np.intp,
np.int8,
Expand All @@ -267,11 +266,11 @@ def _represent_tuple(self, data):
AstropyDumper.add_representer(
np_type, yaml.representer.SafeRepresenter.represent_int
)
for np_type in [np.float_, np.float16, np.float32, np.float64, np.longdouble]:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So for users with numpy versions below 2 that used such data types, these will not be supported anymore?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

np.float_ is just an alias of np.float64, i.e., if on numpy <2 you use it, it will still work:

In [7]: np.float_ is np.float64
Out[7]: True

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, thanks for the clarification!

for np_type in [np.float16, np.float32, np.float64, np.longdouble]:
AstropyDumper.add_representer(
np_type, yaml.representer.SafeRepresenter.represent_float
)
for np_type in [np.complex_, complex, np.complex64, np.complex128]:
for np_type in [complex, np.complex64, np.complex128]:
AstropyDumper.add_representer(np_type, _complex_representer)

AstropyLoader.add_constructor("tag:yaml.org,2002:python/complex", _complex_constructor)
Expand Down
7 changes: 2 additions & 5 deletions astropy/io/votable/converters.py
Expand Up @@ -4,7 +4,6 @@
to/from TABLEDATA_ and BINARY_ formats.
"""


# STDLIB
import re
import sys
Expand Down Expand Up @@ -1362,13 +1361,11 @@ def get_converter(field, config=None, pos=None):
np.int64().dtype.num: "long",
np.complex64().dtype.num: "floatComplex",
np.complex128().dtype.num: "doubleComplex",
np.unicode_().dtype.num: "unicodeChar",
mhvk marked this conversation as resolved.
Show resolved Hide resolved
np.str_().dtype.num: "unicodeChar",
np.bytes_().dtype.num: "char",
}


numpy_dtype_to_field_mapping[np.bytes_().dtype.num] = "char"


def _all_matching_dtype(column):
first_dtype = False
first_shape = ()
Expand Down
10 changes: 4 additions & 6 deletions astropy/io/votable/tests/test_vo.py
Expand Up @@ -11,16 +11,14 @@
import sys
from unittest import mock

import numpy as np

# THIRD-PARTY
import numpy as np
import pytest
from numpy.testing import assert_array_equal

# LOCAL
from astropy.io.votable import tree
from astropy.io.votable.exceptions import W39, VOTableSpecError, VOWarning

# LOCAL
from astropy.io.votable.table import parse, parse_single_table, validate
from astropy.io.votable.xmlutil import validate_schema
from astropy.utils.data import get_pkg_data_filename, get_pkg_data_filenames
Expand Down Expand Up @@ -298,7 +296,7 @@ def test_string_test(self):
)

def test_fixed_string_test(self):
assert issubclass(self.array["string_test_2"].dtype.type, np.unicode_)
mhvk marked this conversation as resolved.
Show resolved Hide resolved
assert issubclass(self.array["string_test_2"].dtype.type, np.str_)
assert_array_equal(
self.array["string_test_2"], ["Fixed stri", "0123456789", "XXXX", "", ""]
)
Expand All @@ -311,7 +309,7 @@ def test_unicode_test(self):
)

def test_fixed_unicode_test(self):
assert issubclass(self.array["fixed_unicode_test"].dtype.type, np.unicode_)
assert issubclass(self.array["fixed_unicode_test"].dtype.type, np.str_)
assert_array_equal(
self.array["fixed_unicode_test"],
["Ceçi n'est", "வணக்கம்", "0123456789", "", ""],
Expand Down
21 changes: 6 additions & 15 deletions astropy/modeling/tests/test_constraints.py
Expand Up @@ -15,7 +15,6 @@
from astropy.modeling.core import Fittable1DModel
from astropy.modeling.parameters import Parameter
from astropy.utils import minversion
from astropy.utils.compat.numpycompat import NUMPY_LT_2_0
from astropy.utils.compat.optional_deps import HAS_SCIPY
from astropy.utils.exceptions import AstropyUserWarning

Expand Down Expand Up @@ -184,6 +183,7 @@ def test_bounds_slsqp(self):
assert intercept + 10**-5 >= bounds["intercept"][0]
assert intercept - 10**-5 <= bounds["intercept"][1]

@pytest.mark.filterwarnings("ignore:The fit may be unsuccessful")
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@WilliamJamieson , if you are not okay with this, please open follow-up PR. Thanks!

@pytest.mark.parametrize("fitter", fitters)
def test_bounds_gauss2d_lsq(self, fitter):
fitter = fitter()
Expand All @@ -204,21 +204,12 @@ def test_bounds_gauss2d_lsq(self, fitter):
theta=0.5,
bounds=bounds,
)
if isinstance(fitter, (fitting.LevMarLSQFitter, fitting.DogBoxLSQFitter)):
with pytest.warns(AstropyUserWarning, match="The fit may be unsuccessful"):
model = fitter(gauss, X, Y, self.data)
if isinstance(fitter, fitting.TRFLSQFitter):
ctx = np.errstate(invalid="ignore", divide="ignore")
else:
ctx2 = nullcontext()
if isinstance(fitter, fitting.TRFLSQFitter):
ctx = np.errstate(invalid="ignore", divide="ignore")
if not NUMPY_LT_2_0 or not SCIPY_LT_1_11_2:
ctx2 = pytest.warns(
AstropyUserWarning, match="The fit may be unsuccessful"
)
else:
ctx = nullcontext()
with ctx, ctx2:
model = fitter(gauss, X, Y, self.data)
ctx = nullcontext()
with ctx:
model = fitter(gauss, X, Y, self.data)
x_mean = model.x_mean.value
y_mean = model.y_mean.value
x_stddev = model.x_stddev.value
Expand Down
2 changes: 1 addition & 1 deletion astropy/table/meta.py
Expand Up @@ -193,7 +193,7 @@ class ConvertError(ValueError):
"""Local conversion error used below."""

# Numpy types supported as variable-length arrays
np_classes = (np.floating, np.integer, np.bool_, np.unicode_)
mhvk marked this conversation as resolved.
Show resolved Hide resolved
np_classes = (np.floating, np.integer, np.bool_, np.str_)

try:
if len(col) == 0 or not all(isinstance(val, np.ndarray) for val in col):
Expand Down
2 changes: 1 addition & 1 deletion astropy/table/tests/test_column.py
Expand Up @@ -268,7 +268,7 @@ def test_item_access_type(self, Column):
Tests for #3095, which forces integer item access to always return a plain
ndarray or MaskedArray, even in the case of a multi-dim column.
"""
integer_types = (int, np.int_)
integer_types = (int, np.int32, np.int64)

for int_type in integer_types:
c = Column([[1, 2], [3, 4]])
Expand Down
12 changes: 6 additions & 6 deletions astropy/table/tests/test_row.py
Expand Up @@ -348,16 +348,16 @@ def test_uint_indexing():
that printing such a row works.

This is non-trivial: adding a signed and unsigned
integer in numpy results in a float, which is an
64 bit integer in numpy results in a float, which is an
invalid slice index.

Regression test for gh-7464.
"""
t = table.Table([[1.0, 2.0, 3.0]], names="a")
assert t["a"][1] == 2.0
assert t["a"][np.int_(1)] == 2.0
assert t["a"][np.uint(1)] == 2.0
assert t[np.uint(1)]["a"] == 2.0
assert t["a"][np.int64(1)] == 2.0
assert t["a"][np.uint64(1)] == 2.0
assert t[np.uint64(1)]["a"] == 2.0

trepr = [
"<Row index=1>",
Expand All @@ -368,8 +368,8 @@ def test_uint_indexing():
]

assert repr(t[1]).splitlines() == trepr
assert repr(t[np.int_(1)]).splitlines() == trepr
assert repr(t[np.uint(1)]).splitlines() == trepr
assert repr(t[np.int64(1)]).splitlines() == trepr
assert repr(t[np.uint64(1)]).splitlines() == trepr


def test_row_get():
Expand Down
4 changes: 2 additions & 2 deletions astropy/time/formats.py
Expand Up @@ -2177,11 +2177,11 @@

def _validate_jd_for_storage(jd):
if isinstance(jd, (float, int)):
return np.array(jd, dtype=np.float_)
return np.array(jd, dtype=float)
if isinstance(jd, np.generic) and (
jd.dtype.kind == "f" and jd.dtype.itemsize <= 8 or jd.dtype.kind in "iu"
):
return np.array(jd, dtype=np.float_)
return np.array(jd, dtype=float)

Check warning on line 2184 in astropy/time/formats.py

View check run for this annotation

Codecov / codecov/patch

astropy/time/formats.py#L2184

Added line #L2184 was not covered by tests
elif isinstance(jd, np.ndarray) and jd.dtype.kind == "f" and jd.dtype.itemsize == 8:
return jd
else:
Expand Down
6 changes: 3 additions & 3 deletions astropy/units/quantity_helper/function_helpers.py
Expand Up @@ -182,13 +182,13 @@ def __call__(self, f=None, helps=None, module=np):
# fmt: off
@function_helper(
helps={
np.copy, np.asfarray, np.real_if_close, np.sort_complex, np.resize,
np.copy, np.real_if_close, np.sort_complex, np.resize,
np.fft.fft, np.fft.ifft, np.fft.rfft, np.fft.irfft,
np.fft.fft2, np.fft.ifft2, np.fft.rfft2, np.fft.irfft2,
np.fft.fftn, np.fft.ifftn, np.fft.rfftn, np.fft.irfftn,
np.fft.hfft, np.fft.ihfft,
np.linalg.eigvals, np.linalg.eigvalsh,
}
mhvk marked this conversation as resolved.
Show resolved Hide resolved
} | ({np.asfarray} if NUMPY_LT_2_0 else set())
mhvk marked this conversation as resolved.
Show resolved Hide resolved
)
# fmt: on
def invariant_a_helper(a, *args, **kwargs):
Expand Down Expand Up @@ -442,7 +442,7 @@ def select(condlist, choicelist, default=0):
def piecewise(x, condlist, funclist, *args, **kw):
from astropy.units import Quantity

# Copied implementation from numpy.lib.function_base.piecewise,
# Copied implementation from numpy.lib._function_base_impl.piecewise,
# taking care of units of function outputs.
n2 = len(funclist)
# undocumented: single condition is promoted to a list of one condition
Expand Down
8 changes: 7 additions & 1 deletion astropy/units/tests/test_quantity_non_ufuncs.py
Expand Up @@ -17,7 +17,12 @@
TBD_FUNCTIONS,
UNSUPPORTED_FUNCTIONS,
)
from astropy.utils.compat import NUMPY_LT_1_23, NUMPY_LT_1_24, NUMPY_LT_1_25
from astropy.utils.compat import (
NUMPY_LT_1_23,
NUMPY_LT_1_24,
NUMPY_LT_1_25,
NUMPY_LT_2_0,
)

needs_array_function = pytest.mark.xfail(
not ARRAY_FUNCTION_ENABLED, reason="Needs __array_function__ support"
Expand Down Expand Up @@ -296,6 +301,7 @@ def test_copy(self):
copy = np.copy(a=self.q)
assert_array_equal(copy, self.q)

@pytest.mark.skipif(not NUMPY_LT_2_0, reason="np.asfarray is removed in NumPy 2.0")
@needs_array_function
def test_asfarray(self):
self.check(np.asfarray)
Expand Down
15 changes: 12 additions & 3 deletions astropy/utils/masked/core.py
Expand Up @@ -19,6 +19,7 @@

import numpy as np

from astropy.utils.compat import NUMPY_LT_2_0
from astropy.utils.data_info import ParentDtypeInfo
from astropy.utils.shapes import NDArrayShapeMethods

Expand Down Expand Up @@ -759,9 +760,17 @@
else:
# Parse signature with private numpy function. Note it
# cannot handle spaces in tuples, so remove those.
in_sig, out_sig = np.lib.function_base._parse_gufunc_signature(
ufunc.signature.replace(" ", "")
)
if NUMPY_LT_2_0:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mhvk , this might not be as clean as you like based on your previous changes. If this bothers you, please open follow-up PR. Thanks!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No worries, that looks fine! Thanks!

in_sig, out_sig = np.lib.function_base._parse_gufunc_signature(
ufunc.signature.replace(" ", "")
)
else:
(

Check warning on line 768 in astropy/utils/masked/core.py

View check run for this annotation

Codecov / codecov/patch

astropy/utils/masked/core.py#L768

Added line #L768 was not covered by tests
in_sig,
out_sig,
) = np.lib._function_base_impl._parse_gufunc_signature(
ufunc.signature.replace(" ", "")
)
axis = kwargs.get("axis", -1)
keepdims = kwargs.get("keepdims", False)
in_masks = []
Expand Down