`xarray>=2025.1.2` has inconsistent treatment of `np.datetime64` and `datetime`

[DahnJ]

What is your issue?

In xarray prior to 2025.1.2 usage of np.datetime64 and datetime resulted in identical result

from datetime import datetime
import numpy as np
import xarray as xr

a = xr.DataArray(coords={'time': [np.datetime64('2020')]}, dims=['time'])
b = xr.DataArray(coords={'time': [datetime(2020,1,1)]}, dims=['time'])

print(a['time'].dtype)
print(b['time'].dtype)

would yield

datetime64[ns]
datetime64[ns]

From version 2025.1.2, this is no longer true, and we get

datetime64[s]
datetime64[ns]

Together with the equality check failing (pandas-dev/pandas#55694), this means comparing DataArrays and Datasets is now subtly dependent on whether datetime or np.datetime64 was used to construct them.

Is this the intended behaviour? Is there a recommended way to deal with this difference beyond requiring the users to always specify np.datetime(..., 'ns')?

[spencerkclark]

In some sense, yes, in alignment with pandas we did intentionally relax the previous automatic casting of all datetime-like objects passed to xarray to nanosecond resolution. For certain use cases it is beneficial to be able to use coarser resolution times. However we agree the dtype equality issue for DatetimeIndex-backed variables is a particularly awkward unintended consequence (see also #10045). Ideally this could be addressed upstream.

In terms of resolutions, specifically "s", "ms", "us", and "ns" are now permitted. Following pandas, coarser resolution np.datetime64 values are cast to "s" resolution, and finer resolution values are cast to "ns" resolution, but anything in between passes through unmodified. Xarray similarly defers to pandas on how to translate datetime.datetime objects into np.datetime64 values. For better or worse pandas still always chooses "ns" resolution in that context.

To the extent that we support datetime values with different resolutions, it seems somewhat inevitable that without special care users will end up with DataArrays of different dtypes, but it would be nice if equality was not sensitive to dtype.

[floriandeboissieu]

From the following experiment, I suppose that Dask may not have made this alignment to pandas changes yet, which can also lead to awkward results when using dates as data, converting to integer for example:

import dask
import numpy as np
import xarray as xr

data = np.array("2016-01-01").astype("datetime64[s]")

da1 = xr.DataArray(data)
da2 = xr.DataArray(dask.array.from_array(data))

assert da1.equals(da2)

assert da1.astype("datetime64[M]").dtype == '<M8[s]'
assert da2.astype("datetime64[M]").dtype == '<M8[M]'

print(da1.astype("datetime64[M]").astype(int).data)
# 1451606400
print(da2.astype("datetime64[M]").astype(int).compute().data)
# 552

[spencerkclark]

Thanks @floriandeboissieu—I think this is a separate issue that was present even before we updated xarray to support non-nanosecond resolution datetime64 values. It has to do with the fact that the code xarray uses to coerce non-supported datetime64 resolution values to supported resolution values currently acts only once the data is in memory (i.e. after dask compute).

dask.array.Array aims to mirror NumPy, and supports all NumPy datetime64 resolutions, so I expect its behavior will stay the same (no coercion to different resolutions); in other words to change this we would need to make changes to xarray. Do you think you could open up a new issue to discuss?

[floriandeboissieu]

@spencerkclark thanks for your reply, and sorry for the noise... I opened a separate issue.