If xarray supports the numpy.datetime64 data format and so does zarr, why is xarray converting my numpy.datetime64 time array to a float array when I write it to zarr?

[christine-e-smit]

This all started because I'm trying to understand how xarray handles fill values and time arrays of type numpy.datetime64 (a.k.a. M8[ns]). My fill values disappeared for some reason in the conversion to zarr and I discovered that xarray was converting my time array to floating point for some reason.

Here is a simple example:

import numpy as np
import numpy.ma as ma
import xarray as xr

# Create a two element time array where the first element is a fill value
time_array = np.array(['1999-12-31','2010-01-01'],
                      dtype="M8[ns]")
mask = np.array([True,False])
time_masked = ma.masked_array(time_array,mask,fill_value=np.datetime64('1999-12-31'))

xr_array = xr.DataArray(data=time_masked,dims=['time'],name="time")
xr_ds = xr.Dataset(dict(time=xr_array))
xr_ds.to_zarr("time_dataset.zarr")

This block of code produces the following warning:

/SOME_PATH/lib/python3.11/site-packages/xarray/coding/times.py:613: RuntimeWarning: invalid value encountered in cast
  int_num = np.asarray(num, dtype=np.int64)

If I then try to read the dataset back out again, I get more errors and the time fill values are set to 2010-01-01:

ds_read = xr.open_dataset("time_dataset.zarr")
print(ds_read["time"])

Read warnings:

/SOME_PATH/lib/python3.11/site-packages/xarray/backends/plugins.py:139: RuntimeWarning: 'netcdf4' fails while guessing
  warnings.warn(f"{engine!r} fails while guessing", RuntimeWarning)
/SOME_PATH/lib/python3.11/site-packages/xarray/coding/times.py:250: RuntimeWarning: invalid value encountered in cast
  flat_num_dates_ns_int = (flat_num_dates * _NS_PER_TIME_DELTA[delta]).astype(
/SOME_PATH/lib/python3.11/site-packages/xarray/coding/times.py:250: RuntimeWarning: invalid value encountered in cast
  flat_num_dates_ns_int = (flat_num_dates * _NS_PER_TIME_DELTA[delta]).astype(

Print out of time array, which no longer has a fill value in the first position:

<xarray.DataArray 'time' (time: 2)>
array(['2010-01-01T00:00:00.000000000', '2010-01-01T00:00:00.000000000'],
      dtype='datetime64[ns]')
Coordinates:
  * time     (time) datetime64[ns] 2010-01-01 2010-01-01

When I look in the actual files under time_dataset.zarr and look at the time_dataset.zarr/.zmetadata file created by xarray, the fill value has been changed to NaN and the datatype is <f8 rather than M8[ns]. It looks as though xarray recognized that these are dates and decided to encode them as NetCDF/CF-1 style "seconds since ". Apparently the fill value did not survive this translation.

{
    "metadata": {
        ".zattrs": {},
        ".zgroup": {
            "zarr_format": 2
        },
        "time/.zarray": {
            "chunks": [
                2
            ],
            "compressor": {
                "blocksize": 0,
                "clevel": 5,
                "cname": "lz4",
                "id": "blosc",
                "shuffle": 1
            },
            "dtype": "<f8",
            "fill_value": "NaN",
            "filters": null,
            "order": "C",
            "shape": [
                2
            ],
            "zarr_format": 2
        },
        "time/.zattrs": {
            "_ARRAY_DIMENSIONS": [
                "time"
            ],
            "calendar": "proleptic_gregorian",
            "units": "days since 2010-01-01 00:00:00"
        }
    },
    "zarr_consolidated_format": 1
}

I know xarray supports datetime64 values (https://docs.xarray.dev/en/stable/user-guide/time-series.html). I know zarr supports datetime64 values (https://zarr.readthedocs.io/en/stable/tutorial.html). So what is going on here?