Data checker

[malerany]

Implement checking of the provided simulation dataset (available variables, units, ...) within the check_data function defined in src/pyhanami/utils/data.py.

Assignees: @malerany

Migrated from GitLab: https://gitlab.earth.bsc.es/ces/hanami/pyhanami/-/issues/2

[malerany]

I implemented a first version of the check_data function. It checks the following for the provided xarray.Dataset:

• That the coordinates time, realization, lat and lon are present
• That the lat and lon coordinates are strictly increasing or decreasing
• That all variables in the dataset are listed in the config file
• That all variables have units, and converts them if they differ from the expected units as defined in the config file

It is available in the develop branch.

[malerany]

After discussion with @kellekai today, I modified the check_data function to raise an error and stop running whenever there is an error in the format of the provided xarray.Dataset, forcing the user to correct the simulated data in the catalogue instead of modifying it inside the check_data function.

[malerany]

I was thinking about adding a check for the format of the time values (e.g., what calendar to use), but I am not sure yet what formats are compatible with all our functions or whether we want to ask for a specific format.

[malerany]

Since @kellekai encountered some issues with the time coordinate format when loading observations, I added a check for it. It is included in the develop branch both for simulations and observations.

It ensures that the dates are stored as numpy.datetime64('ns'), which is the default format that xarray uses when decoding datetime data for standard calendars. If this is not the case, it converts the dates to this format and raises a warning message informing the user that the time values have been converted and encouraging them to use numpy.datetime64(‘ns’) in the future. The same for the observations.

Ultimately, we will probably request the user to provide the data in the right format, but I added this now so we can use the package with the observation datasets that we have now.

[malerany]

In GitLab by @kellekai on Aug 29, 2025, 13:04 UTC:

I think this is straight forward and should be fine, as it probably will raise an error if the conversion is not possible, forcing the user to correct it. In contrast to other sensitive metadata such as units. So, in my opinion, we can keep the automatic conversion.

[malerany]

I noticed a case that was not being taken into account. I encountered a set of monthly observations that were in the numpy.datetime64(‘ns’) format but with an hour different than midnight, i.e., not in the format YYYY-MM-DDT00:00:00 but YYYY-MM-DDT12:00:00. This led to problems when comparing with the simulations. Hence, I modified the code so that whenever something like this happens for daily or coarser data, the datetime is set by default to midnight, and a warning explaining the modification is issued to the user.

[malerany]

I was also wondering if we should check for NaN values in the data, and how we should handle those if so.

[malerany]

I had a meeting with Iker today and he explained to me the approach used in DestinE for the data checker. They mainly test for 6 features:

• Standard compliance: check metadata of the file (e.g., short name or units available as attributes).
• Spatial completeness: check for NaN values (they check that no NaN values are present for atmospheric variables and that they are present for oceanic variables, without checking in which locations they are).
• Spatial consistency: check that the grid is the expected one.
• Temporal completeness: check that there are numeric values for all timesteps in the given period.
• Temporal consistency: mainly check that the time period is the expected one for a given file (for them, the time is included in the file name).
• Physical plausibility: check that each variable has the correct units and its values are within an established range that is physically reasonable.

The scripts they use for each feature are in here: gsv_interface/gsv/dqc (and here are equivalent checks used in esarchive: esarchive-checker).

Some of these features are already in utils/data.py/data_check, like the standard compliance check. I am planning on also implementing checks for the following:

• NaN values
• All timesteps present
• Physical plausibility

Then, we would be missing the spatial consistency one, but since we do not require a specific grid, I do not think it applies to our case. The same for the temporal consistency.

[malerany]

In GitLab by @kellekai on Sep 19, 2025, 06:43 UTC:

Hi, perfect Marta!

I have a few comments/doubts:

• Spatial consistency: It is true that we do not require a grid, this brings me to a different need in our package, the generic regridding. I think, at some point, we need to get into this difficult topic. There are all kinds of grids, including unstructured grids. If at some point we can do the regridding for any kind of grid, we will need to impose the requirement of corner points or area being included in the files.
• Physical plausibility: I was always wondering about that, where are the reference values coming from? I am quite sure that Iker and PA have taken them from a reliable source, and it would be great to document this somewhere in the package. If there is no information, perhaps we can check if such information is available, provided by the WMO.
• also for the future; I have the feeling we can get some information from the NaNs in the ocean fields. For instance, the land-sea mask. We could store it for the data we look at and let the user know if the masks are different for observations and sim data (which is probably most of the time the case).

[malerany]

Hi @kellekai,

I created a new DataChecker class in utils/data_checker.py to perform all the checks that we mentioned more efficiently. I included:

• Standard compliance: check names and format of coordinates (time, lat, lon), and variables’ attributes (long_name and units).
• Spatial completeness: check that there are no NaN values for atmosphere variables, and that there is at least one (but not all) NaN value for the ocean variables. (I added a land_mask placeholder to add the check that you suggested in the future)
• Spatial consistency: I just created a placeholder function to be implemented when we impose requirements for the grids, as you mentioned above.
• Temporal completeness: check that all timesteps are present between the minimum and maximum time values.
• Physical plausibility: check that all variables lie within the established ranges.

Regarding the last one, I asked Iker and he believes that the document they used was not publicly available. He suggested citing the GSV repository as the source for our values, as it is included in ClimateDT. However, we checked and it looks like the DesintE GitHub is not publicly available yet. Do you perhaps know if this is planned to happen in the near future?

[malerany]

Besides, I moved all the metadata associated with each variable into a variables.yaml file for convenience.

[malerany]

Hi @kellekai, Iker just let me know that the GSV repository has just been made public. Perhaps we could add it as our source for the ‘physical plausibility’ checks for now while we look for an external source?

[malerany]

In GitLab by @kellekai on Oct 2, 2025, 11:06 UTC:

yes, lets do that!