Add a function to enrich STM using data from another dataset

[SarahAlidoost]

closes #63

This PR implements two functions:

• stm.enrich_from_dataset: for both points and raster datasets
• utils.crop: a util function for cropping a dataset, either points or raster

see #69 on why I changed _io.py in this PR.

[rogerkuou]

Hi Sarah, thanks for the nice implementation! It works well in general.

I have two comments regarding adding more checks on the incoming data. See above.

These comments come from a test I made with KNMI data. I also attached the notebook and the small example dataset here.

example_test.zip

[rogerkuou]

I did a test using sel to get nearest time location, with datetime64[ns] format. This works under two conditions:

1. For time dimension, in a specific combination: method=nearest + datetime64[ns] format, Xarray requires the time coordinates of ds to be monotonic increasing or decreasing.
2. For lat and lon dimension, the duplicated values should not exists in the corresponding coordinates.

Can we add check in the beginning of enrich_from_dataset to enforce this?

An example check for time:

# This should be True
np.all(np.diff(ds['time'].values) > 0) or np.all(np.diff(ds['time'].values) < 0) 

An example check for lat and lon:

# Following should be True
np.unique(ds['lat'].values).shape == ds['lat'].values.shape
np.unique(ds['lon'].values).shape == ds['lon'].values.shape

[SarahAlidoost]

good point. It seems that xarray does not enforce cf conventions. I added two util functions to do the checks and call them in stem _enrich_from_points_block here.

[SarahAlidoost]

I noticed that with KDTree, these two conditions are not needed. So I removed the checks from _enrich_from_points_block.

[SarahAlidoost]

I added some tests for these conditions

[SarahAlidoost]

@rogerkuou to check if the points are unique, the test np.unique(ds['lat'].values).shape == ds['lat'].values.shape is not enough because it only checks the duplicates in one dimension here lat. However, for example, points can be located on one line.

Instead, we need a test if there are cases where (lat, lon, time) are duplicated. Functions like xarray.Dataset.drop_duplicates and pandas.DataFrame.duplicated can be used to write a test. But these functions only work on dim and not coords. In our cases, lat and lon are coords and space is the dim. So we might need to use unstack which leads to memory problems.

As discussed, scipy KDTree works if coordinates (lat, lon, time) are duplicated and the values of variables e.g. temperature are the same too, I added a test for this. If the values of variables are not the same for duplicated coordinates, MacOs and linux behave differently to pick up a value related to the nearest neighbor. For this extreme case, I suggest creating a new issue and fixing it in another PR.

[SarahAlidoost]

Hi Sarah, thanks for the nice implementation! It works well in general.

I have two comments regarding adding more checks on the incoming data. See above.

These comments come from a test I made with KNMI data. I also attached the notebook and the small example dataset here.

example_test.zip

thanks for the review. It is ready for another look.

[rogerkuou]

Hi @SarahAlidoost, thanks for the nice implementation!

I doced the follow-up issues in #75 and #76

I will merge it now.