Improvements to existing functionality

[andersy005]

• Update to_xarray() method to allow users to pass keyword arguments to this method
• Refactor cmip.py in preparation for cmip6 integration as proposed in CMIP6 version of cmip.py #43

[andersy005]

@matt-long, I've experienced difficulties while trying to merge/concat CMIP data from different institutions. It appears that there are discrepancies among coordinates of different models' outputs. For queries in which there's data from more than one institution, xarray is unable to align the coordinates.

For instance, I saw one variable in which the coordinates in one model are time, lat, lon, bnds and time, rlat, rlon, bnds in another model from a different institution. In addition to

https://github.com/NCAR/intake-esm/blob/67fb828a202a7c7fc15953b63436075554e3ae1c/intake_esm/cmip.py#L240-L244

should we add a check that forbids merging datasets from different institutions? Let me know what would be the right approach.

[matt-long]

@andersy005, I would not expect models from different institutions to be concatenate-able; the different models do indeed have different coordinates. I think we should add a check that ensures prevent this. Another approach would be to return a data structure that includes the datasets from the different models. A dictionary with the institution ID as the key, for instance, could work. This would entail an outer loop, probably best implemented as a method that calls this _open_dataset method.

[andersy005]

@matt-long, this is ready for another look

[matt-long]

Does the code above belong in a _validate_concat_open_dataset method?

[andersy005]

Could you expand on this?

[matt-long]

I think it's probably fine for now. I was wondering whether you could write a general method for validation. Not sure of the API....

def _validate_concat(check_max_instance=None):

  if check_max_instance is not None:
    for fld, max_inst in check_max_instance.items():
        fld_list = self.query_results[fld].unique()
        if len(fld_list) > max_inst:
            raise ValueError(f'message about {fld}')

[andersy005]

I see! Now that we are handling an addition case of datasets that cannot be concatenated by returning a dictionary of dsets in:

https://github.com/NCAR/intake-esm/blob/eb8fdafa0d7b22770bffde968bf36453d9cd5b9a/intake_esm/cmip.py#L254-L264

can you think of other cases that wouldn't be handled by the above code?

[matt-long]

I probably need to work thru some use cases to really develop my intuition here.

[andersy005]

Sounds good! Let me know when you get to play with this.

[matt-long]

Should we have an _open_dataset_groups method that calls the _open_dataset method? Would this allow us to reuse more code between cesm.py and cmip.py?

[andersy005]

I see the advantage of generalizing this. What would be the equivalent of _open_dataset_groups() for cesm data, in other words, what is the equivalent of CMIP institutions in CESM?

[matt-long]

We could think about this as moving toward any arbitrary collection of datasets that should not be concatenated along a dimension. Observations, integrations from different model version or resolutions, for instance. We might want to have a model general dataset_id or something versus insisting on "institution."

[andersy005]

I will take a stab at generalizing it (I am afraid that it may not be a trivial task though)