XMIP Initial Reprocessing Does Not Work as Expected

[nishsilva]

Hi, I am new to xMIP and have been trying to learn the basic operations of processing CMIP6 data using the xMIP tutorial. As a start, I used the initial scripts provided on the tutorial page to label dimensions consistently. Once I run the code the dimensions do not change but show the original dimensions. Can someone provide an insight on what is happening here?

I am posting the code and outputs below

`#!/usr/bin/env python3
#-- coding: utf-8 --
"""
Created on Sat Apr 8 10:20:54 2023

@author: netige
"""

import matplotlib.pyplot as plt
import intake
import dask
%matplotlib inline

url = "https://storage.googleapis.com/cmip6/pangeo-cmip6.json"
col = intake.open_esm_datastore(url)

#load a few models to illustrate the problem
query = dict(experiment_id=['piControl'], table_id='Oyr',
variable_id='o2', grid_label=['gn', 'gr'],
source_id=['IPSL-CM6A-LR', 'CanESM5', 'GFDL-ESM4']
)
cat = col.search(**query)

cat.df['source_id'].unique()
z_kwargs = {'consolidated': True, 'decode_times':False}
with dask.config.set(**{'array.slicing.split_large_chunks': True}):
dset_dict = cat.to_dataset_dict(zarr_kwargs=z_kwargs)#

#show coordinates

for k, ds in dset_dict.items():
print(k)
print(list(ds.dims))

from xmip.preprocessing import rename_cmip6

#load a few models to illustrate the problem
cat = col.search(**query)
cat.df['source_id'].unique()

#pass the preprocessing directly
with dask.config.set(**{'array.slicing.split_large_chunks': True}):
dset_dict_renamed = cat.to_dataset_dict(zarr_kwargs=z_kwargs, preprocess=rename_cmip6)

for k, ds in dset_dict_renamed.items():
print(k)
print(list(ds.dims))`

Output
runfile('/project/pdpanalysis/CMIP6_data_processing/CMIP6_Data_Processing/Nish_040823_Pilot_Data_Processing.ipy', wdir='/project/pdpanalysis/CMIP6_data_processing/CMIP6_Data_Processing')

QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/scratch/7058609.1.geo-int/runtime-netige'

--> The keys in the returned dictionary of datasets are constructed as follows:
'activity_id.institution_id.source_id.experiment_id.table_id.grid_label'
|----------------------------------------| 0.00% [0/3 00:00<?]/project/pdpanalysis/CMIP6_data_processing/CMIP6_Data_Processing/Nish_040823_Pilot_Data_Processing.ipy:36: DeprecationWarning: cdf_kwargs and zarr_kwargs are deprecated and will be removed in a future version. Please use xarray_open_kwargs instead.
dset_dict = cat.to_dataset_dict(zarr_kwargs=z_kwargs)#
|????????????????????????????????????????| 100.00% [3/3 00:11<00:00]CMIP.IPSL.IPSL-CM6A-LR.piControl.Oyr.gn
['y', 'x', 'nvertex', 'member_id', 'dcpp_init_year', 'time', 'olevel', 'axis_nbounds']
CMIP.NOAA-GFDL.GFDL-ESM4.piControl.Oyr.gr
['lat', 'bnds', 'lev', 'lon', 'member_id', 'dcpp_init_year', 'time']
CMIP.CCCma.CanESM5.piControl.Oyr.gn
['i', 'j', 'lev', 'time', 'dcpp_init_year', 'member_id', 'bnds', 'vertices']

--> The keys in the returned dictionary of datasets are constructed as follows:
'activity_id.institution_id.source_id.experiment_id.table_id.grid_label'
/project/pdpanalysis/CMIP6_data_processing/CMIP6_Data_Processing/Nish_040823_Pilot_Data_Processing.ipy:52: DeprecationWarning: cdf_kwargs and zarr_kwargs are deprecated and will be removed in a future version. Please use xarray_open_kwargs instead.
dset_dict_renamed = cat.to_dataset_dict(zarr_kwargs=z_kwargs, preprocess=rename_cmip6)
CMIP.IPSL.IPSL-CM6A-LR.piControl.Oyr.gn???| 100.00% [3/3 00:09<00:00]
['y', 'x', 'time', 'member_id', 'dcpp_init_year', 'lev', 'bnds', 'vertex']
CMIP.NOAA-GFDL.GFDL-ESM4.piControl.Oyr.gr
['y', 'lev', 'x', 'time', 'member_id', 'dcpp_init_year', 'bnds']
CMIP.CCCma.CanESM5.piControl.Oyr.gn
['lev', 'time', 'dcpp_init_year', 'member_id', 'y', 'x', 'vertex', 'bnds']