Cmorize cds uerra reanalysis

[bascrezee]

Open issue:

The recipe_check_obs complains about the Frequency of the data. The frequency is 6h, but the MIP is Lmon. How to make sure that the monthly frequency as read from MIP is overwritten by a user-defined custom frequency? I think @bouweandela recently raised this issue, but I couldn't find it somehow.

[mattiarighi]

I think you mean #237

[bouweandela]

The frequency is 6h, but the MIP is Lmon. How to make sure that the monthly frequency as read from MIP is overwritten by a user-defined custom frequency?

I think you can just set the mip to one of the mips that contain 6 hourly data now.

[mattiarighi]

Can you please add an entry in recipe_check_obs.yml for this dataset.

Do we really need the -REANALYSIS specification in the dataset name?
Isn't CDS-UERRA enough?

[bascrezee]

Do we really need the -REANALYSIS specification in the dataset name?
Isn't CDS-UERRA enough?

We don't necessarily need -REANALYSIS, but we should be more specific then CDS-UERRA. There are two UERRA datasets available on the CDS, one with data on pressure levels and one with data on soil levels. Therefore I suggest renaming it to CDS-UERRA-SOIL-LEVELS, what do you think?

[mattiarighi]

There are two UERRA datasets available on the CDS, one with data on pressure levels and one with data on soil levels. Therefore I suggest renaming it to CDS-UERRA-SOIL-LEVELS, what do you think?

If there are variables available in both datasets (i.e., both on pressure and soil levels), then renaming is necessary. Otherwise I think is redundant.

[bascrezee]

There are two UERRA datasets available on the CDS, one with data on pressure levels and one with data on soil levels. Therefore I suggest renaming it to CDS-UERRA-SOIL-LEVELS, what do you think?

If there are variables available in both datasets (i.e., both on pressure and soil levels), then renaming is necessary. Otherwise I think is redundant.

There are no variables in both datasets. So we can go for CDS-UERRA as a name. I will change it and incorporate your other suggestions.

Something else that I just came across while working with this dataset, is that lon_bnds are missing at the moment. I was surprised that the data checker did not protest about this. At least the area_weighted regridding did complain, that is how I found out. I will adjust the cmorize_obs script accordingly.

[mattiarighi]

In my version level is called soilLayer.

netcdf uerra_volumetric_soil_moisture_1975 {
dimensions:
        time = 1460 ;
        soilLayer = 3 ;
        y = 565 ;
        x = 565 ;
variables:
        float vsw(time, soilLayer, y, x) ;
                vsw:_FillValue = NaNf ;
                vsw:GRIB_paramId = 260199LL ;
                vsw:GRIB_shortName = "vsw" ;
                vsw:GRIB_units = "m**3 m**-3" ;
                vsw:GRIB_name = "Volumetric soil moisture" ;
                vsw:GRIB_cfVarName = "vsw" ;
                vsw:GRIB_dataType = "an" ;
                vsw:GRIB_missingValue = 9999LL ;
                vsw:GRIB_numberOfPoints = 319225LL ;
                vsw:GRIB_typeOfLevel = "soilLayer" ;
                vsw:GRIB_NV = 0LL ;
                vsw:GRIB_stepUnits = 1LL ;
                vsw:GRIB_stepType = "instant" ;
                vsw:GRIB_gridType = "lambert" ;
                vsw:GRIB_gridDefinitionDescription = "Lambert conformal " ;
                vsw:GRIB_LaDInDegrees = 48. ;
                vsw:GRIB_LoVInDegrees = 8. ;
                vsw:GRIB_iScansNegatively = 0LL ;
                vsw:GRIB_jPointsAreConsecutive = 0LL ;
                vsw:GRIB_jScansPositively = 1LL ;
                vsw:GRIB_latitudeOfFirstGridPointInDegrees = 17.612 ;
                vsw:GRIB_latitudeOfSouthernPoleInDegrees = 0. ;
                vsw:GRIB_longitudeOfFirstGridPointInDegrees = 341.68 ;
                vsw:GRIB_longitudeOfSouthernPoleInDegrees = 0. ;
                vsw:GRIB_DyInMetres = 11000. ;
                vsw:GRIB_DxInMetres = 11000. ;
                vsw:GRIB_Latin2InDegrees = 48. ;
                vsw:GRIB_Latin1InDegrees = 48. ;
                vsw:GRIB_Ny = 565LL ;
                vsw:GRIB_Nx = 565LL ;
                vsw:long_name = "Volumetric soil moisture" ;
                vsw:units = "m**3 m**-3" ;
                vsw:coordinates = "valid_time step latitude longitude" ;
        int64 time(time) ;
                time:long_name = "initial time of forecast" ;
                time:standard_name = "forecast_reference_time" ;
                time:units = "seconds since 1970-01-01T00:00:00+00:00" ;
                time:calendar = "proleptic_gregorian" ;
        double step ;
                step:_FillValue = NaN ;
                step:long_name = "time since forecast_reference_time" ;
                step:standard_name = "forecast_period" ;
                step:units = "hours" ;
        int64 soilLayer(soilLayer) ;
                soilLayer:long_name = "original GRIB coordinate for key: level(soilLayer)" ;
                soilLayer:units = "1" ;
        double latitude(y, x) ;
                latitude:_FillValue = NaN ;
                latitude:units = "degrees_north" ;
                latitude:standard_name = "latitude" ;
                latitude:long_name = "latitude" ;
        double longitude(y, x) ;
                longitude:_FillValue = NaN ;
                longitude:units = "degrees_east" ;
                longitude:standard_name = "longitude" ;
                longitude:long_name = "longitude" ;
        double valid_time(time) ;
                valid_time:_FillValue = NaN ;
                valid_time:standard_name = "time" ;
                valid_time:long_name = "time" ;
                valid_time:units = "seconds since 1970-01-01T00:00:00+00:00" ;
                valid_time:calendar = "proleptic_gregorian" ;

// global attributes:
                :GRIB_edition = 2LL ;
                :GRIB_centre = "eswi" ;
                :GRIB_centreDescription = "Norrkoping" ;
                :GRIB_subCentre = 0LL ;
                :Conventions = "CF-1.7" ;
                :institution = "Norrkoping" ;
                :history = "2019-11-05T01:19:47 GRIB to CDM+CF via cfgrib-0.9.5.5/ecCodes-2.14.0 with {\"source\": \"/cache/tmp/34d59c1c-240c-491e-a8c8-7bcdefafe855-adaptor.uerra_mars.external-1572915943.4409108-4005-6-tmp.grib\", \"filter_by_keys\": {}, \"encode_cf\": [\"parameter\", \"time\", \"geography\", \"vertical\"]}" ;
}

[bascrezee]

Ok, since you have the latest version of the dataset, we should go with that. Can you change level to soilLayer and see if it works ?

[mattiarighi]

It goes through now, but it stops after the regridding since it cannot find the files to concatenate. I guess the problem is at these lines:

ESMValTool/esmvaltool/cmorizers/obs/cmorize_obs_cds_uerra.py

Lines 159 to 160 in aee3056

	filelist = glob.glob(os.path.join(
	cfg['work_dir'], 'uerra_regridded_{0}??.nc'.format(year)))

The files in the work_dir are named differently:

cds-uerra_regridded_volumetric_soil_moisture_????.nc

[bascrezee]

Yes, that should be changed as well, I just changed it. Of course it would be better to have the filenames inferred directly from the cmor_config, such that it will work when adding new variables. No time to implement this at the moment though.

[mattiarighi]

I understand, but this solution is quite error prone, since whatever variable you specify you always get the volumetric soil moisture to be concatenated.

[bascrezee]

Ok, I can change it, but it will take a while, since I have to download the data again given that it has changed.

[mattiarighi]

I can send them to you via the DKRZ cloud if you wish (just a couple of years for testing).
It would be much faster.

[bascrezee]

I can send them to you via the DKRZ cloud if you wish (just a couple of years for testing).
It would be much faster.

Yes, that sounds good :) CDS is offline now...

[mattiarighi]

Here you go (valid for 24 hours).

[bascrezee]

I updated the script, ready for testing again !

[mattiarighi]

Can you solve the remaining codacy issue?

[mattiarighi]

Tested successfully!