cmorizer for GRACE data #1694

bascrezee · 2020-07-02T09:32:46Z

Test the CMORized data using recipes/example/recipe_check_obs.yml, to make sure the CMOR checks pass without errors
Add the new dataset to the table in the documentation
Tag @mattiarighi in this pull request, so that the new dataset can be added to the OBS data pool at DKRZ and synchronized with CEDA-Jasmin
Add additional references as listed here: https://grace.jpl.nasa.gov/data/get-data/jpl_global_mascons/. @mattiarighi is it possible to add multiple references to one dataset?

Closes #1693

bascrezee · 2020-07-02T09:37:49Z

@mattiarighi Can you tell me if GRACE classifies as Tier2 or Tier3 data?

esmvaltool/cmorizers/obs/cmorize_obs_grace.py

bouweandela · 2020-07-08T13:49:16Z

@bascrezee Could you please use the isort tool to sort the imports?

mattiarighi · 2020-07-09T09:29:53Z

Can you tell me if GRACE classifies as Tier2 or Tier3 data?

This is Tier 3, since an EarthData account is needed.

bascrezee · 2020-07-15T07:47:54Z

@mattiarighi Can you tell me how to deal with the multiple references for this dataset?

mattiarighi · 2020-07-15T10:38:52Z

@mattiarighi Can you tell me how to deal with the multiple references for this dataset?

I think this is not supported. Just give the most recent one.

doc/sphinx/source/input.rst

esmvaltool/references/grace.bibtex

mattiarighi · 2020-10-16T14:01:55Z

Please add a testing diagnostic in recipe_check_obs.yml.

Co-authored-by: Mattia Righi <mattia.righi@dlr.de>

bascrezee · 2020-10-19T11:53:52Z

Please add a testing diagnostic in recipe_check_obs.yml.

The diagnostics related to this dataset, unfortunately live in the private branch. Do you have a candidate diagnostic for testing? Just a map plot would be perfect. Or do you mean simply an entry with 'null' script as for the other datasets? Of course I can do that.

bascrezee · 2020-10-26T11:01:41Z

@mattiarighi see my questions above. Thanks!

mattiarighi · 2020-10-27T07:35:56Z

The testing diagnostics in recipe_check_obs.yml are just dummy diagnostics, the script entry is set to null.
That recipe is only used for checking the data reading and the cmor compliancy.

hb326 · 2020-11-25T16:51:01Z

@jvegasbsc, @remi-kazeroni started to work on this. Should we also just wait for #1657 to be merged, or should we keep working on it to get it merged?

jvegreg · 2020-11-26T14:19:11Z

You can keep working on this, but it's better to wait for #1657 to merge. Conversion is straightforward, so it will only take me a few minutes to adapt it

remi-kazeroni

The cmorizer is not working with the latest (and only available) version of the data. The reason is the time range of the dataset has changed and does not correspond to was is hardcoded. It would be more appropriate to get those from the csv table.
Less important: the names of the datafile and the aux datafiles are harcoded whereas these could be get from the config file (GRACE.yml)

I'm happy to make the changes if needed

remi-kazeroni · 2020-11-26T14:05:00Z

esmvaltool/cmorizers/obs/cmor_config/GRACE.yml

+    'CLM4.SCALE_FACTOR.JPL.MSCNv02CRI.nc'
+
+grace_table:
+  'https://podaac-tools.jpl.nasa.gov/drive/files/allData/tellus/L3/docs/GRACE_GRACE-FO_Months_RL06.csv'


Wouldn't it be better to use the name of the csv file instead of its url? Otherwise, the csv file name is hardcoded in the cmorizer

remi-kazeroni · 2020-11-26T14:07:59Z

esmvaltool/cmorizers/obs/cmor_config/GRACE.yml

+    mip: Lmon
+    raw: lwe_thickness
+    file: 'GRCTellus.JPL.200204_202003.GLO.RL06M.MSCNv02CRI.nc'
+


The file name is not correct anymore as the dataset was updated with 2020 data. The previous file is not available anymore. The correct name is: GRCTellus.JPL.200204_202009.GLO.RL06M.MSCNv02CRI.nc

remi-kazeroni · 2020-11-26T14:28:18Z

esmvaltool/cmorizers/obs/cmorize_obs_grace.py

+
+    months_table_file = os.path.join(cfg['in_dir'],
+                                     'GRACE_GRACE-FO_Months_RL06.csv')
+    grace_months_table = pd.read_csv(months_table_file)


Perhaps better not to hardcode the filenames but use the config file: GRACE.yml?

remi-kazeroni · 2020-11-26T14:30:48Z

esmvaltool/cmorizers/obs/cmorize_obs_grace.py

+    gain_file = os.path.join(cfg['rawobsdir'], 'Tier3/GRACE/',
+                             'CLM4.SCALE_FACTOR.JPL.MSCNv02CRI.nc')
+    lsm_file = os.path.join(cfg['rawobsdir'], 'Tier3/GRACE/',
+                            'LAND_MASK.CRI.nc')


Here as well the filename are not read from the config file.

remi-kazeroni · 2020-11-26T14:32:38Z

esmvaltool/cmorizers/obs/cmorize_obs_grace.py

+    time_axis = []
+    start_date = datetime(2002, 4, 15)
+    end_date = datetime(2019, 12, 15)
+    while start_date <= end_date:


The start and end dates are hardcoded. So the cmorize fails with another version of the dataset. It would be usefull to get the time bounds from the csv file. That would work with newer data

remi-kazeroni · 2020-11-26T14:52:00Z

esmvaltool/cmorizers/obs/cmor_config/GRACE.yml

+  version: '1'
+  modeling_realm: satellite
+  source: ''
+  reference: 'grace'


The source is missing

remi-kazeroni · 2020-11-27T16:16:24Z

I have just made the changes ot fix the cmorizer based on the review I made. It would be great if someone could review this PR.

bouweandela · 2020-12-07T12:54:33Z

The tests are failing on CircleCI, but probably not because of changes in this pull request. The easiest way to solve this is probably to merge the master branch into this branch. It's always nice to make sure all tests are green before requesting a human review.

remi-kazeroni · 2020-12-08T11:00:15Z

The tests are failing on CircleCI, but probably not because of changes in this pull request. The easiest way to solve this is probably to merge the master branch into this branch. It's always nice to make sure all tests are green before requesting a human review.

Thanks, this has fixed the failing tests. Codacy issue is also fixed, it's all green now.

bascrezee

Thanks @remi-kazeroni for picking this up! The changes that you made look good to me.

remi-kazeroni · 2020-12-08T13:47:10Z

Thanks @remi-kazeroni for picking this up! The changes that you made look good to me.

You're welcome @bascrezee! It was just a few fixes to account for the latest GRACE data release.

remi-kazeroni · 2020-12-11T13:02:39Z

@axel-lauer I think this is ready to merge

cmorizer for GRACE data

6a9e25f

added GRACE to doc

4977b76

bascrezee marked this pull request as ready for review July 2, 2020 09:50

bascrezee requested a review from mattiarighi July 2, 2020 09:50

bascrezee mentioned this pull request Jul 2, 2020

Add custom variable lweGrace ESMValGroup/ESMValCore#692

Merged

bascrezee added 2 commits July 2, 2020 13:47

Moved URL to config file

f002a01

change calendar from proleptic_gregorian to gregorian

20d3c87

bouweandela reviewed Jul 8, 2020

View reviewed changes

esmvaltool/cmorizers/obs/cmorize_obs_grace.py Outdated Show resolved Hide resolved

ran isort

72fd5d7

mattiarighi added the observations label Jul 9, 2020

mattiarighi and others added 3 commits July 13, 2020 11:24

Fix keyword argument

14bea84

Merge branch 'master' into cmorize-grace

e3db53d

Merge branch 'fix_cmorizer' into cmorize-grace

6bb4be6

bascrezee added 3 commits July 15, 2020 10:02

Read months table from disk

35b947a

ask user to inspect months table

dcfdefd

removed unused import

b391621

bouweandela mentioned this pull request Sep 24, 2020

Topics for ESMValTool workshop in November #1817

Closed

mattiarighi reviewed Oct 16, 2020

View reviewed changes

doc/sphinx/source/input.rst Outdated Show resolved Hide resolved

mattiarighi reviewed Oct 16, 2020

View reviewed changes

esmvaltool/references/grace.bibtex Outdated Show resolved Hide resolved

bascrezee and others added 2 commits October 19, 2020 10:50

Update doc/sphinx/source/input.rst

32f501e

Co-authored-by: Mattia Righi <mattia.righi@dlr.de>

Update esmvaltool/references/grace.bibtex

a710c7e

Co-authored-by: Mattia Righi <mattia.righi@dlr.de>

hb326 added this to ESMValTool - needs review in November 2020 Nov 24, 2020

remi-kazeroni self-requested a review November 24, 2020 15:35

remi-kazeroni moved this from ESMValTool - needs review to ESMValTool - in review in November 2020 Nov 24, 2020

remi-kazeroni requested changes Nov 26, 2020

View reviewed changes

remi-kazeroni moved this from ESMValTool - in review to ESMValTool - needs review in November 2020 Nov 26, 2020

remi-kazeroni moved this from Needs Review to In Progress in November 2020 Nov 27, 2020

remi-kazeroni added 4 commits November 27, 2020 15:36

config file updated

41b6ab9

time bounds read from csv

766b627

filenames from config file

4da63a6

check csv file

f3c4f72

remi-kazeroni moved this from In Progress to Needs Review in November 2020 Nov 30, 2020

remi-kazeroni added 4 commits December 7, 2020 17:09

Merge branch 'master' into cmorize-grace

9fec826

fix codacy issue

4d621f5

fix codacy issue

b0017de

fix codacy issue

fd0a465

bascrezee commented Dec 8, 2020

View reviewed changes

remi-kazeroni approved these changes Dec 11, 2020

View reviewed changes

axel-lauer merged commit a1d02f0 into master Dec 11, 2020
6 checks passed

axel-lauer deleted the cmorize-grace branch December 11, 2020 13:41

November 2020 automation moved this from Needs Review to Merged Dec 11, 2020

This was referenced Dec 18, 2020

add GRACE data to recipe_check_obs #1962

Closed

Add GRACE to the recipe check_obs #1963

Merged

remi-kazeroni mentioned this pull request Sep 17, 2021

Streamline observations download #1657

Merged

9 tasks

cmorizer for GRACE data #1694

cmorizer for GRACE data #1694

bascrezee commented Jul 2, 2020 •

edited

bascrezee commented Jul 2, 2020

bouweandela commented Jul 8, 2020

mattiarighi commented Jul 9, 2020

bascrezee commented Jul 15, 2020

mattiarighi commented Jul 15, 2020

mattiarighi commented Oct 16, 2020

bascrezee commented Oct 19, 2020 •

edited

bascrezee commented Oct 26, 2020

mattiarighi commented Oct 27, 2020

hb326 commented Nov 25, 2020

jvegreg commented Nov 26, 2020

remi-kazeroni left a comment

remi-kazeroni Nov 26, 2020

remi-kazeroni Nov 26, 2020

remi-kazeroni Nov 26, 2020

remi-kazeroni Nov 26, 2020

remi-kazeroni Nov 26, 2020

remi-kazeroni Nov 26, 2020

remi-kazeroni commented Nov 27, 2020

bouweandela commented Dec 7, 2020

remi-kazeroni commented Dec 8, 2020

bascrezee left a comment

remi-kazeroni commented Dec 8, 2020

remi-kazeroni commented Dec 11, 2020

cmorizer for GRACE data #1694

cmorizer for GRACE data #1694

Conversation

bascrezee commented Jul 2, 2020 • edited

bascrezee commented Jul 2, 2020

bouweandela commented Jul 8, 2020

mattiarighi commented Jul 9, 2020

bascrezee commented Jul 15, 2020

mattiarighi commented Jul 15, 2020

mattiarighi commented Oct 16, 2020

bascrezee commented Oct 19, 2020 • edited

bascrezee commented Oct 26, 2020

mattiarighi commented Oct 27, 2020

hb326 commented Nov 25, 2020

jvegreg commented Nov 26, 2020

remi-kazeroni left a comment

Choose a reason for hiding this comment

remi-kazeroni Nov 26, 2020

Choose a reason for hiding this comment

remi-kazeroni Nov 26, 2020

Choose a reason for hiding this comment

remi-kazeroni Nov 26, 2020

Choose a reason for hiding this comment

remi-kazeroni Nov 26, 2020

Choose a reason for hiding this comment

remi-kazeroni Nov 26, 2020

Choose a reason for hiding this comment

remi-kazeroni Nov 26, 2020

Choose a reason for hiding this comment

remi-kazeroni commented Nov 27, 2020

bouweandela commented Dec 7, 2020

remi-kazeroni commented Dec 8, 2020

bascrezee left a comment

Choose a reason for hiding this comment

remi-kazeroni commented Dec 8, 2020

remi-kazeroni commented Dec 11, 2020

bascrezee commented Jul 2, 2020 •

edited

bascrezee commented Oct 19, 2020 •

edited