Skip to content

cmorizer for GRACE data #1694

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 21 commits into from
Dec 11, 2020
Merged

cmorizer for GRACE data #1694

merged 21 commits into from
Dec 11, 2020

Conversation

bascrezee
Copy link
Contributor

@bascrezee bascrezee commented Jul 2, 2020

  • Test the CMORized data using recipes/example/recipe_check_obs.yml, to make sure the CMOR checks pass without errors
  • Add the new dataset to the table in the documentation
  • Tag @mattiarighi in this pull request, so that the new dataset can be added to the OBS data pool at DKRZ and synchronized with CEDA-Jasmin
  • Add additional references as listed here: https://grace.jpl.nasa.gov/data/get-data/jpl_global_mascons/. @mattiarighi is it possible to add multiple references to one dataset?

Closes #1693

@bascrezee
Copy link
Contributor Author

@mattiarighi Can you tell me if GRACE classifies as Tier2 or Tier3 data?

@bascrezee bascrezee marked this pull request as ready for review July 2, 2020 09:50
@bascrezee bascrezee requested a review from mattiarighi July 2, 2020 09:50
@bouweandela
Copy link
Member

@bascrezee Could you please use the isort tool to sort the imports?

@mattiarighi
Copy link
Contributor

Can you tell me if GRACE classifies as Tier2 or Tier3 data?

This is Tier 3, since an EarthData account is needed.

@bascrezee
Copy link
Contributor Author

@mattiarighi Can you tell me how to deal with the multiple references for this dataset?

@mattiarighi
Copy link
Contributor

@mattiarighi Can you tell me how to deal with the multiple references for this dataset?

I think this is not supported. Just give the most recent one.

@mattiarighi
Copy link
Contributor

Please add a testing diagnostic in recipe_check_obs.yml.

bascrezee and others added 2 commits October 19, 2020 10:50
Co-authored-by: Mattia Righi <mattia.righi@dlr.de>
Co-authored-by: Mattia Righi <mattia.righi@dlr.de>
@bascrezee
Copy link
Contributor Author

bascrezee commented Oct 19, 2020

Please add a testing diagnostic in recipe_check_obs.yml.

The diagnostics related to this dataset, unfortunately live in the private branch. Do you have a candidate diagnostic for testing? Just a map plot would be perfect. Or do you mean simply an entry with 'null' script as for the other datasets? Of course I can do that.

@bascrezee
Copy link
Contributor Author

@mattiarighi see my questions above. Thanks!

@mattiarighi
Copy link
Contributor

The testing diagnostics in recipe_check_obs.yml are just dummy diagnostics, the script entry is set to null.
That recipe is only used for checking the data reading and the cmor compliancy.

@remi-kazeroni remi-kazeroni self-requested a review November 24, 2020 15:35
@hb326
Copy link
Contributor

hb326 commented Nov 25, 2020

@jvegasbsc, @remi-kazeroni started to work on this. Should we also just wait for #1657 to be merged, or should we keep working on it to get it merged?

@jvegreg
Copy link
Contributor

jvegreg commented Nov 26, 2020

You can keep working on this, but it's better to wait for #1657 to merge. Conversion is straightforward, so it will only take me a few minutes to adapt it

Copy link
Contributor

@remi-kazeroni remi-kazeroni left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The cmorizer is not working with the latest (and only available) version of the data. The reason is the time range of the dataset has changed and does not correspond to was is hardcoded. It would be more appropriate to get those from the csv table.
Less important: the names of the datafile and the aux datafiles are harcoded whereas these could be get from the config file (GRACE.yml)

I'm happy to make the changes if needed

'CLM4.SCALE_FACTOR.JPL.MSCNv02CRI.nc'

grace_table:
'https://podaac-tools.jpl.nasa.gov/drive/files/allData/tellus/L3/docs/GRACE_GRACE-FO_Months_RL06.csv'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't it be better to use the name of the csv file instead of its url? Otherwise, the csv file name is hardcoded in the cmorizer

mip: Lmon
raw: lwe_thickness
file: 'GRCTellus.JPL.200204_202003.GLO.RL06M.MSCNv02CRI.nc'

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The file name is not correct anymore as the dataset was updated with 2020 data. The previous file is not available anymore. The correct name is: GRCTellus.JPL.200204_202009.GLO.RL06M.MSCNv02CRI.nc


months_table_file = os.path.join(cfg['in_dir'],
'GRACE_GRACE-FO_Months_RL06.csv')
grace_months_table = pd.read_csv(months_table_file)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps better not to hardcode the filenames but use the config file: GRACE.yml?

gain_file = os.path.join(cfg['rawobsdir'], 'Tier3/GRACE/',
'CLM4.SCALE_FACTOR.JPL.MSCNv02CRI.nc')
lsm_file = os.path.join(cfg['rawobsdir'], 'Tier3/GRACE/',
'LAND_MASK.CRI.nc')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here as well the filename are not read from the config file.

time_axis = []
start_date = datetime(2002, 4, 15)
end_date = datetime(2019, 12, 15)
while start_date <= end_date:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The start and end dates are hardcoded. So the cmorize fails with another version of the dataset. It would be usefull to get the time bounds from the csv file. That would work with newer data

version: '1'
modeling_realm: satellite
source: ''
reference: 'grace'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The source is missing

@remi-kazeroni
Copy link
Contributor

I have just made the changes ot fix the cmorizer based on the review I made. It would be great if someone could review this PR.

@bouweandela
Copy link
Member

The tests are failing on CircleCI, but probably not because of changes in this pull request. The easiest way to solve this is probably to merge the master branch into this branch. It's always nice to make sure all tests are green before requesting a human review.

@remi-kazeroni
Copy link
Contributor

The tests are failing on CircleCI, but probably not because of changes in this pull request. The easiest way to solve this is probably to merge the master branch into this branch. It's always nice to make sure all tests are green before requesting a human review.

Thanks, this has fixed the failing tests. Codacy issue is also fixed, it's all green now.

Copy link
Contributor Author

@bascrezee bascrezee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @remi-kazeroni for picking this up! The changes that you made look good to me.

@remi-kazeroni
Copy link
Contributor

Thanks @remi-kazeroni for picking this up! The changes that you made look good to me.

You're welcome @bascrezee! It was just a few fixes to account for the latest GRACE data release.

@remi-kazeroni
Copy link
Contributor

@axel-lauer I think this is ready to merge

@axel-lauer axel-lauer merged commit a1d02f0 into master Dec 11, 2020
@axel-lauer axel-lauer deleted the cmorize-grace branch December 11, 2020 13:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

New obs dataset GRACE
7 participants