Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extended ESRL cmorizer #1937

Merged
merged 8 commits into from
Dec 9, 2020
Merged

Extended ESRL cmorizer #1937

merged 8 commits into from
Dec 9, 2020

Conversation

bettina-gier
Copy link
Contributor

@bettina-gier bettina-gier commented Dec 4, 2020

Required for #1915


Checklist for technical review

  • Create an issue to discuss what you are going to do, if you haven't done so already (and add the link at the bottom)
  • The pull request has a descriptive title that can be used as a one line summary in the changelog
  • The code is composed of functions of no more than 50 lines and uses meaningful names for variables and follows the style guide
  • Documentation is available
  • Please use yamllint to check that your YAML files do not contain mistakes
  • (Only if really necessary) Add any additional dependencies needed for the diagnostic script to setup.py, esmvaltool/install/R/r_requirements.txt or esmvaltool/install/Julia/Project.toml (depending on the language of your script) and also to package/meta.yaml for conda dependencies (includes Python and others, but not R/Julia). Also check that the license of the dependency you want to add and any of its dependencies are compatible with Apache2.0.

New or updated data reformatting script:

  • Add a new dataset to the table in the documentation
  • Add a test for the CMORized data to recipes/example/recipe_check_obs.yml and run the recipe, to make sure the CMOR checks pass without errors
  • There are clear instructions on how to obtain the data
  • Tag @remi-kazeroni in this pull request, so that the new dataset can be added to the OBS data pool at DKRZ and synchronized with CEDA-Jasmin

Automated checks pass, status can be seen below the pull request:

  • Circle/CI tests pass. If the tests are failing, click the Details link to find out why.
  • Preferably Codacy code quality checks pass, however a few remaining hard to solve Codacy issues are still acceptable. If there is an error, click the link to find out why. If you suspect Codacy may be wrong, please ask by commenting.
  • The documentation is building successfully on readthedocs and looks well formatted, click the Details link to see it.

Checklist for scientific review

New or updated data reformatting script:

  • The numbers and units of the data look physically meaningful

If you need help with any of the items on the checklists above, please do not hesitate to ask by commenting in the issue or pull request.

Old NCL ESRL cmorizer only worked for baseline observatories. New cmorizer rewritten in python, includes capability to process surface flask monthly data, as well as global mean data from ESRL website. Additionally writes altitude to the netcdf files, gets this information from the ESRL website table, no external file required. Kept the "OBS" datatable so recipes using data from the old cmorizer will not need to be changed. Will download data if not present in RAW folder.
Changed recipe_check_obs.yml to include one of each case. No change to documentation necessary as ESRL cmorizer already noted there.
Required for #1915

@bettina-gier
Copy link
Contributor Author

@axel-lauer Wanna do me the honor of a review, since you looked at Sabrinas original implementation for just the main station data? Remaining codacy error is the use of unsecure ftp for data download, copied this from the CT2019 cmorizer script. Feel free to change if you have a better idea.

@axel-lauer
Copy link
Contributor

axel-lauer commented Dec 7, 2020

Is there a reason why the new CMORizer only produces only ASK, GLOBAL and MLO? What happened to BRW, SMO, SPO?

@bettina-gier
Copy link
Contributor Author

I just put them there as example, the cmorizer can handle all base stations (BRW, SMO, SPO, MLO), the global average from their website, as well as all the single flask sites in the ESRL network (https://www.esrl.noaa.gov/gmd/dv/site/?program=ccgg). So I used one station for each example. I wasn't sure if it should take all the possible stations as default input.

@axel-lauer
Copy link
Contributor

The old CMORizer would output BRW, MLO, SMO, SPO. I would therefore suggest to additionally output BRW, SMO, SPO with the new version.

@bettina-gier
Copy link
Contributor Author

I've added support for an "all" keyword put in as default, which will process all available datasets - so this also includes brw, spo, mlo and smo -, and added a flag for downloading additional files, in case someone only wants to format all datasets locally available.
Also extended comment section in the yml file to better explain what files are supported.

I hope that addresses your concern sufficiently!

Copy link
Contributor

@axel-lauer axel-lauer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good. I tested the CMORizer and it does exactly what it is supposed to do. The only thing left to do would be to try to fix (some of) the Codacy issues. @bettina-gier could you please take a look? Other than that, this is ready for merging.

@bettina-gier
Copy link
Contributor Author

Thanks for testing Axel!
Fixed all I could, only remaining one is the "insecure" use of the ftp package, but that is already merged with the CT2019 cmorizer so doesn't introduce anything new.

Copy link
Contributor

@axel-lauer axel-lauer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing the Codacy issues.

@axel-lauer axel-lauer merged commit c989957 into master Dec 9, 2020
@axel-lauer axel-lauer deleted the ESRL_cmorizer branch December 9, 2020 12:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants