Transferring some repositories over

[natgeo-wong]

Hi!

I plan to transfer over both the ClimateSatellite.jl, GeoRegions.jl and ClimateERA.jl repositories into the JuliaClimate organization. Would like to request permission to modify settings for them as well.

I might be renaming the ClimateERA.jl repository, however. Does anyone have suggestions for names? Or do people think that ClimateERA is fine?

The following things still need to be done for these repositories:

• ClimateSatellite.jl
  • Documentation
  • Jupyter Notebook Examples
  • TRMM, RSS and MODIS Satellite retrieval algorithms
• ClimateERA.jl (or whatever name I finalize on)
  • Documentation
  • Jupyter Notebook Examples
  • Calculations for basic variables not given (e.g. meridional streamfunction, eddy kinetic energy, etc.)

All questions and comments are welcome.

On a separate but related notebook, where should I put Jupyter Notebook examples? In a separate repository here? Or inside my packages?

Tagging @gaelforget, @Balinus and @Datseris as they are the main contributors to most of the packages of the JuliaClimate community thus far.

Edit: Transferring GeoRegionsjl over as well. Same issues with Documentation and examples also apply, but I plan to get them up once the semester ends.

[Datseris]

Awesome!

On a separate but related notebook, where should I put Jupyter Notebook examples? In a separate repository here? Or inside my packages?

I don't recommend inside the package. Takes space! What I Do in JuliaDynamics is have a central repo that hosts all heavy things, e.g. videos, high quality images, and tutorials. Maybe we should consider the same here? In the case of JuliaDynamics, the same repo hosts the organization website:

https://juliadynamics.github.io/JuliaDynamics/

https://github.com/JuliaDynamics/JuliaDynamics

[Balinus]

Great!

For the Notebooks, there is https://github.com/JuliaClimate/GlobalOceanNotebooks

Perhaps the Repo can me renamed to someting more general or we can create a new repo ClimateNotebooks, something like that?

[natgeo-wong]

Pinging @gaelforget! could I have permissions for GeoRegions.jl and ClimateSatellite.jl? Thanks!

[gaelforget]

Pinging @gaelforget! could I have permissions for GeoRegions.jl and ClimateSatellite.jl? Thanks!

Will do asap -- hopefully tonight...

[gaelforget]

I don't recommend inside the package. Takes space!

Agreed. That's sort of how I use the globalocenanotebooks folder. Couple things I could add:

• you can use https://github.com/mwouts/jupytext , just commit the paired light script file (.jl), and add something *.ipynb in .gitignore. It's kind of the best of both world when you dont care for GitHub rendering the ipynb. (even when I do include the ipynb I put the jl file cause diff is generally useless for ipynb)
• I like the idea of relatively focused repos also for notebook folders. I have been thinking about getting another going on feedback mechanisms for example. Also I could imagine that when one notebook folder becomes too large then it needs to be split up to e.g. avoid slow cloning / downloading, long builds (dependencies), or just hit GitHub size limits etc.

What I Do in JuliaDynamics is have a central repo that hosts all heavy things, e.g. videos, high quality images, and tutorials. Maybe we should consider the same here? In the case of JuliaDynamics, the same repo hosts the organization website:

https://juliadynamics.github.io/JuliaDynamics/

https://github.com/JuliaDynamics/JuliaDynamics

Impressive! I guess you have the website there too.

What's the simplest way to try the notebooks in mybinder.org for example?

It's probably in the docs but by just looking at the README.md in JuliaDynamics/JuliaDynamics I am not immediately sure where dependency spec files are (toml etc) for example. It's kind of nice to be able to just push a binder to explore.

ps. sorry if some of this is unclear -- probably typing too quickly...

[gaelforget]

Great!

For the Notebooks, there is https://github.com/JuliaClimate/GlobalOceanNotebooks

I definitely like this idea -- please dont hesitate, any or you, to send PRs if you'd like. We can always figure things out from there.

couple side notes:

I have a few things setup to build with this (see https://github.com/JuliaClimate/GlobalOceanNotebooks/blob/master/environment.yml) in addition to the packages in Project.toml like the RISE & jupytext plugins

In some cases I left dependencies out of the standard build though -- e.g. the commented bloc of code at the top of the climatetools notebook (https://github.com/JuliaClimate/GlobalOceanNotebooks/blob/master/DataStructures/02_ClimateTools.ipynb)

Perhaps the Repo can me renamed to someting more general or we can create a new repo ClimateNotebooks, something like that?

I am open to renaming at some point but maybe I would rather postpone doing it for now -- see where things go etc and maybe we readily should create more repos

[Datseris]

All notebooks are in the tutorial folder here, including exercises and other material I use in my workshops:

https://github.com/JuliaDynamics/JuliaDynamics/tree/master/tutorials

I personally would much rather if all climate-science related tutorials and notebooks are in a single repo, even if their topic is diverse. Folders can help organizing and makes everything easier to manage from our perspective. At least I personally see no benefit in having multiple repos for tutorials, jupyter notebooks, and the like.

I guess you have the website there too.

Yeap, hosted on github pages as well!

[gaelforget]

Pinging @gaelforget! could I have permissions for GeoRegions.jl and ClimateSatellite.jl? Thanks!

I just set ClimateEasy-owners' privileges to admin for those two repos. All set?

[natgeo-wong]

Actually in order to configure Travis CI, I need to be an organization owner >.> I'm so sorry for the trouble.

[Balinus]

I've approved the request for travis-ci on your repo. Tell us if it does not work.

[gaelforget]

Regarding the ClimateERA.jl name maybe I would suggest something like AtmosphericReanalysisData.jl, AtmosphericReanalyses.jl, ClimateReanalyses.jl or if you wanted to be more broad ReanalysisData.jl (e.g. to potentially incl. ocean & land products) or ECMWFdata.jl to be more restrictive (may you'd need to check on their trademarks?).

My sense is that products NCEP, MERRA, JAMSTEC, etc can also be of great interest and providing some sort of common interface to be facilitate intercomparisons (no pressure; I am basically trying to think in the future) would be really nice. Happy to keep thinking about other options if you'd like.

[natgeo-wong]

Travis CI works now! Thanks so much!

I have not yet looked at NCEP or MERRA data, so I do not know precisely what their storage format is like. Does anyone know if these datasets are also saved in NetCDF?

If I do decide to include these other atmospheric reanalysis products, I will need to think of a way to download those data directly using Julia, if possible (it is not possible to do this for ERA Reanalysis, unfortunately, since their API is strictly Python-centric).

[Balinus]

MERRA2 is on netCDF

https://disc.gsfc.nasa.gov/datasets/M2T1NXINT_5.12.4/summary?keywords=%22MERRA-2%22

[gaelforget]

Right! I actually went to gather web links that I forgot to post here. Will do ASAP

…

Sent from my iPhone

On May 13, 2020, at 8:14 AM, Philippe Roy ***@***.***> wrote:  MERRA2 is on netCDF https://disc.gsfc.nasa.gov/datasets/M2T1NXINT_5.12.4/summary?keywords=%22MERRA-2%22 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[gaelforget]

Right! I actually went to gather web links that I forgot to post here. Will do ASAP
…

Here are a few things that might be useful:

https://climatedataguide.ucar.edu/climate-data/nasa-merra
https://climatedataguide.ucar.edu/climate-data/ncep-ncar-r1-overview
https://climatedataguide.ucar.edu/climate-data/ncep-reanalysis-r2

https://psl.noaa.gov/data/gridded/
https://psl.noaa.gov/thredds/catalog/Datasets/ncep.reanalysis/catalog.html
https://psl.noaa.gov/thredds/catalog/Datasets/ncep.reanalysis2/catalog.html

https://catalog.pangeo.io/browse/master/
https://github.com/pangeo-data/pangeo-datastore/blob/master/intake-catalogs/atmosphere.yaml

https://gmao.gsfc.nasa.gov/reanalysis/MERRA-2/
https://gmao.gsfc.nasa.gov/reanalysis/MERRA-2/data_access/

Some side notes :

• there is no rush to adding any of these. It's just an idea that seems appealing to me.
• I have a bit of experience with computing air-sea fluxes using ERA product (e.g. I use ERA-interim via bulk formulae in ECCO v4, https://doi.org/10.5194/gmd-8-3071-2015 ) and also have an ongoing line of collaboration with MERRA2 / GMAO folks ( https://doi.org/10.1029/2019GL085837 & https://doi.org/10.1016/j.ocemod.2018.10.006 ) which hopefully could help down the line.
• with @udistr (lead author of the work cited above) we started putting together a package called https://github.com/gaelforget/AirSeaFluxes.jl -- maybe we could try recomputing air-sea fluxes from MERRA2 & ERA using the included bulk formulae some time

Have you given more thought to the package name / scope @natgeo-wong ?

[natgeo-wong]

A thought that occurred to me is that there could eventually be a package that deals with all reanalysis packages, much like in ClimateSatellite. However, that would be a major task to undertake and I don't have the expertise right now to push such a change (I need to get familiar with the download system for MERRA and NCEP first).

I'm taking that I might move ClimateERA over first, and then when the time is right (and when I'm more familiar with NCEP and MERRA), I would create a new package with the next name and deprecate ClimateERA.

I like either ReanalysisData.jl or ClimateReanalysis.jl!

Sorry haven't been too responsive lately. Working on IscaTools and in using it for a class project the last few days.

[Balinus]

ReanalysisData.jl is a nice generic name while ClimateReanalysis.jl seems a little bit more specific.

The proposed plan seems promising. No need to rush doing things we can't support in the long-term. It's better, imho, to better develop core features.