Meeting with Ouranos: Oct 2020

[agstephens]

Proposed agenda for meeting - 01/10/2020:

1. Introductions

2. Project update: c3s_34e [Ag]

3. Project updates: relevant Ouranos/CRIM projects [David/Tom?]

• of particular interest to 34e:
  • latest status of regridding work
  • latest status of NcML work

4. Discussion:

• change procedures for clisops:

  • what do we need to change about current procedures?
  • what does Ouranos need from 34e?
  • should master always be fully working?
  • do we need a package for creating/pushing a set of related releases?

• can we make a new release by mid-October?:

  • needed by 34e

• coding style:

  • use of Typing library in clisops

• sub-setting by level in clisops.core.subset:

  • issues
  • are we happy structure of subset.py?
  • who will write prototype?
  • who will review?
  • who will accept?

• plans for average.py:

  • very limited functionality

• plans for regrid.py:

  • xESMF

• xarray extension for clisops

• plans for improving checks and fixes code in dachar and daops:

  • plan to work with ESMValTool team
  • learning from PANGEO work for CMIP6

• other activities worth noting

5. Agree next steps and next meeting

[agstephens]

@huard @Zeitsperre @tomLandry @ellesmith88 @cehbrecht
please review the proposed agenda above and make/suggest changes. Thanks

[huard]

@agstephens Looks good, nothing to add on my end.

[tomLandry]

Ok on my end too. I think we made some relevant advances in our pilot project with OGC. The base services (subset, avg, compute) were used through OGC API - Processing (an Open API), instead of WPS1. For the rest, David is closer than me to the technical work that was done in our project(s). I invited my colleague Francis to join, unsure if he'll be there.

[agstephens]

Thanks folks, look forward to chatting later.

[agstephens]

C3S_34e meeting with Canadian Partners (Ouranos, CRIM) - 01/10/2020

Attendees

Ag Stephens, CEDA
Elle Smith, CEDA
Martin Schupfner, DKRZ
Carsten Ehbrecht, DKRZ
Travis Logan, Ouranos
David Huard, Ouranos
Pascal Bourgault, Ouranos
Trevor James Smith, Ouranos
Tom Landry, CRIM

Actions

• AS: set up a monthly meeting in this Thursday slot.
• AS: to discuss code management procedures with the 34e team
• TJS: Add a Pull Request template for CLISOPS: Add a Pull Request template for CLISOPS clisops#70
• AS: liaise with team to get a new release published this month.
• AS: discuss with 34e team how we add support for static typing: Add some static type support clisops#67
• AS: put a "what we have learnt about dask" item on the next meeting agenda

Agenda

See: #59

Notes

Project updates: C3S_34e

• funded by EU Copernicus
• C3S - Copernicus Climate Change Services
  • We shall provide access to ESGF data, yet access is possible to the whole files only
  • subsets, averaged, regridded datasets would be more efficient to save bandwith
• WPS service: birdhouse
• client to embed in the birdhouse / C3S infrastructure

Project updates: Ouranos

• xESMF added to PANGEO:

  • new release at v0.4
  • is being extended to allow area/polygon averaging

• NcML usage:

  • creating datasets (eg. concatenating historical and rcp by member_id) in NcML
  • workflow:
    • write a json config file containing fixes
    • batch write NcML files from that
  • files can be added by scanning folders for patterns
  • variables can be renamed, attributes altered (without touching the original files)
  • data has to be appropriate (some calendars, time units not yet supported)
  • significant data cleaning required with some non-compliant CMIP5 files
  • have not needed to extend the NcML specification
  • PANGEO: is planning to build NcML in to Xarray (2021)
    • interpretation/Translation of the NcML keys as xarray functions should be possible

Project updates: Ouranos

• Showed diagram of interconnections between activities and projects

• Some key OGC developments/standards that we should track:

  • EDR - Environmental Data Retrieval
  • DAPA - Data Access And Processing

• Weaver:

  • gets a Docker image of an app
  • registers a WPS
  • presents the WPS as an OGC Open API

• "climatedata.ca" is an activity similar to C3S Climate Data Store

• PyGeoAPI:

  • a relatively new stack/library that is gaining traction in the community

Discussion

1. Monthly meeting:

We agree a monthly meeting between our groups would be useful.

• AS: set up a monthly meeting in this Thursday slot.

2. Procedures for clisops:

Following discussion we agree to follow best practice in terms of:

• Any dependency changes need to be communicated to all developers.

  • All Pull Requests should:
    • be reviewed by at least one author before acceptance
    • mention any breaking changes
    • mention any dependency changes
    • include history in the release notes
  • The tests should always run on the "master" branch
  • Releases should include all important history in release notes

• AS: to discuss code management procedures with the 34e team

• TJS: Add a Pull Request template for CLISOPS: Add a Pull Request template for CLISOPS clisops#70

3. New release to be created for mid-October:

• AS: liaise with team to get a new release published this month.

4. Coding style:

34e is happy to add more static typing support to clisops.

• AS: discuss with 34e team how we add support for static typing: Add some static type support clisops#67

Structure of the clisops.core.subset module:

• in future we may choose to refactor it
• but it works fine right now :-)

5. Regridding:

• Ouranos is developing the core of xESMF
• 34e will build the high-level wrapper function: clisops.core.regrid::regrid

6. Future meetings:

• Travis requested a future demo/info about the 34e experience of using Dask

• AS: put a "what we have learnt about dask" item on the next meeting agenda

7. xarray extension for clisops:

This is not a high priority but we can easily extend xarray to add a clisops ("cso") namespace. See:

roocs/clisops#37

Ag: ESMValTool interested in dachar/daops -> progressing daops, dachar fixing
Travis: approach of json configs to specify fixes for a number of datasets, else fixes are only listed in the NcML

8. Link between "climatedata.ca" and the C3S CDS:

• noted that we should keep communicating about the possible links here
• in future it would be good to set up a meeting to share info/plans and look for collaborations

[agstephens]

@ellesmith88 @sol1105 @cehbrecht @tomLandry @huard @Zeitsperre : I've written the minutes from our meeting last week here in the github issue. Please review and let me know if you have any corrections/updates. Thanks