Xclim : Xarray-based climate data analytics

[Zeitsperre]

Submitting Author: Trevor James Smith (@Zeitsperre)
All current maintainers: (@Zeitsperre, @tlogan2000, @aulemahal)
Package Name: xclim
One-Line Description of Package: Climate indices computation package based on Xarray
Repository Link:  https://github.com/Ouranosinc/xclim
Version submitted: v0.40.0
Editor: @Batalex
Reviewer 1: @jmunroe
Reviewer 2: @aguspesce
Archive:
JOSS DOI:
Version accepted: v0.42.0
Date accepted (month/day/year): 04/11/2023

---

Description

xclim is an operational Python library for climate services, providing numerous climate-related indicator tools with an extensible framework for constructing custom climate indicators, statistical downscaling and bias adjustment of climate model simulations, as well as climate model ensemble analysis tools.

xclim is built using xarray_ and can seamlessly benefit from the parallelization handling provided by dask. Its objective is to make it as simple as possible for users to perform typical climate services data treatment workflows. Leveraging xarray and dask, users can easily bias-adjust climate simulations over large spatial domains or compute indices from large climate datasets.

Scope 

• Please indicate which category or categories this package falls under:
• Data retrieval
• Data extraction
• Data munging
• Data deposition
• Reproducibility
• Geospatial
• Education
• Data visualization*

Please fill out a pre-submission inquiry before submitting a data visualization package. For more info, see notes on categories of our guidebook.

• For all submissions, explain how and why the package falls under the categories you indicated above. In your explanation, please address the following points (briefly, 1-2 sentences for each): 

  - Who is the target audience, and what are scientific applications of this package?

xclim aims to position itself as a climate services tool for any researchers interested in using Climate and Forecast Conventions compliant datasets to perform climate analyses. This tool is optimized for working with Big Data in the climate science domain and can function as an independent library for one-off analyses in Jupyter notebooks or as a backend engine for performing climate data analyses over PyWPS (e.g. Finch). It was primarily developed targeting earth and environmental science audiences and researchers, originally for calculating climate indicators for the Canadian government web service ClimateData.ca.

The primary domains that xclim is built for are in calculating climate indicators, performing statistical correction / bias adjustment of climate model output variables/simulations, and in performing climate model simulation ensemble statistics.

  - Are there other Python packages that accomplish the same thing? If so, how does yours differ?

icclim is another library for the computation of climate indices. Starting with version 5.0 of icclim, some of the core computations rely on xclim. See explanations about differences between xclim and icclim.

scikit-downscale is a library offering algorithms for statistical downscaling. xclim drew inspiration for its fit-predict architecture. The suite of downscaling algorithms offered differ.

Technical checks

For details about the pyOpenSci packaging requirements, see our packaging guide. Confirm each of the following by checking the box.  This package:

• does not violate the Terms of Service of any service it interacts with. 
• has an OSI approved license.
• contains a README with instructions for installing the development version. (Under CONTRIBUTING)
• includes documentation with examples for all functions.
• contains a vignette with examples of its essential functions and uses.
• has a test suite.
• has continuous integration, such as Travis CI, AppVeyor, CircleCI, and/or others.

Publication options

• Do you wish to automatically submit to the Journal of Open Source Software? If so:

 

JOSS Checks

 

• The package has an obvious research application according to JOSS's definition in their submission requirements. Be aware that completing the pyOpenSci review process does not guarantee acceptance to JOSS. Be sure to read their submission requirements (linked above) if you are interested in submitting to JOSS.
• The package is not a "minor utility" as defined by JOSS's submission requirements: "Minor 'utility' packages, including 'thin' API clients, are not acceptable." pyOpenSci welcomes these packages under "Data Retrieval", but JOSS has slightly different criteria.
• The package contains a paper.md matching JOSS's requirements with a high-level description in the package root or in inst/.
• The package is deposited in a long-term repository with the DOI: https://doi.org/10.5281/zenodo.2795043

Note: Do not submit your package separately to JOSS
  

Are you OK with Reviewers Submitting Issues and/or pull requests to your Repo Directly?

This option will allow reviewers to open smaller issues that can then be linked to PR's rather than submitting a more dense text based review. It will also allow you to demonstrate addressing the issue via PR links.

• Yes, I am OK with reviewers submitting requested changes as issues to my repo. Reviewers will then link to the issues in their submitted review.

Code of conduct

• I agree to abide by pyOpenSci's Code of Conduct during the review process and in maintaining my package should it be accepted.

Please fill out our survey

• Last but not least, please fill out our pre-review survey. This helps us track
  submission and improve our peer review process. We will also ask our reviewers 
  and editors to fill this out.

P.S. *Have feedback/comments about our review process? Leave a comment here

Editor and Review Templates

Editor and review templates can be found here

[Zeitsperre]

To whom it may concern, our paper is available on a separate branch within xclim. The Pull Request containing the paper can be found here: Ouranosinc/xclim#250

[lwasser]

hi @Zeitsperre welcome to pyOpenSci!! are you a part of the pangeo community by chance? i think i saw xclim discussed in a few pangeo threads.

[Zeitsperre]

Hi @lwasser, nice to meet you!

We are not formally part of the Pangeo community (we do not receive compensation from them), however xclim is mentioned in their list of xarray-based projects (https://docs.xarray.dev/en/latest/ecosystem.html#ecosystem). A handful of xclim's developers have also made significant contributions to packages in the Pangeo ecosystem as well (xarray, xesmf, flox, cf-xarray, etc.) and some key developers of xarray have contributed to xclim, through issue raising or code contributions. I think it's safe to say that the objectives of xclim are in line with those of Pangeo.

[lwasser]

awesome! nice to meet you as well @Zeitsperre ! I asked because we are working on a partnership with pangeo to curate that list via peer review. as a part of that curation, we have a small set of checks that are specific to the pangeo community including:

• ​​Consume and produce high-level data structures (e.g. xarray datasets / pandas dataframes) wherever feasible
• ​​Operate lazily when called on dask data structure
• ​​Avoid file I/O unless specifically an I/O package

So we'd want to include you in that given your are in the list of tools. i haven't done a deep dive into your package but these are universal pangeo standards.If the review goes smoothly, we'd then tag your package as being pangeo community related / vetted.

do this package adhere to the above pangeo standards? Many thanks - i'm just trying to figure out the best path here to support our partnership :) i just started working on this and have an open pr here to update our review submission template!

[Zeitsperre]

I believe we tick all the boxes you listed:

• ​​Consume and produce high-level data structures (e.g. xarray datasets / pandas dataframes) wherever feasible (all functions operate on either xr.DataArray or xr.Dataset objects)
• ​​Operate lazily when called on dask data structure (both indices and indicators seemlessly integrate with dask)
• ​​Avoid file I/O unless specifically an I/O package (xarray is left to handle all file loading or writing)

I know that the core developers of xclim would gladly welcome feedback on how to better organize/structure our code base, and xclim being considered high enough in quality to be formally Pangeo-endorsed would be great!

[Zeitsperre]

Just to add some clarification, I just remembered that we have some functionality to build indicator catalogues using YAML configurations (https://xclim.readthedocs.io/en/latest/notebooks/extendxclim.html#YAML-file).

We also built a translation engine for providing multi-language climate indicator metadata based on JSON descriptions (currently, we have translations to French, but any other languages are supported; https://xclim.readthedocs.io/en/latest/internationalization.html).

These files are loaded on import or can be supplied by the user explicitly and are solely used for configuration. Hope this helps!

[lwasser]

perfect! Thank you @Zeitsperre for the speedy replies! I am going to followup and also get our editor in chief involved @NickleDave I stepped in only because i'm actively working with pangeo now so i just wanted to check in to see if this review could be supported via that collaboration as well. And it does sound like it could. more soon!

[NickleDave]

Welcome @Zeitsperre!

At first glance looks like you have got most everything in line with our initial checks.

I will add the completed checks by end of day tomorrow.

[NickleDave]

Hi again @Zeitsperre -- Happy to report that xclim passed initial editor-in-chief checks (below) with flying colors.

I have one suggestion, which we strongly recommend, but do not require:

• switch to pyproject.toml with static metadata, to adhere to pep 517 + 621, away from legacy setup.py file
  • looks like you mainly use setup.py to clean up README for build if I'm not mistaken?
  • I wonder if you could use other tools to the same ends? E.g., if you use hatch in place of setuptools there's a plug-in that will "prettify" the README for you, see https://talkpython.fm/episodes/show/395/tools-for-readme.md-creation-and-maintenance
  • I know you have a lot of config stuff in the setup.cfg, so switching to another dev tool might be a hassle, but having metadata in the pyproject.toml instead of a setup.py would be ideal

We expect to know how we will go about selecting editors by the end of day Monday (Jan. 23rd).
@lwasser needs to talk a little more with people about the Pangeo collaboration.
We will report back to you then with a timeline for the review.

Editor in Chief Checks

Please check our Python packaging guide for more information on the elements
below.

• Installation The package can be installed from a community repository such as PyPI (preferred), and/or a community channel on conda (e.g. conda-forge, bioconda).
  • The package imports properly into a standard Python environment import package-name.
    • verified for both pip (version 0.40.0)
• Fit The package meets criteria for fit and overlap.
• Documentation The package has sufficient online documentation to allow us to evaluate package function and scope without installing the package. This includes:
  • User-facing documentation that overviews how to install and start using the package.
    • two notes:
      • I wonder if it would help to move some text from about.md to the index? So a user immediately knows what xclim is and does. Getting hit with a long table of contents can be daunting
      • "Basic usage" is great although looks like it could use maybe use some very minor revision of text just for flow + readability
  • Short tutorials that help a user understand how to use the package and what it can do for them.
    • plenty of tutorials, looks excellent
  • API documentation (documentation for your code's functions, classes, methods and attributes): this includes clearly written docstrings with variables defined using a standard docstring format. We suggest using the Numpy docstring format.
• Core GitHub repository Files
  • README The package has a README.md file with clear explanation of what the package does, instructions on how to install it, and a link to development instructions.
  • Contributing File The package has a CONTRIBUTING.md file that details how to install and contribute to the package.
  • Code of Conduct The package has a Code of Conduct file.
  • License The package has an OSI approved license.
    NOTE: We prefer that you have development instructions in your documentation too.
• Issue Submission Documentation All of the information is filled out in the YAML header of the issue (located at the top of the issue template).
• Automated tests Package has a testing suite and is tested via GitHub actions or another Continuous Integration service.
• Repository The repository link resolves correctly.
• Package overlap The package doesn't entirely overlap with the functionality of other packages that have already been submitted to pyOpenSci.
• Archive (JOSS only, may be post-review): The repository DOI resolves correctly.
• Version (JOSS only, may be post-review): Does the release version given match the GitHub release (v1.0.0)?

---

• Initial onboarding survey was filled out
  We appreciate each maintainer of the package filling out this survey individually. 🙌
  Thank you authors in advance for setting aside five to ten minutes to do this. It truly helps our organization. 🙌

---

---

Editor comments

[Zeitsperre]

Thanks, @NickleDave! I'm glad to see that all the hours spent reading PEPs and adopting good practices has been helpful here!

I'll be opening a PR later today to address the PEP 517 and 621 compliance comment. We don't do anything too fancy in our setup.py, so I don't imagine it should be difficult to adopt pyproject.toml. I'll link that here once it's underway.

[Zeitsperre]

@NickleDave

I just merged changes to address some of your comments, primarily for pyproject.toml and the ReadMe suggestion. I opted to adopt flit as it seemed to be a very stable and no-nonsense setup engine. It took a bit of time to migrate some configurations around, but it looks/works much better.

Agreed on your point about Basic Usage. I'll be opening a ticket to address that down-the-line. There are a fair amount of Pull Requests currently open, but we tend to address things in a reasonable amount of time.

Let me know if you have any other suggestions, thanks!

[NickleDave]

Hi @Zeitsperre!

I just merged changes to address some of your comments

Awesome! Glad to hear it.

Agreed on your point about Basic Usage.

Thank you for even making note of that. It was just a nitpick and really more of a list to give an editor things they might want to revisit. Your docs look really great over all and I'm sure they'll look better after the review.

I'm sorry for not getting back to you by the end of the day yesterday--totally my fault.

I want to let you know we do have an editor, @Batalex! 🎉🎉🎉 who will introduce themselves and officially start the review in the next couple of days. We are actively recruiting reviewers right now as well.