Regridding functionalities (powered by xESMF) (#243)

* clisops.ops.regrid, clisops.core.regrid, clisops.core.Weights and clisops.core.Grid added. Allowing the remapping of geospatial data on various grids by applying the xESMF Regridder (https://pangeo-xesmf.readthedocs.io/en/latest/). * Extending the removal of redundant _FillValue attributes to all data variables and coordinates. * Extending the removal of redundant coordinates in the coordinates variable attribute from bounds to all data variables. * GitHub Workflows for upstream dependencies are now examined a schedule or via workflow_dispatch. * black steps are now called lint for clarity/inclusiveness of other linting hooks. * pre-commit hooks now include checks for TOML files, and for ReadTheDocs and GitHub Actions configuration files. --------- Co-authored-by: ellesmith88 <e.s.smith@hotmail.co.uk> Co-authored-by: Ag Stephens <ag.stephens@stfc.ac.uk> Co-authored-by: Eleanor Smith <esmith88@sci4.jasmin.ac.uk> Co-authored-by: Martin Schupfner <martin.schupfner@web.de> Co-authored-by: Trevor James Smith <10819524+Zeitsperre@users.noreply.github.com>
roocs · Nov 23, 2023 · a999bcb · a999bcb
1 parent 37967bd
commit a999bcb
Show file tree

Hide file tree

Showing 36 changed files with 5,830 additions and 107 deletions.
diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
@@ -7,12 +7,13 @@ on:
   pull_request:
 
 jobs:
-  black:
+  lint:
+    name: Linting Suite
     runs-on: ubuntu-latest
     strategy:
       matrix:
         tox-env:
-          - black
+          - lint
     steps:
       - name: Cancel previous runs
         uses: styfle/cancel-workflow-action@0.11.0
@@ -28,8 +29,8 @@ jobs:
         run: tox -e ${{ matrix.tox-env }}
 
   pypi:
-    name: Pip with Python${{ matrix.python-version }}
-    needs: black
+    name: Python${{ matrix.python-version }} (PyPI + Tox)
+    needs: lint
     runs-on: ubuntu-latest
     strategy:
       matrix:
@@ -61,17 +62,19 @@ jobs:
         COVERALLS_SERVICE_NAME: github
 
   conda:
-    name: Anaconda Build with Python${{ matrix.python-version }} (upstream=${{ matrix.upstream }})
-    needs: black
+    name: Python${{ matrix.python-version }} (Anaconda, upstream=${{ matrix.upstream }})
+    needs: lint
     runs-on: ubuntu-latest
     strategy:
       fail-fast: false
       matrix:
         include:
           - python-version: "3.9"
-            upstream: true
+            upstream: false
           - python-version: "3.10"
             upstream: false
+          - python-version: "3.11"
+            upstream: false
     defaults:
       run:
         shell: bash -l {0}

diff --git a/.github/workflows/upstream.yml b/.github/workflows/upstream.yml
@@ -0,0 +1,81 @@
+name: Test Upstream Dependencies
+
+on:
+  push:
+    branches:
+      - master
+    paths-ignore:
+      - HISTORY.rst
+      - README.rst
+      - pyproject.toml
+      - setup.cfg
+      - clisops/__init__.py
+  schedule:
+    - cron: "0 0 * * *" # Daily “At 00:00” UTC
+  workflow_dispatch: # allows you to trigger the workflow run manually
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  upstream-dev:
+    name: test-upstream-dev (Python${{ matrix.python-version }})
+    runs-on: ubuntu-latest
+    if: |
+      (github.event_name == 'schedule') ||
+      (github.event_name == 'workflow_dispatch') ||
+      (github.event_name == 'push')
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version:
+          - "3.10"
+    defaults:
+      run:
+        shell: bash -l {0}
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0 # Fetch all history for all branches and tags.
+      - name: Setup Conda (Micromamba) with Python${{ matrix.python-version }}
+        uses: mamba-org/setup-micromamba@v1
+        with:
+          cache-downloads: true
+          cache-environment: true
+          environment-file: environment.yml
+          create-args: >-
+            conda
+            python=${{ matrix.python-version }}
+            pytest-reportlog
+      - name: Conda and Mamba versions
+        run: |
+          conda --version
+          echo "micromamba: $(micromamba --version)"
+      - name: Install upstream versions
+        run: |
+          python -m pip install -r requirements_upstream.txt
+      - name: Install CLISOPS
+        run: |
+          python -m pip install --no-user --editable ".[dev]"
+      - name: Install upstream versions
+        run: |
+          python -m pip install -r requirements_upstream.txt
+      - name: Check versions
+        run: |
+          conda list
+          python -m pip check || true
+      - name: Run Tests
+        if: success()
+        id: status
+        run: |
+          python -m pytest --durations=10 --cov=clisops --cov-report=term-missing --report-log output-${{ matrix.python-version }}-log.jsonl
+      - name: Generate and publish the report
+        if: |
+          failure()
+          && steps.status.outcome == 'failure'
+          && github.event_name == 'schedule'
+          && github.repository_owner == 'roocs'
+        uses: xarray-contrib/issue-from-pytest-log@v1
+        with:
+          log-path: output-${{ matrix.python-version }}-log.jsonl
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -1,51 +1,55 @@
 default_language_version:
-    python: python3
+  python: python3
 
 repos:
--   repo: https://github.com/asottile/pyupgrade
+  - repo: https://github.com/asottile/pyupgrade
     rev: v3.15.0
     hooks:
-    -   id: pyupgrade
+      - id: pyupgrade
         args: [ '--py38-plus' ]
--   repo: https://github.com/pre-commit/pre-commit-hooks
+  - repo: https://github.com/pre-commit/pre-commit-hooks
     rev: v4.5.0
     hooks:
-    -   id: trailing-whitespace
+      - id: trailing-whitespace
         exclude: setup.cfg
-    -   id: end-of-file-fixer
+      - id: end-of-file-fixer
         exclude: setup.cfg
-    -   id: check-yaml
-    -   id: debug-statements
-    -   id: mixed-line-ending
--   repo: https://github.com/psf/black-pre-commit-mirror
+      - id: check-toml
+      - id: check-yaml
+      - id: debug-statements
+      - id: mixed-line-ending
+  - repo: https://github.com/psf/black-pre-commit-mirror
     rev: 23.11.0
     hooks:
     -   id: black
-        args: ["--target-version", "py38"]
--   repo: https://github.com/pycqa/flake8
+  - repo: https://github.com/pycqa/flake8
     rev: 6.1.0
     hooks:
-    -   id: flake8
-        args: ['--config=setup.cfg']
--   repo: https://github.com/PyCQA/isort
+      - id: flake8
+        args: [ '--config=setup.cfg' ]
+  - repo: https://github.com/PyCQA/isort
     rev: 5.12.0
     hooks:
-    -   id: isort
-        args: ['--profile', 'black']
+      - id: isort
 #-   repo: https://github.com/pycqa/pydocstyle
 #    rev: 6.1.1
 #    hooks:
 #    -   id: pydocstyle
 #        args: ["--convention=numpy"]
--   repo: https://github.com/kynan/nbstripout
+  - repo: https://github.com/kynan/nbstripout
     rev: 0.6.1
     hooks:
-    -   id: nbstripout
+      - id: nbstripout
         files: ".ipynb"
--   repo: meta
+  - repo: https://github.com/python-jsonschema/check-jsonschema
+    rev: 0.27.1
     hooks:
-    -   id: check-hooks-apply
-    -   id: check-useless-excludes
+      - id: check-github-workflows
+      - id: check-readthedocs
+  - repo: meta
+    hooks:
+      - id: check-hooks-apply
+      - id: check-useless-excludes
 
 ci:
   autofix_commit_msg: |
@@ -55,5 +59,5 @@ ci:
   autoupdate_branch: ''
   autoupdate_commit_msg: '[pre-commit.ci] pre-commit autoupdate'
   autoupdate_schedule: quarterly
-  skip: [ ]
+  skip: []
   submodules: false
diff --git a/.readthedocs.yml b/.readthedocs.yml
@@ -20,7 +20,7 @@ build:
     python: "mambaforge-22.9"
 
 conda:
-  environment: environment.yml
+  environment: docs/environment.yml
 
 # Optionally set the version of Python and requirements required to build your docs
 python:

diff --git a/HISTORY.rst b/HISTORY.rst
@@ -4,10 +4,22 @@ Version History
 v0.12.0 (unreleased)
 --------------------
 
+New Features
+^^^^^^^^^^^^
+* ``clisops.ops.regrid``, ``clisops.core.regrid``, ``clisops.core.Weights`` and ``clisops.core.Grid`` added (#TBA). Allowing the remapping of geospatial data on various grids by applying the `xESMF <https://pangeo-xesmf.readthedocs.io/en/latest/>`_ regridder.
+
 Bug Fixes
 ^^^^^^^^^
 * Calling `subset_shape()` with a `locstream case` (#288) returned all coordinates inside `inner_mask` which is equivalent to the bounding box of the polygon, not the area inside the polygon. Fixed by defining the `inner_mask` in `subset_shape()` for the locstream case. (#292).
 
+Other Changes
+^^^^^^^^^^^^^
+* Extending the removal of redundant _FillValue attributes to all data variables and coordinates.
+* Extending the removal of redundant coordinates in the coordinates variable attribute from bounds to all data variables.
+* GitHub Workflows for upstream dependencies are now examined a schedule or via `workflow_dispatch` (#243).
+* `black` steps are now called `lint` for clarity/inclusiveness of other linting hooks. (#243).
+* pre-commit hooks now include checks for TOML files, and for ReadTheDocs and GitHub Actions configuration files. (#243).
+
 v0.11.0 (2023-08-22)
 --------------------
 

diff --git a/README.rst b/README.rst
@@ -30,6 +30,24 @@ The package provides the following operations:
 * average
 * regrid
 
+Online Demo
+-----------
+
+..
+  todo: Links have to be adjusted to the master or respective branch!
+
+You can try clisops online using Binder (just click on the binder link below),
+or view the notebooks on NBViewer.
+
+.. image:: https://mybinder.org/badge_logo.svg
+   :target: https://mybinder.org/v2/gh/roocs/clisops/master?filepath=notebooks
+   :alt: Binder Launcher
+
+.. image:: https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg
+   :target: https://nbviewer.jupyter.org/github/roocs/clisops/tree/master/notebooks/
+   :alt: NBViewer
+   :height: 20
+
 Credits
 -------
 

diff --git a/binder/environment.yml b/binder/environment.yml
@@ -0,0 +1,37 @@
+name: clisops
+channels:
+ - conda-forge
+dependencies:
+ - python >=3.8,<3.12
+ - flit
+ - bottleneck >=1.3.1
+ - cf_xarray >=0.8.5
+ - cftime >=1.4.1
+ - dask >=2.6.0
+ - gdal >=3.0
+ - geopandas >=0.11
+ - loguru >=0.5.3
+ - netCDF4 >=1.4
+ - numpy >=1.16
+ - packaging
+ - pandas >=1.0.3
+ - pooch
+ - poppler >=0.67
+ - pyproj >=3.3.0
+ - requests >=2.0
+ - roocs-grids>=0.1.2
+ - roocs-utils >=0.6.4,<0.7
+ - shapely >=1.9
+ - xarray >=0.21,<2023.3.0 # https://github.com/pydata/xarray/issues/7794
+ - xesmf >=0.8.2
+ - cartopy>=0.20.2
+ - jupyterlab
+ - GitPython
+ - matplotlib>=3.5.2
+ # Upstream
+ - pip
+ - pip:
+   - psy-maps
+   - clisops
+#   - cf-xarray @ git+https://github.com/xarray-contrib/cf-xarray/@main#egg=cf-xarray
+#   - roocs-utils @ git+https://github.com/roocs/roocs-utils.git@master#egg=roocs-utils
diff --git a/binder/postBuild b/binder/postBuild
@@ -1,2 +1,3 @@
-pip install matplotlib
-pip install .
+set -e
+
+git clone https://github.com/roocs/mini-esgf-data ~/.mini-esgf-data
diff --git a/clisops/core/__init__.py b/clisops/core/__init__.py
@@ -9,3 +9,5 @@
     subset_time_by_components,
     subset_time_by_values,
 )
+
+from .regrid import Grid, Weights, regrid, weights_cache_init, weights_cache_flush