Upgrade to tox v4.0 and adjust local testing default configurations

.github/workflows/main.yml

@@ -20,7 +20,7 @@ on:
       - submitted
 
 env:
-  MAIN_TESTDATA_BRANCH: main
+  XCLIM_TESTDATA_BRANCH: main
 
 jobs:
   black:
@@ -40,7 +40,7 @@ jobs:
         with:
           python-version: "3.8"
       - name: Install tox
-        run: pip install tox~=3.0
+        run: pip install tox~=4.0
       - name: Run linting suite
         run: tox -e black
 
@@ -56,7 +56,7 @@ jobs:
         with:
           python-version: "3.8"
       - name: Install tox
-        run: pip install tox~=3.0
+        run: pip install tox~=4.0
       - name: Test with tox
         run: tox -e py38
         env:
@@ -76,7 +76,7 @@ jobs:
         with:
           python-version: "3.9"
       - name: Install tox
-        run: pip install tox~=3.0
+        run: pip install tox~=4.0
       - name: Test with tox
         run: tox -e py39-upstream-lm3-coverage
         env:
@@ -115,8 +115,9 @@ jobs:
           python -m pip install --no-user .
       - name: Test
         run: |
+          pip list
           pip check
-          pytest xclim --durations=10
+          pytest xclim --num-workers=logical --durations=10 --cov=xclim --cov-report=term-missing
         env:
           CONDA_EXE: mamba
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
@@ -156,7 +157,7 @@ jobs:
         with:
           python-version: "3.8"
       - name: Install tox
-        run: pip install tox~=3.0
+        run: pip install tox~=4.0
       - name: Test with tox
         run: tox -e py38-lm3-coverage
         env:
@@ -184,10 +185,6 @@ jobs:
         run: pylint --rcfile=pylintrc --disable=import-error --exit-zero xclim
       - name: Test with tox
         run: tox -e notebooks_doctests
-      - name: Report coverage
-        run: |
-          pip install --upgrade coveralls
-          coveralls
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           COVERALLS_FLAG_NAME: run-notebooks_doctests-py38

HISTORY.rst

@@ -13,6 +13,18 @@ New indicators
 * New indices and indicators for determining upwelling radiation (`shortwave_upwelling_radiation_from_net_downwelling` and `longwave_upwelling_radiation_from_net_downwelling`; CF variables `rsus` and `rlus`) from net and downwelling radiation (shortwave: `rss` and `rsds`; longwave: `rls` and `rlds`). (:pull:`1271`).
 * New indice and indicator (``dryness_index``) for estimating soil humidity classifications for winegrowing regions (based on Riou et al. (1994)). (:issue:`355`, :pull:`1235`).
 
+Breaking changes
+^^^^^^^^^^^^^^^^
+* `xclim` testing default behaviours have been changed (:issue:`1295`, :pull:`1297`):
+   * Running `$ pytest` will no longer use `pytest-xdist` distributed testing be default (can be set with ``-n auto|logical|#``. Coverage is also no longer gathered/reported by default.
+   * Running `$ tox` will now set `pytest-xdist` to use ``-n logical`` processes (with a max of 10).
+   * Default behaviour for testing is to no longer always fetch `xclim-testdata`. If testdata is found in ``$HOME/.xclim_testing_data``, files will be copied to individual processes, otherwise, will be fetched as needed.
+* Environment variables evaluated when running pytest have been changed (:issue:`1295`, :pull:`1297`):
+   * For testing against specific branches of `xclim-testdata`: ``MAIN_TESTDATA_BRANCH`` -> ``XCLIM_TESTDATA_BRANCH``
+   * The option to skip fetching of testdata (``SKIP_TEST_DATA``) has been removed
+   * A new environment variable (``XCLIM_PREFETCH_TESTING_DATA``) is now available to gather `xclim-testdata` before running test ensemble (default: `False`).
+   * Environment variables are now passed to `tox` on execution.
+
 Bug fixes
 ^^^^^^^^^
 * ``build_indicator_module_from_yaml`` now accepts a ``reload`` argument. When re-building a module that already exists, ``reload=True`` removes all previous indicator before creating the new ones. (:issue:`1192`, :pull:`1284`).
@@ -27,7 +39,8 @@ Internal changes
 * Coveralls GitHub Action removed as it did not support ``pyproject.toml``-based configurations. (:pull:`1278`).
 * Add a remark about how `xclim`'s CFFWIS is different from the original 1982 implementation. (:issue:`1104`, :pull:`1284`).
 * Update CI runs to use Python3.9 when examining upstream dependencies. Replace `setup-conda` action with `provision-with-micromamba` action. (:pull:`1286`).
-* Update CI runs to always use `tox~=3.0` and the `latest` Ubuntu images (now `v22.04`). (:pull:`1288`).
+* ~Update CI runs to always use `tox~=3.0` and the `latest` Ubuntu images (now `v22.04`). (:pull:`1288`).~
+   * CI and tox defaults now use `tox~4.0` and the `latest` Ubuntu images. (:pull:`1297`).
 * `SBCK` installation command now points to the official development repository. (:pull:`1288`).
 * Some references in the BibTeX were updated to point to better resources. (:pull:`1288`).
 * Add a GitHub CI workflow for performing dependency security review scanning. (:pull:`1287`).

environment.yml

@@ -58,6 +58,7 @@ dependencies:
     - statsmodels
     - tokenize-rt
     - tox
+#    - tox-conda  # Will be added when a tox@v4.0+ compatible plugin is released.
     - xdoctest
     - yamale
     - yamllint

pyproject.toml

@@ -70,6 +70,7 @@ dev = [
   "pytest-xdist[psutil] >=3.2",
   "tokenize-rt",
   "tox",
+  # "tox-conda",  # Will be added when a tox@v4.0+ compatible plugin is released.
   "xdoctest",
   "yamale",
   "yamllint",
@@ -165,10 +166,8 @@ ignore_missing_imports = true
 addopts = [
   "--verbose",
   "--color=yes",
-  "--cov=xclim",
-  "--cov-report=term-missing",
-  "--numprocesses=auto",
-  "--maxprocesses=6",
+  "--numprocesses=0",
+  "--maxprocesses=8",
   "--dist=worksteal"
 ]
 norecursedirs = ["docs/notebooks/*"]

tox.ini

@@ -1,16 +1,15 @@
 [tox]
+min_version = 4.0
 env_list =
     black
     docs
-    notebooks_doctests
+    notebooks_doctests{-coverage,}
 ;    opt-slow
     py38-lm3
     py39-upstream-doctest
     py310-slow
-isolated_build = True
 requires =
     pip >= 21.0
-    tox ~=3.0
 opts = -v
 
 [testenv:black]
@@ -35,6 +34,7 @@ commands =
     blackdoc --check docs
     isort --check xclim
     yamllint --config-file=.yamllint.yaml xclim
+commands_post =
 
 [testenv:docs]
 description = Build the documentation with makefile under {basepython}
@@ -44,6 +44,7 @@ setenv =
     READTHEDOCS = 1
 commands =
     make docs
+commands_post =
 allowlist_externals =
     env
     make
@@ -55,11 +56,11 @@ allowlist_externals =
 ;conda_env = environment-dev.yml
 ;extras =
 
-[testenv:notebooks_doctests]
+[testenv:notebooks_doctests{-coverage,}]
 description = Run notebooks and doctests with pytest under {basepython}
 commands =
-    pytest --no-cov --nbval --dist=loadscope docs/notebooks --durations=10 --ignore=docs/notebooks/example.ipynb
-    pytest --rootdir xclim/testing/tests/ --xdoctest xclim --ignore=xclim/testing/tests/ --durations=10
+    pytest --no-cov --nbval --dist=loadscope docs/notebooks --ignore=docs/notebooks/example.ipynb
+    pytest --rootdir xclim/testing/tests/ --xdoctest xclim --ignore=xclim/testing/tests/
 
 # Requires tox-conda compatible with tox@v4.0
 ;[testenv:opt-{slow,not_slow}]
@@ -76,27 +77,31 @@ commands =
 description = Run tests with pytest under {basepython}
 setenv =
     COV_CORE_SOURCE =
+    PYTEST_ADDOPTS = --numprocesses=logical --durations=10
+    coverage: PYTEST_ADDOPTS = --numprocesses=logical --durations=10 --cov=xclim --cov-report=term-missing
     PYTHONPATH = {toxinidir}
 passenv =
     CI
     CONDA_EXE
     COVERALLS_*
     GITHUB_*
     LD_LIBRARY_PATH
-    MAIN_TESTDATA_BRANCH
+    XCLIM_*
 extras = dev
 deps =
     coverage: coveralls
     lm3: git+https://github.com/OpenHydrology/lmoments3.git@develop#egg=lmoments3
     upstream: -rrequirements_upstream.txt
 install_command = python -m pip install --no-user {opts} {packages}
 download = True
-commands =
+commands_pre =
     pip list
     pip check
-    doctest: pytest --no-cov --rootdir xclim/testing/tests/ --xdoctest xclim --ignore=xclim/testing/tests/ --durations=10
-    !slow: pytest xclim -m "not slow" --durations=10
-    slow: pytest xclim --durations=10
+commands =
+    doctest: pytest --no-cov --rootdir xclim/testing/tests/ --xdoctest xclim --ignore=xclim/testing/tests/
+    !slow: pytest xclim -m "not slow"
+    slow: pytest xclim
+commands_post =
     coverage: - coveralls
 allowlist_externals =
     git

xclim/testing/tests/conftest.py

@@ -22,8 +22,8 @@
 from xclim.testing.utils import _default_cache_dir  # noqa
 from xclim.testing.utils import open_dataset as _open_dataset
 
-MAIN_TESTDATA_BRANCH = os.getenv("MAIN_TESTDATA_BRANCH", "main")
-SKIP_TEST_DATA = os.getenv("SKIP_TEST_DATA")
+TESTDATA_BRANCH = os.getenv("XCLIM_TESTDATA_BRANCH", "main")
+PREFETCH_TESTING_DATA = os.getenv("XCLIM_PREFETCH_TESTING_DATA")
 
 
 @pytest.fixture
@@ -518,7 +518,7 @@ def cmip3_day_tas(threadsafe_data_dir):
     ds = _open_dataset(
         "cmip3/tas.sresb1.giss_model_e_r.run1.atm.da.nc",
         cache_dir=threadsafe_data_dir,
-        branch=MAIN_TESTDATA_BRANCH,
+        branch=TESTDATA_BRANCH,
     )
     yield ds.tas
     ds.close()
@@ -527,7 +527,7 @@ def cmip3_day_tas(threadsafe_data_dir):
 @pytest.fixture(scope="session")
 def open_dataset(threadsafe_data_dir):
     def _open_session_scoped_file(
-        file: str | os.PathLike, branch: str = MAIN_TESTDATA_BRANCH, **xr_kwargs
+        file: str | os.PathLike, branch: str = TESTDATA_BRANCH, **xr_kwargs
     ):
         return _open_dataset(
             file, cache_dir=threadsafe_data_dir, branch=branch, **xr_kwargs
@@ -544,7 +544,7 @@ def add_imports(xdoctest_namespace, threadsafe_data_dir) -> None:
     ns["xr"] = xclim.testing  # xr.open_dataset(...) -> xclim.testing.open_dataset(...)
     ns["xclim"] = xclim
     ns["open_dataset"] = partial(
-        _open_dataset, cache_dir=threadsafe_data_dir, branch=MAIN_TESTDATA_BRANCH
+        _open_dataset, cache_dir=threadsafe_data_dir, branch=TESTDATA_BRANCH
     )  # Needed for modules where xarray is imported as `xr`
 
 
@@ -583,7 +583,7 @@ def atmosds(threadsafe_data_dir) -> xr.Dataset:
     return _open_dataset(
         threadsafe_data_dir.joinpath("atmosds.nc"),
         cache_dir=threadsafe_data_dir,
-        branch=MAIN_TESTDATA_BRANCH,
+        branch=TESTDATA_BRANCH,
     )
 
 
@@ -610,16 +610,19 @@ def gather_session_data(threadsafe_data_dir, worker_id, xdoctest_namespace):
     When running pytest with multiple workers, one worker will copy data remotely to _default_cache_dir while
     other workers wait using lockfile. Once the lock is released, all workers will copy data to their local
     threadsafe_data_dir."""
-    if worker_id == "master":
-        if not SKIP_TEST_DATA:
-            populate_testing_data(branch=MAIN_TESTDATA_BRANCH)
-    else:
-        if not SKIP_TEST_DATA:
+
+    if (
+        not _default_cache_dir.joinpath(TESTDATA_BRANCH).exists()
+        or PREFETCH_TESTING_DATA
+    ):
+        if worker_id in "master":
+            populate_testing_data(branch=TESTDATA_BRANCH)
+        else:
             _default_cache_dir.mkdir(exist_ok=True)
             test_data_being_written = FileLock(_default_cache_dir.joinpath(".lock"))
             with test_data_being_written as fl:
                 # This flag prevents multiple calls from re-attempting to download testing data in the same pytest run
-                populate_testing_data(branch=MAIN_TESTDATA_BRANCH)
+                populate_testing_data(branch=TESTDATA_BRANCH)
                 _default_cache_dir.joinpath(".data_written").touch()
             fl.acquire()
         shutil.copytree(_default_cache_dir, threadsafe_data_dir)

xclim/testing/tests/data.py

@@ -17,7 +17,7 @@
 from xclim.testing import open_dataset as _open_dataset
 from xclim.testing.utils import _default_cache_dir  # noqa
 
-MAIN_TESTDATA_BRANCH = os.getenv("MAIN_TESTDATA_BRANCH", "main")
+TESTDATA_BRANCH = os.getenv("XCLIM_TESTDATA_BRANCH", "main")
 TD = Path(__file__).parent / "data"
 
 
@@ -34,7 +34,7 @@ def generate_atmos(cache_dir: Path):
     with _open_dataset(
         "ERA5/daily_surface_cancities_1990-1993.nc",
         cache_dir=cache_dir,
-        branch=MAIN_TESTDATA_BRANCH,
+        branch=TESTDATA_BRANCH,
     ) as ds:
         tn10 = calendar.percentile_doy(ds.tasmin, per=10)
         t10 = calendar.percentile_doy(ds.tas, per=10)
@@ -60,7 +60,7 @@ def generate_atmos(cache_dir: Path):
 
 def populate_testing_data(
     temp_folder: Path | None = None,
-    branch: str = MAIN_TESTDATA_BRANCH,
+    branch: str = TESTDATA_BRANCH,
     _local_cache: Path = _default_cache_dir,
 ):
     """Perform calls to GitHub for the relevant testing data."""
@@ -112,8 +112,6 @@ def add_example_file_paths(cache_dir: Path) -> dict[str]:
     ns["path_to_tasmax_file"] = "NRCANdaily/nrcan_canada_daily_tasmax_1990.nc"
     ns["path_to_tasmin_file"] = "NRCANdaily/nrcan_canada_daily_tasmin_1990.nc"
     ns["path_to_tas_file"] = "ERA5/daily_surface_cancities_1990-1993.nc"
-    ns["path_to_multi_shape_file"] = str(TD / "multi_regions.json")
-    ns["path_to_shape_file"] = str(TD / "southern_qc_geojson.json")
     ns["path_to_ensemble_file"] = "EnsembleReduce/TestEnsReduceCriteria.nc"
 
     # For core.utils.load_module example
@@ -151,9 +149,7 @@ def add_example_file_paths(cache_dir: Path) -> dict[str]:
     atmos_file = cache_dir.joinpath("atmosds.nc")
 
     # Give access to dataset variables by name in xdoctest namespace
-    with _open_dataset(
-        atmos_file, branch=MAIN_TESTDATA_BRANCH, cache_dir=cache_dir
-    ) as ds:
+    with _open_dataset(atmos_file, branch=TESTDATA_BRANCH, cache_dir=cache_dir) as ds:
         for variable in ds.data_vars:
             ns[f"{variable}_dataset"] = ds.get(variable)