Streamline local and remote development testing with tox/ GitHub CI

[Zeitsperre]

Addressing a Problem?

There are three issues at hand here:

• Presently, local development and CI tests are largely identical workflows thanks to tools like tox. While this reduces the need to update our development steps considerably (documentation and CI stages), there are instances where this is not well suited to local development. One such issue is the use of pytest-cov, which is helpful for CI and noisy for local developers.

• In order to speed up testing, in a recent release we adopted pytest-xdist as a means of distributing tests between processes. GitHub CI offers Linux VMs with two cores, so this effectively halves testing time needed (which is great!). This becomes an issue when running tests for individual tests locally, as regardless of the number of tests being run, workers need to be spun-up to meet this configuration (automatic distribution; 6 workers maximum). To add insult to injury, importing xclim remains something of a lengthy process (several seconds), despite advancements that have sped this up considerably.

• For us to maintain and ensure that the test are being run with the most recent testing data, the test data is synchronized with the main branch of xclim-testdata at every call of pytest by default. This adds a lot of delay to the running of tests as these files are always verified at every run.

Potential Solution

Most of these configurations work very well for our CI configurations. With some tweaking, there are some ways to further streamline this but we seem to have a workable configuration for our test ensemble.

We need to define a set of configurations for local development, and this will neccessitate making some decisions:

• How do we want to enable distributed testing (pytest-xdist) when running tests locally?
  • Where should distributed testing configurations be housed?
• What logic/checks should we use to determine when we should perform a synchronization of the local testdata with remote?
  • Can we think of a better approach to gathering the testdata across workers?

Some approaches could be:

• Only run distributed testing configuration when the entirety of the test ensemble is run (i.e. when running tests with tox, or when running on CI).
• Using a different approach to copying over test data between workers (see: https://pytest-xdist.readthedocs.io/en/latest/how-to.html#specifying-rsync-dirs-in-an-ini-file)
• Only synchronizing the testing data with remote if the folder is not present.

Perhaps it would be good to define a an ideal set of configs for each use-case and seeing how best we can accomodate them?

• CI Testing
• Local Testing (full suite of tests)
• Local Testing (individual tests)

Additional context

No response

Contribution

• I would be willing/able to open a Pull Request to contribute this feature.

Code of Conduct

• I agree to follow this project's Code of Conduct