Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ENH] dependency and diff test switch for individual estimators to decorate non-suite tests #5084

Merged
merged 44 commits into from Aug 13, 2023

Conversation

fkiraly
Copy link
Collaborator

@fkiraly fkiraly commented Aug 12, 2023

This PR introduces a test switch for individual estimators that can be used to decorate non-suite pytest tests.

The test switch reads the pytest parameters, and if used in a skipif decorator will run the test iff all soft dependencies are present, and if the estimator module has changed (if the ONLY_CHANGED_MODULES switch is on).

This should allow us to minimize test time, by gradually removing all estimator specific costly tests using this decorator.

As a proof of concept, this PR decorates the most costly non-suite tests in the code base (mostly various heavy time series classifier ensembles and such) to be executed only upon module change.

Includes the following changes:

  • also checks out main on the test-nosoftdeps-full CI element so it can compute a non-trivial diff to main
  • moves the runtime intensive ConformalIntervals doctest into separate pytest tests to allow it to be decorated
  • adds missing "numba" dependency tag to the HIVE-COTE classifiers

As both the module of ConformalIntervals and the HIVE-COTE classifiers change (see above), we can consider these test cases, as we would expect the specific non-suite tests for the respective estimators to run in this CI element, while we would not expect the other decorated tests to run.

@fkiraly fkiraly added maintenance Continuous integration, unit testing & package distribution module:tests test framework functionality - only framework, excl specific tests enhancement Adding new functionality labels Aug 12, 2023
yarnabrina
yarnabrina previously approved these changes Aug 13, 2023
Copy link
Collaborator

@yarnabrina yarnabrina left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added 2 minor non-blocking suggestions, and skipped conformal testing.

sktime/forecasting/tests/test_conformal.py Outdated Show resolved Hide resolved
sktime/tests/test_switch.py Show resolved Hide resolved
sktime/tests/test_switch.py Outdated Show resolved Hide resolved
fkiraly and others added 3 commits August 13, 2023 16:50
Co-authored-by: Anirban Ray <39331844+yarnabrina@users.noreply.github.com>
@fkiraly fkiraly merged commit ac7a0cf into main Aug 13, 2023
24 checks passed
@fkiraly fkiraly deleted the test-switch branch August 13, 2023 18:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Adding new functionality maintenance Continuous integration, unit testing & package distribution module:tests test framework functionality - only framework, excl specific tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants