[MNT] Unit testing revamp part 1: `check_estimator` #1479

MatthewMiddlehurst · 2024-04-27T00:05:42Z

Reference Issues/PRs

What does this implement/fix? Explain your changes.

Updates the check_estimator method to use a scikit-learn like method for generating estimator tests rather than the current pytest fixtures. The old method is still available for now in check_estimator_legacy. parametrize_with_checks simply outputs the generated checks as parameters for a pytest function.

_yield_estimator_checks.py ports tests over from test_all_estimators. The data used and the calling of estimator methods (i.e. fit and predict) has to change due to the detachment from the "scenarios" framework used previously.

Quick run down of contents:

Has some small changes to base classes mostly regarding meta-data, as the storing of it in class attributes breaks the condition of no changes in predict/transform.
Implements parametrize_with_checks and check_estimator for non-deprecated interfaces. These are only run on test estimators for now.
Implements the current BaseEstimator tests as checks for the above without the "scenario" framework.

aeon-actions-bot · 2024-04-27T00:06:05Z

Thank you for contributing to `aeon`

I would have added the following labels to this PR based on the changes made: [ $\color{#2C2F20}{\textsf{testing}}$ ], however some package labels are already present.

The Checks tab will show the status of our automated tests. You can click on individual test runs in the tab or "Details" in the panel below to see more information if there is a failure.

If our pre-commit code quality check fails, any trivial fixes will automatically be pushed to your PR unless it is a draft.

Don't hesitate to ask questions on the aeon Slack channel if you have any.

…t/aeon into mm/check-estimator

…heck-estimator

…heck-estimator # Conflicts: # aeon/testing/test_softdeps.py

…heck-estimator # Conflicts: # aeon/testing/estimator_checks/_legacy/_legacy_estimator_checks.py # aeon/utils/tests/test_check_estimator.py

baraline

Had a quick look and I must say that the testing pipeline seems way clearer to me with this format, from testing data to which test are run for which estimator.

I'll delve a bit more into it tomorrow, just some typos in found while reading through it.

aeon/base/_base_collection.py

aeon/base/_base.py

aeon/testing/estimator_checking/_estimator_checks.py

MatthewMiddlehurst · 2024-07-02T22:06:59Z

Actually, with forecasting more officially deprecated now, I can probably remove the legacy check function. Can keep for now if people disagree.

…heck-estimator # Conflicts: # docs/api_reference/utils.rst

TonyBagnall · 2024-07-03T12:58:05Z

Actually, with forecasting more officially deprecated now, I can probably remove the legacy check function. Can keep for now if people disagree.

I vote remove it

aeon/testing/estimator_checking/_estimator_checks.py

aeon/transformations/series/base.py

…heck-estimator

TonyBagnall

LGTM, onwards!

* [BUG] fix padder fit_values (aeon-toolkit#1752) * fix padder * declare var * [DOC,MNT] Documentation updates and deprecation warning for `v1.0.0` (aeon-toolkit#1743) * forecasting docs * v1.0.0 mass deprecation message * index fix * installation * typos * comment * pushing to forks is crazy * dont create branches * [MNT] Unit testing revamp part 1: `check_estimator` (aeon-toolkit#1479) * remove no soft deps job and expand full no soft deps job * no PR testing flag * module soft deps * is this actually returning an empty list * import modules now * clear out module soft dep * check_estimator * check_estimator updates * import * more complete functions and tests * fix * coverage and paramterize function test * testing data start * testing data start * testing data cont * real partial * cont * testing data changes * examples revert * fourier features * Revert "fourier features" This reverts commit 567c5e1. * legacy skip * init * move to legacy * example data update * testing data * imports * examples * testing update * multiindex * multiindex fixes * multiindex returns * test * test * docs, tests and fixes * comments * last bits * refactor testing folder * imports * series generators * fixes * make_series legacy * merge * forecasting functions * fixes * fixes * fixes * testing * examples * convert checks * fixes * docs * ad datasets * more docs * forecasting compose * forecasting * delete forecasting api docs * docs * revert and comment init * imports * import test * uncomment * docstrings * no imports * package name * import * docstring * fixes * remove legacy and doc fix * [ENH] Channel selection rocket wrapper (aeon-toolkit#1504) * base channel selector * base channel selector * random channel selector * elbow class test * random channel selector * base channel selector * base channel selector * revert registry * draft channel selector * class method * test channel scorer * docstring * remove unnecessary ExponentTransform import * deprecate DateTime * Revert "deprecate DateTime" This reverts commit 98b29e6. * channel selection * remove dependency * move tests to fit * move tests to fit * channel scorer test * add to API * maint * maint --------- Co-authored-by: Tony Bagnall <ajb@uea.ac.uk>

* [BUG] fix padder fit_values (aeon-toolkit#1752) * fix padder * declare var * [DOC,MNT] Documentation updates and deprecation warning for `v1.0.0` (aeon-toolkit#1743) * forecasting docs * v1.0.0 mass deprecation message * index fix * installation * typos * [MNT] Unit testing revamp part 1: `check_estimator` (aeon-toolkit#1479) * remove no soft deps job and expand full no soft deps job * no PR testing flag * module soft deps * is this actually returning an empty list * import modules now * clear out module soft dep * check_estimator * check_estimator updates * import * more complete functions and tests * fix * coverage and paramterize function test * testing data start * testing data start * testing data cont * real partial * cont * testing data changes * examples revert * fourier features * Revert "fourier features" This reverts commit 567c5e1. * legacy skip * init * move to legacy * example data update * testing data * imports * examples * testing update * multiindex * multiindex fixes * multiindex returns * test * test * docs, tests and fixes * comments * last bits * refactor testing folder * imports * series generators * fixes * make_series legacy * merge * forecasting functions * fixes * fixes * fixes * testing * examples * convert checks * fixes * docs * ad datasets * more docs * forecasting compose * forecasting * delete forecasting api docs * docs * revert and comment init * imports * import test * uncomment * docstrings * no imports * package name * import * docstring * fixes * remove legacy and doc fix * [ENH] Channel selection rocket wrapper (aeon-toolkit#1504) * base channel selector * base channel selector * random channel selector * elbow class test * random channel selector * base channel selector * base channel selector * revert registry * draft channel selector * class method * test channel scorer * docstring * remove unnecessary ExponentTransform import * deprecate DateTime * Revert "deprecate DateTime" This reverts commit 98b29e6. * channel selection * remove dependency * move tests to fit * move tests to fit * channel scorer test * add to API * [DEP] Exclusion list (aeon-toolkit#1730) * remove dependency * excluded list * [MNT] Removing `tensorflow_addons` dependency (aeon-toolkit#1421) * tensorflow bound * add manually instance normalization * empty commit * fix doxs * fix bug * add maintainer * re arrange * dep on utils * fix comments * add version and file to header * re add conflict * remove addon * only adodn remove * add typeguard dep * remove addons from test * remove tag of python<3.12 from base class * edit pyptoject for test details * remove typeguard * remove addons * remove typeguard * fix bug in python version * remove config on python < 3.12 * remove python version from test * remove python version from base * use group norm * re-add 3.12 limit * remove tags and add python version * skip random state clr test * fix test all networks * set y for channel selection test * set y for channel selection test --------- Co-authored-by: hadifawaz1999 <hadifawaz2291999@gmail.com> Co-authored-by: Tony Bagnall <ajb@uea.ac.uk> * comment * Empty commit for CI * uncomment --------- Co-authored-by: Tony Bagnall <ajb@uea.ac.uk> Co-authored-by: hadifawaz1999 <hadifawaz2291999@gmail.com> Co-authored-by: MatthewMiddlehurst <MatthewMiddlehurst@users.noreply.github.com>

…ting CI (#1764) * [MNT] Bot to make empty commits (#15) * CI fun * comments and token * [MNT] empty commit PR 2 (#16) * CI fun * comments and token * comments * skip dirty check * Empty commit for CI * empty comment 3 (#17) * comment * prs only * comment * Empty commit for CI * empty commit 4 (#18) * comment * correct branch * Empty commit for CI * empty commit 5 (#19) * comment * branch * empty commit 6 (#20) * [BUG] fix padder fit_values (#1752) * fix padder * declare var * [DOC,MNT] Documentation updates and deprecation warning for `v1.0.0` (#1743) * forecasting docs * v1.0.0 mass deprecation message * index fix * installation * typos * comment * pushing to forks is crazy * dont create branches * [MNT] Unit testing revamp part 1: `check_estimator` (#1479) * remove no soft deps job and expand full no soft deps job * no PR testing flag * module soft deps * is this actually returning an empty list * import modules now * clear out module soft dep * check_estimator * check_estimator updates * import * more complete functions and tests * fix * coverage and paramterize function test * testing data start * testing data start * testing data cont * real partial * cont * testing data changes * examples revert * fourier features * Revert "fourier features" This reverts commit 567c5e1. * legacy skip * init * move to legacy * example data update * testing data * imports * examples * testing update * multiindex * multiindex fixes * multiindex returns * test * test * docs, tests and fixes * comments * last bits * refactor testing folder * imports * series generators * fixes * make_series legacy * merge * forecasting functions * fixes * fixes * fixes * testing * examples * convert checks * fixes * docs * ad datasets * more docs * forecasting compose * forecasting * delete forecasting api docs * docs * revert and comment init * imports * import test * uncomment * docstrings * no imports * package name * import * docstring * fixes * remove legacy and doc fix * [ENH] Channel selection rocket wrapper (#1504) * base channel selector * base channel selector * random channel selector * elbow class test * random channel selector * base channel selector * base channel selector * revert registry * draft channel selector * class method * test channel scorer * docstring * remove unnecessary ExponentTransform import * deprecate DateTime * Revert "deprecate DateTime" This reverts commit 98b29e6. * channel selection * remove dependency * move tests to fit * move tests to fit * channel scorer test * add to API * maint * maint --------- Co-authored-by: Tony Bagnall <ajb@uea.ac.uk> * empty commit 7 (#21) * comment * remove branch param * fixes * Empty commit for CI * empty commit 8 (#22) * comment * repo and branch * comment * empty commit 9 (#23) * [BUG] fix padder fit_values (#1752) * fix padder * declare var * [DOC,MNT] Documentation updates and deprecation warning for `v1.0.0` (#1743) * forecasting docs * v1.0.0 mass deprecation message * index fix * installation * typos * [MNT] Unit testing revamp part 1: `check_estimator` (#1479) * remove no soft deps job and expand full no soft deps job * no PR testing flag * module soft deps * is this actually returning an empty list * import modules now * clear out module soft dep * check_estimator * check_estimator updates * import * more complete functions and tests * fix * coverage and paramterize function test * testing data start * testing data start * testing data cont * real partial * cont * testing data changes * examples revert * fourier features * Revert "fourier features" This reverts commit 567c5e1. * legacy skip * init * move to legacy * example data update * testing data * imports * examples * testing update * multiindex * multiindex fixes * multiindex returns * test * test * docs, tests and fixes * comments * last bits * refactor testing folder * imports * series generators * fixes * make_series legacy * merge * forecasting functions * fixes * fixes * fixes * testing * examples * convert checks * fixes * docs * ad datasets * more docs * forecasting compose * forecasting * delete forecasting api docs * docs * revert and comment init * imports * import test * uncomment * docstrings * no imports * package name * import * docstring * fixes * remove legacy and doc fix * [ENH] Channel selection rocket wrapper (#1504) * base channel selector * base channel selector * random channel selector * elbow class test * random channel selector * base channel selector * base channel selector * revert registry * draft channel selector * class method * test channel scorer * docstring * remove unnecessary ExponentTransform import * deprecate DateTime * Revert "deprecate DateTime" This reverts commit 98b29e6. * channel selection * remove dependency * move tests to fit * move tests to fit * channel scorer test * add to API * [DEP] Exclusion list (#1730) * remove dependency * excluded list * [MNT] Removing `tensorflow_addons` dependency (#1421) * tensorflow bound * add manually instance normalization * empty commit * fix doxs * fix bug * add maintainer * re arrange * dep on utils * fix comments * add version and file to header * re add conflict * remove addon * only adodn remove * add typeguard dep * remove addons from test * remove tag of python<3.12 from base class * edit pyptoject for test details * remove typeguard * remove addons * remove typeguard * fix bug in python version * remove config on python < 3.12 * remove python version from test * remove python version from base * use group norm * re-add 3.12 limit * remove tags and add python version * skip random state clr test * fix test all networks * set y for channel selection test * set y for channel selection test --------- Co-authored-by: hadifawaz1999 <hadifawaz2291999@gmail.com> Co-authored-by: Tony Bagnall <ajb@uea.ac.uk> * comment * Empty commit for CI * uncomment --------- Co-authored-by: Tony Bagnall <ajb@uea.ac.uk> Co-authored-by: hadifawaz1999 <hadifawaz2291999@gmail.com> Co-authored-by: MatthewMiddlehurst <MatthewMiddlehurst@users.noreply.github.com> --------- Co-authored-by: MatthewMiddlehurst <MatthewMiddlehurst@users.noreply.github.com> Co-authored-by: Tony Bagnall <ajb@uea.ac.uk> Co-authored-by: hadifawaz1999 <hadifawaz2291999@gmail.com>

MatthewMiddlehurst added 7 commits April 26, 2024 13:11

remove no soft deps job and expand full no soft deps job

fb28b42

no PR testing flag

73fafb2

module soft deps

426f328

is this actually returning an empty list

da050ef

import modules now

83d0def

clear out module soft dep

d2f53b7

check_estimator

3c28040

MatthewMiddlehurst added maintenance Continuous integration, unit testing & package distribution testing Testing related issue or pull request labels Apr 27, 2024

MatthewMiddlehurst added 3 commits April 27, 2024 15:39

Merge branch 'mm/no-soft-dep-tests' of https://github.com/aeon-toolki…

14450c7

…t/aeon into mm/check-estimator

check_estimator updates

180b31f

import

a1d0523

MatthewMiddlehurst added the codecov actions Run the codecov action on a PR label Apr 28, 2024

MatthewMiddlehurst added 16 commits April 30, 2024 00:06

Merge branch 'main' of https://github.com/aeon-toolkit/aeon into mm/c…

23b73fd

…heck-estimator

more complete functions and tests

201fb4d

fix

0c22b26

Merge branch 'main' of https://github.com/aeon-toolkit/aeon into mm/c…

0cb7cba

…heck-estimator

coverage and paramterize function test

944987b

testing data start

7649e7f

testing data start

55043e3

Merge branch 'main' of https://github.com/aeon-toolkit/aeon into mm/c…

4bd26fe

…heck-estimator

testing data cont

6b875cf

Merge branch 'main' of https://github.com/aeon-toolkit/aeon into mm/c…

562a929

…heck-estimator # Conflicts: # aeon/testing/test_softdeps.py

real partial

3373fc3

Merge branch 'main' of https://github.com/aeon-toolkit/aeon into mm/c…

0a8a69c

…heck-estimator # Conflicts: # aeon/testing/estimator_checks/_legacy/_legacy_estimator_checks.py # aeon/utils/tests/test_check_estimator.py

cont

3a354b7

testing data changes

05a09a4

examples revert

b60fcec

fourier features

567c5e1

MatthewMiddlehurst added 5 commits July 2, 2024 17:44

uncomment

f2be7b2

docstrings

454d1f6

no imports

3be2b36

package name

d8dee60

import

835fc26

MatthewMiddlehurst marked this pull request as ready for review July 2, 2024 19:02

MatthewMiddlehurst requested review from aiwalter and CodeLionX as code owners July 2, 2024 19:02

MatthewMiddlehurst changed the title ~~[MNT] check_estimator~~ [MNT] Unit testing revamp part 1: check_estimator Jul 2, 2024

baraline reviewed Jul 2, 2024

View reviewed changes

aeon/base/_base_collection.py Show resolved Hide resolved

aeon/base/_base.py Show resolved Hide resolved

aeon/testing/estimator_checking/_estimator_checks.py Outdated Show resolved Hide resolved

MatthewMiddlehurst added 3 commits July 3, 2024 00:05

docstring

428e48b

fixes

e19e8d6

Merge branch 'main' of https://github.com/aeon-toolkit/aeon into mm/c…

c9c6cb9

…heck-estimator # Conflicts: # docs/api_reference/utils.rst

This was referenced Jul 3, 2024

[BUG] CollectionTransformers with fit_is_empty is true #1750

Closed

[BUG] Move reset #1751

Closed

TonyBagnall reviewed Jul 3, 2024

View reviewed changes

aeon/testing/estimator_checking/_estimator_checks.py Outdated Show resolved Hide resolved

TonyBagnall reviewed Jul 3, 2024

View reviewed changes

aeon/transformations/series/base.py Show resolved Hide resolved

MatthewMiddlehurst added 2 commits July 3, 2024 16:49

remove legacy and doc fix

938b722

Merge branch 'main' of https://github.com/aeon-toolkit/aeon into mm/c…

6b70b33

…heck-estimator

TonyBagnall approved these changes Jul 3, 2024

View reviewed changes

MatthewMiddlehurst requested a review from baraline July 3, 2024 18:46

baraline approved these changes Jul 4, 2024

View reviewed changes

MatthewMiddlehurst merged commit 0cf32d1 into main Jul 4, 2024
16 checks passed

MatthewMiddlehurst deleted the mm/check-estimator branch July 4, 2024 09:16

MatthewMiddlehurst mentioned this pull request Jul 15, 2024

[MNT] Unit testing revamp part 2: classification #1770

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MNT] Unit testing revamp part 1: `check_estimator` #1479

[MNT] Unit testing revamp part 1: `check_estimator` #1479

MatthewMiddlehurst commented Apr 27, 2024 •

edited

Loading

aeon-actions-bot bot commented Apr 27, 2024

baraline left a comment

MatthewMiddlehurst commented Jul 2, 2024

TonyBagnall commented Jul 3, 2024

TonyBagnall left a comment

[MNT] Unit testing revamp part 1: check_estimator #1479

[MNT] Unit testing revamp part 1: check_estimator #1479

Conversation

MatthewMiddlehurst commented Apr 27, 2024 • edited Loading

Reference Issues/PRs

What does this implement/fix? Explain your changes.

aeon-actions-bot bot commented Apr 27, 2024

Thank you for contributing to aeon

baraline left a comment

Choose a reason for hiding this comment

MatthewMiddlehurst commented Jul 2, 2024

TonyBagnall commented Jul 3, 2024

TonyBagnall left a comment

Choose a reason for hiding this comment

[MNT] Unit testing revamp part 1: `check_estimator` #1479

[MNT] Unit testing revamp part 1: `check_estimator` #1479

MatthewMiddlehurst commented Apr 27, 2024 •

edited

Loading

Thank you for contributing to `aeon`