Move component importing to runtime and include test to check for warnings #1045
Conversation
| def test_default_parameters(cls): | ||
|
|
||
| assert cls.default_parameters == cls().parameters, f"{cls.__name__}'s default parameters don't match __init__." | ||
|
|
||
|
|
||
| @pytest.mark.parametrize("cls", all_components()) | ||
| def test_default_parameters_raise_no_warnings(cls): | ||
| with warnings.catch_warnings(record=True) as w: |
There was a problem hiding this comment.
This test catches warnings with record=True and asserts none have been raised.
Codecov Report
@@ Coverage Diff @@
## main #1045 +/- ##
==========================================
+ Coverage 94.31% 99.91% +5.59%
==========================================
Files 183 183
Lines 10156 10167 +11
==========================================
+ Hits 9579 10158 +579
+ Misses 577 9 -568
Continue to review full report at Codecov.
|
| _all_transformers = get_importable_subclasses(Transformer, used_in_automl=False) | ||
| all_components = _all_estimators + _all_transformers | ||
|
|
||
| def _all_estimators(): |
There was a problem hiding this comment.
@jeremyliweishih If the solution is to make sure that components don't issue warnings when they are initialized with default arguments, do we need to change these to be functions instead of just lists?
There was a problem hiding this comment.
@freddyaboulton The idea with changing these lists to methods is so that even if warnings or errors occur they do so on runtime instead of import-time which I think is a better design choice as it is less confusing for users (plus the warnings or errors only generate at where they're called in the docs instead of whenever evalml is imported).
There was a problem hiding this comment.
I see but my hypothesis is that if we make sure components don't issue warnings on init we wouldn't see warnings regardless of whether _all_estimators is a function or a list?
There was a problem hiding this comment.
@freddyaboulton got it and I definitely agree with it! If we do make sure warnings do not surface (which should happen with the test) it might be better just to keep the lists the same to reduce change. What do you think @dsherry?
There was a problem hiding this comment.
Great discussion!
@freddyaboulton I think its dangerous for us to continue instantiating components at import-time. If we ever add a component in the future which prints out a warning message by default, or has a stack trace, we can't even import the library! This way, import always works, even if an individual component doesn't. It's a future-proofing safety change.
We can have a separate discussion about how to validate component definitions. In fact that's tracked by #513
There was a problem hiding this comment.
@dsherry I see your point!
jeremyliweishih
requested review from
dsherry,
angela97lin,
eccabay and
freddyaboulton
…1017_text_warning
…/evalml into js_1017_text_warning
There was a problem hiding this comment.
Looks good! Thanks for making the change @jeremyliweishih !
| def test_default_parameters(cls): | ||
|
|
||
| assert cls.default_parameters == cls().parameters, f"{cls.__name__}'s default parameters don't match __init__." | ||
|
|
||
|
|
||
| @pytest.mark.parametrize("cls", all_components()) | ||
| def test_default_parameters_raise_no_warnings(cls): | ||
| with warnings.catch_warnings(record=True) as w: |
| def test_default_parameters(cls): | ||
|
|
||
| assert cls.default_parameters == cls().parameters, f"{cls.__name__}'s default parameters don't match __init__." | ||
|
|
||
|
|
||
| @pytest.mark.parametrize("cls", all_components()) | ||
| def test_default_parameters_raise_no_warnings(cls): | ||
| with warnings.catch_warnings(record=True) as w: |
Fixes #1017. Docs here.