Bugfix: sklearn 0.22 #571
Bugfix: sklearn 0.22 #571
Conversation
Very unnecessary :/
`check_is_fitted` was changed (see #570) in a way that we won't raise a `NotInitializedError` anymore despite the net not being initialized. This is now solved by basically porting the old `check_is_fitted` behavior to skorch.
sklearn's (_)safe_indexing used to work with tuples as indices, now doesn't.
In sklearn 0.22, make_scorer returns a special object that must be treated differently in skorch.
|
I'm not so happy about these breaking changes being made by sklearn, since they affect public API and, at least from skimming, are not announced in the changelog. |
|
The util module has been in this gray area regarding changing behavior, as stated in https://scikit-learn.org/stable/developers/utilities.html under “warning”. Historically this policy was put into place, because public functions were put into utils without throughly testing its functionality. I see how these types of changes can be a bit painful. Going forward, if sklearn wants to better support an ecosystem of estimators, I think sklearn needs to consider some of its utils as “first class” (this means deprecations) and not this gray area. I’ll be sure to bring this up with the rest of the team. |
Interesting. I didn't know that. sklearn is a huge codebase, so these things are bound to happen. I think it's especially important to communicate these kind of changes in the release notes.
Great, thanks! |
| if ( | ||
| hasattr(module, 'startswith') and | ||
| module.startswith('sklearn.metrics.') and | ||
| not module.startswith('sklearn.metrics.scorer') and | ||
| not module.startswith('sklearn.metrics.tests.') | ||
| not module.startswith('sklearn.metrics.tests.') and | ||
| not scoring.__class__.__name__ in scorer_names |
There was a problem hiding this comment.
Is the scoring.__class__.__name__ check needed? make_scorer returned these objects in 0.21 as well.
There was a problem hiding this comment.
I don't quite understand your suggestion. We need this new line to to check if we get any of these objects. I tried to remove the new line and get failing tests with sklearn 0.22 but not 0.21.
Currently, skorch won't work with sklearn 0.22 because some code was changed that is more geared towards sklearn-internal use, which we, however, relied upon.
A list of failures:
check_is_fittedwas changed (see Deperecated check_is_fitted call #570) in a way that we won't raise aNotInitializedErroranymore despite the net not being initialized. This is now solved by basically porting the oldcheck_is_fittedbehavior to skorch.safe_indexingwas changed (probably here) in a way that it no longer works with tuples as indices; the following code works in sklearn 0.21 but not in 0.22:This is solved by not using
safe_indexingin case of tuples anymore.convert_sklearn_metric_functionnow fails withmake_scorer(...)because that returns an sklearn_PredictScorerobject (or similar), which we don't detect. This is now detected.The changes were tested with scikit-learn=0.21.3 and scikit-learn==0.22 (no GPU).
Furthermore, pylint now complains about imports that are not at top level (basically about all our tests), which is why I blankly removed that warning.