Sklearn 1.1.1 Upgrade #3525

chukarsten · 2022-05-17T22:07:55Z

Upgrades EvalML to scikit-learn 1.1.1!

codecov · 2022-05-17T22:13:17Z

Codecov Report

Merging #3525 (291f2bd) into main (d6462a8) will increase coverage by 0.1%.
The diff coverage is 100.0%.

@@           Coverage Diff           @@
##            main   #3525     +/-   ##
=======================================
+ Coverage   99.7%   99.7%   +0.1%     
=======================================
  Files        335     335             
  Lines      33260   33350     +90     
=======================================
+ Hits       33131   33221     +90     
  Misses       129     129

Impacted Files	Coverage Δ
evalml/pipelines/component_graph.py	`99.8% <ø> (ø)`
evalml/tests/component_tests/test_components.py	`99.1% <ø> (ø)`
...ation_pipeline_tests/test_binary_classification.py	`100.0% <ø> (ø)`
...n_pipeline_tests/test_multiclass_classification.py	`100.0% <ø> (ø)`
...tests/regression_pipeline_tests/test_regression.py	`100.0% <ø> (ø)`
...valml/tests/pipeline_tests/test_component_graph.py	`99.9% <ø> (ø)`
evalml/tests/pipeline_tests/test_pipeline_utils.py	`99.8% <ø> (ø)`
evalml/tests/pipeline_tests/test_pipelines.py	`99.8% <ø> (ø)`
...elines/components/transformers/imputers/imputer.py	`100.0% <100.0%> (ø)`
...components/transformers/imputers/simple_imputer.py	`100.0% <100.0%> (ø)`
... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d6462a8...291f2bd. Read the comment docs.

chukarsten · 2022-06-06T23:33:15Z

evalml/tests/model_understanding_tests/test_metrics.py

@@ -216,9 +216,9 @@ def test_precision_recall_curve(data_type, make_data_type):
    recall = precision_recall_curve_data.get("recall")


All changes in these files result from this sklearn PR: scikit-learn/scikit-learn#23214.

This change was also merged into scikit-learn 1.1.1, the post 1.1.0 fix release and is partially why we have to move the lower limit of sklearn to 1.1.1 instead of 1.1.0.

evalml/tests/demo_tests/test_datasets.py

chukarsten · 2022-06-09T23:25:42Z

docs/source/user_guide/objectives.ipynb

@@ -72,7 +72,7 @@
    "X.ww.init(logical_types={\"provider\": \"Categorical\", \"region\": \"Categorical\",\n",
    "                         \"currency\": \"Categorical\", \"expiration_date\": \"Categorical\"})\n",
    "objective = F1()\n",
-    "pipeline = BinaryClassificationPipeline(component_graph=['Simple Imputer', 'DateTime Featurizer', 'One Hot Encoder', 'Random Forest Classifier'])\n",
+    "pipeline = BinaryClassificationPipeline(component_graph=['Imputer', 'DateTime Featurizer', 'One Hot Encoder', 'Random Forest Classifier'])\n",


Had to change this as the SimpleImputer now struggles handling both boolean and categorical data within the same dataframe. Made the choice to follow the paradigm of the Imputer spinning up SimpleImputers for numeric, categorical and boolean data separately.

Makes sense

chukarsten · 2022-06-09T23:29:49Z

evalml/pipelines/components/transformers/imputers/imputer.py

        random_seed=0,
        **kwargs,
    ):
        if categorical_impute_strategy not in self._valid_categorical_impute_strategies:
            raise ValueError(
                f"{categorical_impute_strategy} is an invalid parameter. Valid categorical impute strategies are {', '.join(self._valid_numeric_impute_strategies)}"
            )
-        elif numeric_impute_strategy not in self._valid_numeric_impute_strategies:


Seems like it's possible for neither, both, or either of these strategies to be invalid, so removed elif.

chukarsten · 2022-06-09T23:31:30Z

evalml/pipelines/components/transformers/imputers/imputer.py

        return self

    def transform(self, X, y=None):
-        """Transforms data X by imputing missing values. 'None' values are converted to np.nan before imputation and are treated as the same.


Removed this because I don't think this is true anymore.

chukarsten · 2022-06-10T17:01:53Z

evalml/tests/component_tests/test_imputer.py

@@ -458,28 +573,35 @@ def test_imputer_int_preserved():
    assert {k: type(v) for k, v in transformed.ww.logical_types.items()} == {0: Integer}


-def test_imputer_bool_preserved():
-    X = pd.DataFrame(pd.Series([True, False, True, np.nan] * 4))
+@pytest.mark.parametrize("null_type", ["pandas_na", "numpy_nan", "python_none"])


Expanded coverage a bit and made it so the failed tests are more obvious.

chukarsten · 2022-06-10T17:03:00Z

evalml/tests/component_tests/test_simple_imputer.py

-        {"int with None": [1, 0, 5, 2], "float with None": [0.1, 0.0, 0.5, 0.2]}
-    )
-    assert_frame_equal(expected, transformed, check_dtype=False)
+X = pd.DataFrame(


chukarsten · 2022-06-10T17:03:47Z

evalml/tests/component_tests/test_simple_imputer.py

+        ["categorical col", "float col"],
+    ],
+)
+def test_simple_imputer_errors_with_bool_and_categorical_columns(


Added this to cover the SimpleImputer moving over to not covering mixed cat/bool cols.

chukarsten · 2022-06-10T17:05:18Z

evalml/tests/utils_tests/test_gen_utils.py

@@ -830,3 +831,17 @@ def test_year_start_separated_by_gap():
    assert are_datasets_separated_by_gap_time_index(
        train, test, {"time_index": "time_index", "gap": 2}
    )
+
+
+def test_is_categorical_actually_boolean():


Remove this when #3556 is added.

chukarsten · 2022-06-10T17:05:37Z

evalml/utils/gen_utils.py

@@ -49,6 +49,25 @@ def import_or_raise(library, error_msg=None, warning=False):
            raise Exception(msg)


+def is_categorical_actually_boolean(df, df_col):


Remove this when #3556 is added.

freddyaboulton · 2022-06-10T18:05:30Z

core-requirements.txt

@@ -1,8 +1,8 @@
 numpy>=1.21.0
 pandas>=1.4.0
 scipy>=1.5.0
-scikit-learn>=0.24.0,<1.1.0
-scikit-optimize>=0.8.1


Shouldn't this change (scikit-optimize) also be reflected in meta.yaml?

freddyaboulton · 2022-06-10T18:38:58Z

@chukarsten Thank you for putting this together. Might be worth running the release perf tests to see if we're getting some bugs?

freddyaboulton

@chukarsten Thank you for this! There are some small comments regarding target imputer and the requirements files but other than that I think this is good to go.

docs/source/release_notes.rst

freddyaboulton · 2022-06-13T15:06:41Z

evalml/pipelines/components/transformers/imputers/target_imputer.py

@@ -120,6 +121,14 @@ def transform(self, X, y):

        transformed = self._component_obj.transform(y_df)
        y_t = pd.Series(transformed[:, 0], index=y_ww.index)
+
+        # TODO: Fix this after WW adds inference of object type booleans to BooleanNullable


Do we need this? This changes output type from categorical to boolean but having the output type be categorical until the ww + nullable type work is ok since the target will be the same for the model

evalml/utils/gen_utils.py

freddyaboulton · 2022-06-13T15:33:25Z

.github/meta.yaml

@@ -72,7 +72,7 @@ outputs:
        - lightgbm >=2.3.1
        - lime >=0.2.0.1
        - python >=3.8.*
-        - imbalanced-learn >=0.8.0


I think we also need to bump imbalanced learn in all the other files? And scitkit-optimize here?

freddyaboulton · 2022-06-13T15:45:04Z

evalml/pipelines/components/transformers/imputers/simple_imputer.py

+        imputer = SkImputer(
+            strategy=impute_strategy,
+            fill_value=fill_value,
+            missing_values=pd.NA,


Using np.nan here makes my local tests pass for the imputer/simple_imputer. Any idea what's up?

evalml/tests/component_tests/test_imputer.py

docs/source/release_notes.rst

evalml/pipelines/components/transformers/imputers/imputer.py

evalml/pipelines/components/transformers/imputers/simple_imputer.py

… BooleanNullable rather than categorical and removed the SimpleImputer's changing of BooleanNullable to categorical.

…emed there was some interaction between columns and generating the errors.

…xpected output of BooleanNullable to be BooleanNullable.

…nd updated the new separated test to include this.

…o the testing. Added some comments and removed the old imputer test.

…ying to call Masked Array which no longer exists in sklearn.utils.fixes.

…out some of the bug fixes introduced in 1.1.1.

…t-automl.

…instead of the Imputer on data that had both categorical and boolean data in the same frame. Added a test to catch this in the SimpleImputer.

…ly Boolean.

….py.

Removed ipykernel

gsheni changed the title ~~Sklearn 11 upgrade~~ Sklearn 1.1 upgrade May 17, 2022

chukarsten force-pushed the sklearn_11_upgrade branch 3 times, most recently from 4c7e676 to 3523f29 Compare June 6, 2022 21:31

chukarsten changed the title ~~Sklearn 1.1 upgrade~~ Sklearn 1.1.1 Upgrade Jun 7, 2022

chukarsten force-pushed the sklearn_11_upgrade branch from 9f89b0c to 3dba63e Compare June 9, 2022 22:00

chukarsten marked this pull request as ready for review June 9, 2022 23:23

auto-assign bot assigned chukarsten Jun 9, 2022

chukarsten requested review from eccabay, ParthivNaresh, freddyaboulton, gsheni, christopherbunn, dsherry, fjlanasa and jeremyliweishih June 10, 2022 16:44

chukarsten commented Jun 10, 2022

View reviewed changes

freddyaboulton reviewed Jun 10, 2022

View reviewed changes

freddyaboulton approved these changes Jun 13, 2022

View reviewed changes

eccabay reviewed Jun 13, 2022

View reviewed changes

chukarsten added 8 commits June 17, 2022 12:21

Update to sklearn 1.1 in core-reqs.txt.

9d8299d

Modified some of the tests to set known boolean columns with nulls to…

329e1f3

… BooleanNullable rather than categorical and removed the SimpleImputer's changing of BooleanNullable to categorical.

EXPERIMENTAL: broke some of the simple imputer tests up because it se…

6a3ed22

…emed there was some interaction between columns and generating the errors.

Changed the SimpleImputer to use pd.NA as the missing value and the e…

3dec178

…xpected output of BooleanNullable to be BooleanNullable.

EXPERIMENTAL: Consolidated some of the tests into a parameterized one.

bd2c72c

Rewrote the Imputer to split apart boolean from categorical columns a…

e1d9d78

…nd updated the new separated test to include this.

Got the full imputer test working and added the median numeric mode t…

d43cda5

…o the testing. Added some comments and removed the old imputer test.

Refactored the boolean preservation test.

9f5c2c8

chukarsten and others added 21 commits June 17, 2022 12:21

Fixed component graph doctest.

7c65f8e

Upgraded minimum of scikit-optimize to 0.9.0 to prevent skopt from tr…

071f00c

…ying to call Masked Array which no longer exists in sklearn.utils.fixes.

Forgot a comma in the component graph doctest.

1fb35ca

Raised the minimum version of sklearn to 1.1.1 as our tests fail with…

6ef0592

…out some of the bug fixes introduced in 1.1.1.

Bumped meta.yaml up to sklearn 1.1.1.

36472d9

Reverted accidental change to number of test runner cores for git-tes…

e9b4f04

…t-automl.

Added coverage for numeric_impute_strategy == constant

ea29de0

Add testing for the imputer's boolean strategy being invalid.

b4dba7b

Lint.

1bdf814

Fixed imputer test.

481a670

Fixed the docs where the SimpleImputer was invoked in the user_guide …

82bbe27

…instead of the Imputer on data that had both categorical and boolean data in the same frame. Added a test to catch this in the SimpleImputer.

Updated SimpleImputer to identify Categorical columns that are secret…

cdfa25a

…ly Boolean.

Moved functionality to identify nullable boolean columns to gen_utils…

a311c39

….py.

Lint: added a Raises field to SimpleImputer.

298bc65

Upped the min version of sklearn in setup.cfg.

14fa820

Properly added the exception raising in the doc string.

d86cea1

Update objectives.ipynb

3d1f47f

Removed ipykernel

Missed a refactor to use the is_categorical_actually_boolean function.

5d2c8fd

Addressed changes to test_imputer.py, gen_utils.py and release notes.

3ade137

Updated release notes per Becca's rec.

ff47751

Addressed Becca's concerns.

7ce31d7

chukarsten force-pushed the sklearn_11_upgrade branch from e83d082 to 7ce31d7 Compare June 17, 2022 18:48

chukarsten and others added 2 commits June 17, 2022 15:32

Fixed test_invalid_strategy_parameters

df30164

Merge branch 'main' into sklearn_11_upgrade

291f2bd

chukarsten merged commit acd13e2 into main Jun 21, 2022

chukarsten deleted the sklearn_11_upgrade branch June 21, 2022 03:37

This was referenced Jun 23, 2022

Release 0.54.0 #3586

Closed

Release 0.54.0 #3587

Merged

tamargrey mentioned this pull request Feb 15, 2023

Remove Nullable type logic from Imputer Components and Refactor #3999

Closed

tamargrey mentioned this pull request Mar 1, 2023

Refactor imputer components to remove unnecessary logic #4038

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sklearn 1.1.1 Upgrade #3525

Sklearn 1.1.1 Upgrade #3525

chukarsten commented May 17, 2022 •

edited

Loading

codecov bot commented May 17, 2022 •

edited

Loading

chukarsten Jun 6, 2022

chukarsten Jun 7, 2022

chukarsten Jun 9, 2022

freddyaboulton Jun 10, 2022

chukarsten Jun 9, 2022

chukarsten Jun 9, 2022

chukarsten Jun 10, 2022

chukarsten Jun 10, 2022

chukarsten Jun 10, 2022

chukarsten Jun 10, 2022

chukarsten Jun 10, 2022

freddyaboulton Jun 10, 2022

freddyaboulton commented Jun 10, 2022 •

edited

Loading

freddyaboulton left a comment

freddyaboulton Jun 13, 2022

freddyaboulton Jun 13, 2022

freddyaboulton Jun 13, 2022

		@@ -216,9 +216,9 @@ def test_precision_recall_curve(data_type, make_data_type):
		recall = precision_recall_curve_data.get("recall")

		@@ -49,6 +49,25 @@ def import_or_raise(library, error_msg=None, warning=False):
		raise Exception(msg)


		def is_categorical_actually_boolean(df, df_col):

Sklearn 1.1.1 Upgrade #3525

Sklearn 1.1.1 Upgrade #3525

Conversation

chukarsten commented May 17, 2022 • edited Loading

codecov bot commented May 17, 2022 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

freddyaboulton commented Jun 10, 2022 • edited Loading

freddyaboulton left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chukarsten commented May 17, 2022 •

edited

Loading

codecov bot commented May 17, 2022 •

edited

Loading

freddyaboulton commented Jun 10, 2022 •

edited

Loading