Update dependencies to fix `ImportError: cannot import name 'MaskedArray' from 'sklearn.utils.fixes'` errors #1540

angela97lin · 2020-12-10T02:39:33Z

Repro steps:

Create new virtualenv with no packages installed. I used python 3.7.6
Install the following:
numpy>=1.19.1
pandas>1.1.0
scikit-learn==0.23
scikit-optimize==0.8
Run pip install -r requirements.txt and install pytest
Run tests

This should raise the following:

================================================================= ERRORS =================================================================
_____________________________________________________ ERROR collecting test session ______________________________________________________
../../.pyenv/versions/3.7.6/lib/python3.7/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1006: in _gcd_import
    ???
<frozen importlib._bootstrap>:983: in _find_and_load
    ???
<frozen importlib._bootstrap>:953: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:219: in _call_with_frames_removed
    ???
<frozen importlib._bootstrap>:1006: in _gcd_import
    ???
<frozen importlib._bootstrap>:983: in _find_and_load
    ???
<frozen importlib._bootstrap>:953: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:219: in _call_with_frames_removed
    ???
<frozen importlib._bootstrap>:1006: in _gcd_import
    ???
<frozen importlib._bootstrap>:983: in _find_and_load
    ???
<frozen importlib._bootstrap>:967: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:677: in _load_unlocked
    ???
<frozen importlib._bootstrap_external>:728: in exec_module
    ???
<frozen importlib._bootstrap>:219: in _call_with_frames_removed
    ???
evalml/__init__.py:9: in <module>
    import evalml.pipelines
evalml/pipelines/__init__.py:1: in <module>
    from .components import (
evalml/pipelines/components/__init__.py:2: in <module>
    from .estimators import (
evalml/pipelines/components/estimators/__init__.py:2: in <module>
    from .classifiers import (LogisticRegressionClassifier,
evalml/pipelines/components/estimators/classifiers/__init__.py:1: in <module>
    from .logistic_regression import LogisticRegressionClassifier
evalml/pipelines/components/estimators/classifiers/logistic_regression.py:3: in <module>
    from skopt.space import Real
../../.pyenv/versions/3.7.6/lib/python3.7/site-packages/skopt/__init__.py:55: in <module>
    from .searchcv import BayesSearchCV
../../.pyenv/versions/3.7.6/lib/python3.7/site-packages/skopt/searchcv.py:16: in <module>
    from sklearn.utils.fixes import MaskedArray
E   ImportError: cannot import name 'MaskedArray' from 'sklearn.utils.fixes' (/Users/angela.lin/.pyenv/versions/3.7.6/lib/python3.7/site-packages/sklearn/utils/fixes.py)
======================================================== short test summary info =========================================================
ERROR  - ImportError: cannot import name 'MaskedArray' from 'sklearn.utils.fixes' (/Users/angela.lin/.pyenv/versions/3.7.6/lib/python3....
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================================ 1 error in 3.06s ============================================================

Upgrading scikit-learn==0.23.1 and scikit-optimize==0.8.1 fixes this :)

--

This PR also tracks updating numpy and pandas versions. Since we rely on woodwork, we'll get errors (see issue) with our current minimal versions.

codecov · 2020-12-10T02:46:43Z

Codecov Report

Merging #1540 (6bc73da) into main (d77e854) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1540   +/-   ##
=======================================
  Coverage   100.0%   100.0%           
=======================================
  Files         232      232           
  Lines       16639    16639           
=======================================
  Hits        16631    16631           
  Misses          8        8

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d77e854...6bc73da. Read the comment docs.

jeremyliweishih

I think this looks good! Adding a warning to our release notes would be helpful especially since we're jumping quite a bit on our min versions. Maybe just like how we do breaking changes.

angela97lin · 2020-12-10T16:51:56Z

core-requirements.txt

+numpy>=1.19.1
+pandas>=1.1.0


Woodwork/Featuretool minimal requirements

freddyaboulton

Thanks for fixing this @angela97lin !

core-requirements.txt

bchen1116

LGTM! Agree with Freddy, if numpy and pandas are already on WW/FT requirements, do we need them on our requirements.txt as well?

dsherry

@angela97lin looks good! I left some questions. Could we try not updating the min numpy version and see what happens?

I'm also curious why pandas 1.1.0 is required for woodwork (?) -- let's follow up with them on that. I wonder if they can lower the min pandas version. The more permissive we can make these ranges, the more likely people will be to use our stuff heh (certainly at the risk of bugs)

core-requirements.txt

dsherry · 2020-12-11T23:20:43Z

core-requirements.txt

@@ -1,8 +1,8 @@
-numpy>=1.16.4
-pandas>=0.25.0
+numpy>=1.19.1


@angela97lin what happens if you keep this PR as-is, but delete this change so that older numpy versions are still allowed? Do the other changes still solve the problem, or is updating our min numpy also required?

@dsherry I posted in the original issue, but we get this error if we try to specifically install a version less than these because of Woodwork requirements:

ERROR: featuretools 0.22.0 has requirement pandas!=1.1.0,!=1.1.1,<2.0.0,>=0.24.1, but you'll have pandas 1.1.0 which is incompatible. ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. woodwork 0.0.6 requires numpy>=1.19.1, but you have numpy 1.16.4 which is incompatible. ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. woodwork 0.0.6 requires pandas>=1.1.0, but you have pandas 0.25.0 which is incompatible.

core-requirements.txt

dsherry · 2020-12-11T23:21:45Z

core-requirements.txt

 scipy>=1.2.1
-scikit-learn>=0.23
-scikit-optimize>=0.8
+scikit-learn>=0.23.1


@angela97lin so the problem was specifically in sklearn 0.23.0?

@angela97lin so the problem was specifically in sklearn 0.23.0?

Combination of scikit-learn 0.23 and scikit-optimize 0.8 and numpy: scikit-optimize/scikit-optimize#902

angela97lin · 2020-12-12T06:05:53Z

@dsherry Could be wrong but I don't think Woodwork can handle 0.25, because they rely so heavily on the new nullable structures (pd.NA, the new nullable float that is coming soon or was recently introduced in Pandas 1.X 🤔

dsherry

@angela97lin thanks for the details :) I filed an issue in woodwork to track lowering the numpy/pandas limits to match featuretools and evalml.

Yes I see that pd.NA was first added in pandas 1.0.0, so perhaps the pandas version needs to stay at something like pandas>=1.1.0. I still think the numpy version limit in woodwork is too high and that we can lower it.

Let's merge this PR, and once woodwork puts out a release with lower pip reqs for numpy and/or pandas, we can update.

update dependencies

092fcc7

angela97lin added 3 commits December 10, 2020 01:57

update dependencies

d0a7424

release notes

fe6f4d3

Merge branch 'main' into 1519_update_dependencies

563731a

angela97lin self-assigned this Dec 10, 2020

angela97lin added this to the December 2020 milestone Dec 10, 2020

angela97lin marked this pull request as ready for review December 10, 2020 16:45

angela97lin requested review from dsherry, freddyaboulton, bchen1116, jeremyliweishih and ParthivNaresh December 10, 2020 16:45

jeremyliweishih approved these changes Dec 10, 2020

View reviewed changes

angela97lin commented Dec 10, 2020

View reviewed changes

freddyaboulton approved these changes Dec 10, 2020

View reviewed changes

core-requirements.txt Show resolved Hide resolved

bchen1116 approved these changes Dec 10, 2020

View reviewed changes

dsherry approved these changes Dec 11, 2020

View reviewed changes

dsherry mentioned this pull request Dec 14, 2020

Lower numpy/pandas required versions to match featuretools/evalml alteryx/woodwork#457

Closed

dsherry approved these changes Dec 14, 2020

View reviewed changes

angela97lin added 2 commits December 14, 2020 11:37

Merge branch 'main' into 1519_update_dependencies

eb7eb1e

add to release notes

6bc73da

angela97lin merged commit 8a4e763 into main Dec 14, 2020

angela97lin deleted the 1519_update_dependencies branch December 14, 2020 17:43

dsherry mentioned this pull request Dec 29, 2020

Release v0.17.0 #1623

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update dependencies to fix `ImportError: cannot import name 'MaskedArray' from 'sklearn.utils.fixes'` errors #1540

Update dependencies to fix `ImportError: cannot import name 'MaskedArray' from 'sklearn.utils.fixes'` errors #1540

angela97lin commented Dec 10, 2020 •

edited

codecov bot commented Dec 10, 2020 •

edited

jeremyliweishih left a comment •

edited

angela97lin Dec 10, 2020

freddyaboulton left a comment

bchen1116 left a comment

dsherry left a comment

dsherry Dec 11, 2020

angela97lin Dec 12, 2020

dsherry Dec 11, 2020

angela97lin Dec 12, 2020

angela97lin commented Dec 12, 2020

dsherry left a comment

		numpy>=1.19.1
		pandas>=1.1.0

Update dependencies to fix ImportError: cannot import name 'MaskedArray' from 'sklearn.utils.fixes' errors #1540

Update dependencies to fix ImportError: cannot import name 'MaskedArray' from 'sklearn.utils.fixes' errors #1540

Conversation

angela97lin commented Dec 10, 2020 • edited

codecov bot commented Dec 10, 2020 • edited

Codecov Report

jeremyliweishih left a comment • edited

Choose a reason for hiding this comment

angela97lin Dec 10, 2020

Choose a reason for hiding this comment

freddyaboulton left a comment

Choose a reason for hiding this comment

bchen1116 left a comment

Choose a reason for hiding this comment

dsherry left a comment

Choose a reason for hiding this comment

dsherry Dec 11, 2020

Choose a reason for hiding this comment

angela97lin Dec 12, 2020

Choose a reason for hiding this comment

dsherry Dec 11, 2020

Choose a reason for hiding this comment

angela97lin Dec 12, 2020

Choose a reason for hiding this comment

angela97lin commented Dec 12, 2020

dsherry left a comment

Choose a reason for hiding this comment

Update dependencies to fix `ImportError: cannot import name 'MaskedArray' from 'sklearn.utils.fixes'` errors #1540

Update dependencies to fix `ImportError: cannot import name 'MaskedArray' from 'sklearn.utils.fixes'` errors #1540

angela97lin commented Dec 10, 2020 •

edited

codecov bot commented Dec 10, 2020 •

edited

jeremyliweishih left a comment •

edited