[BUG] Debugging imblearn/sklearn version problems

[hayesall]

Describe the bug

Over the last few days, users have reported issues after installing imbalanced-learn (e.g. #723, #725, #726)

TL;DR: Users with systems that build based on setup.py experience errors.

Steps/Code to Reproduce

Set up an environment to imitate a user who currently has scikit-learn==0.22.1:

$ conda create -n test python=3.7
$ conda activate test
$ pip install scikit-learn==0.22.1
Collecting scikit-learn==0.22.1
  Using cached scikit_learn-0.22.1-cp37-cp37m-manylinux1_x86_64.whl (7.0 MB)
...

Versions

System, Dependency Information

System Information

• python : 3.7.7 (default, May 7 2020, 21:25:33) [GCC 7.3.0]
• executable: /home/hayesall/miniconda3/envs/test/bin/python
• machine : Linux-5.3.0-59-generic-x86_64-with-debian-buster-sid

Python Dependencies

• pip : 20.0.2
• setuptools: 47.1.1.post20200604
• imblearn : 0.7.0
• sklearn : 0.22.1
• numpy : 1.18.5
• scipy : 1.4.1
• Cython : None
• pandas : None
• keras : None
• tensorflow: None
• joblib : 0.15.1

---

Case 1: Installing with pip

These are the default paths on my system:

>>> import sys
>>> for entry in sys.path:
...   print(entry)
/home/hayesall/miniconda3/envs/test/lib/python37.zip
/home/hayesall/miniconda3/envs/test/lib/python3.7
/home/hayesall/miniconda3/envs/test/lib/python3.7/lib-dynload
/home/hayesall/miniconda3/envs/test/lib/python3.7/site-packages

Installing imbalanced-learn with pip on Ubuntu uses the wheel from PyPi:

$ pip install imbalanced-learn
...
Collecting scikit-learn>=0.23
  Downloading scikit_learn-0.23.1-cp37-cp37m-manylinux1_x86_64.whl (6.8 MB)
... (it correctly upgrades scikit-learn)

And the paths appear correct following installation:

>>> import sys
>>> for entry in sys.path:
...   print(entry)
/home/hayesall/miniconda3/envs/test/lib/python37.zip
/home/hayesall/miniconda3/envs/test/lib/python3.7
/home/hayesall/miniconda3/envs/test/lib/python3.7/lib-dynload
/home/hayesall/miniconda3/envs/test/lib/python3.7/site-packages

Case 2: Installing with setup.py

Again, these seem to be the default paths on my system:

>>> import sys
>>> for entry in sys.path:
...   print(entry)
/home/hayesall/miniconda3/envs/test/lib/python37.zip
/home/hayesall/miniconda3/envs/test/lib/python3.7
/home/hayesall/miniconda3/envs/test/lib/python3.7/lib-dynload
/home/hayesall/miniconda3/envs/test/lib/python3.7/site-packages

But now, installing imbalanced-learn with setup.py

git clone git@github.com:scikit-learn-contrib/imbalanced-learn.git
cd imbalanced-learn
python setup.py install

Summary of the install log

running install
running bdist_egg
running egg_info
writing imbalanced_learn.egg-info/PKG-INFO
writing dependency_links to imbalanced_learn.egg-info/dependency_links.txt
writing requirements to imbalanced_learn.egg-info/requires.txt
writing top-level names to imbalanced_learn.egg-info/top_level.txt
reading manifest file 'imbalanced_learn.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'CONTRIBUTING.ms'
writing manifest file 'imbalanced_learn.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/imblearn
...
creating 'dist/imbalanced_learn-0.7.0-py3.7.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing imbalanced_learn-0.7.0-py3.7.egg
creating /home/hayesall/miniconda3/envs/test/lib/python3.7/site-packages/imbalanced_learn-0.7.0-py3.7.egg
Extracting imbalanced_learn-0.7.0-py3.7.egg to /home/hayesall/miniconda3/envs/test/lib/python3.7/site-packages
Adding imbalanced-learn 0.7.0 to easy-install.pth file

Installed /home/hayesall/miniconda3/envs/test/lib/python3.7/site-packages/imbalanced_learn-0.7.0-py3.7.egg
Processing dependencies for imbalanced-learn==0.7.0
Searching for scikit-learn>=0.23
Reading https://pypi.org/simple/scikit-learn/
Downloading https://files.pythonhosted.org/packages/b8/7e/74e707b66490d4eb05f702966ad0990881127acecf9d5cdcef3c95ec6c16/scikit_learn-0.23.1-cp37-cp37m-manylinux1_x86_64.whl#sha256=0e7b55f73b35537ecd0d19df29dd39aa9e076dba78f3507b8136c819d84611fd
Best match: scikit-learn 0.23.1
Processing scikit_learn-0.23.1-cp37-cp37m-manylinux1_x86_64.whl
Installing scikit_learn-0.23.1-cp37-cp37m-manylinux1_x86_64.whl to /home/hayesall/miniconda3/envs/test/lib/python3.7/site-packages
Adding scikit-learn 0.23.1 to easy-install.pth file

Installed /home/hayesall/miniconda3/envs/test/lib/python3.7/site-packages/scikit_learn-0.23.1-py3.7-linux-x86_64.egg

This method creates three new entries under site-packages:

>>> import sys
>>> for entry in sys.path:
...   print(entry)
/home/hayesall/miniconda3/envs/test/lib/python37.zip
/home/hayesall/miniconda3/envs/test/lib/python3.7
/home/hayesall/miniconda3/envs/test/lib/python3.7/lib-dynload
/home/hayesall/miniconda3/envs/test/lib/python3.7/site-packages
/home/hayesall/miniconda3/envs/test/lib/python3.7/site-packages/imbalanced_learn-0.7.0-py3.7.egg
/home/hayesall/miniconda3/envs/test/lib/python3.7/site-packages/scikit_learn-0.23.1-py3.7-linux-x86_64.egg
/home/hayesall/miniconda3/envs/test/lib/python3.7/site-packages/threadpoolctl-2.1.0-py3.7.egg

... but does not appear to properly clean up the old scikit-learn version:

>>> import sklearn
>>> sklearn.__version__
'0.22.1'
>>> sklearn.__file__
'/home/hayesall/miniconda3/envs/test/lib/python3.7/site-packages/sklearn/__init__.py'

It looks like something during setup is creating an easy-install.pth with the following contents:

./imbalanced_learn-0.7.0-py3.7.egg
./scikit_learn-0.23.1-py3.7-linux-x86_64.egg
./threadpoolctl-2.1.0-py3.7.egg

[xjyribro]

Hi,

Unfortunately, Anaconda doesn't have the latest version of SKLearn (they are on ver 0.22.1). This means I still get the error when I run my code on Jupyter. I tried installing the latest SKlearn via pip but I think Anaconda reads from their own file path.

I'm thinking if :
I can replace the SKlearn lib in Anaconda with the one installed via pip; but I don't want to mess with the files.
OR
I can wait for Anaconda to update their lib.

Any comments about this would be helpful!

[glemaitre]

@xjyribro you can install the version from the conda-forge: conda insall scikit-learn -c conda-forge.
It is packaged by the people from scikit-learn so it will be OK. When the package will be available in the default channel then you can use it.

[glemaitre]

@hayesall I think that people should not use python setup.py install and instead pip install . as mentioned in the documentation. I remember some old deprecation from pip that was complaining about not cleaning properly stuff when using the former solution. I did the following:

 2001  conda create -n test python=3.7
 2002  conda activate test
 2003  pip install scikit-learn==0.22.1
 2004  cd Documents/packages/
 2005  ls
 2006  cd imbalanced-learn
 2007  pip install .

and it seems to work:

09:32 $ pip install .
Processing /home/glemaitre/Documents/packages/imbalanced-learn
Requirement already satisfied: numpy>=1.13.3 in /home/glemaitre/miniconda3/envs/test/lib/python3.7/site-packages (from imbalanced-learn==0.7.0) (1.19.0)
Requirement already satisfied: scipy>=0.19.1 in /home/glemaitre/miniconda3/envs/test/lib/python3.7/site-packages (from imbalanced-learn==0.7.0) (1.5.0)
Collecting scikit-learn>=0.23
  Using cached scikit_learn-0.23.1-cp37-cp37m-manylinux1_x86_64.whl (6.8 MB)
Requirement already satisfied: joblib>=0.11 in /home/glemaitre/miniconda3/envs/test/lib/python3.7/site-packages (from imbalanced-learn==0.7.0) (0.15.1)
Collecting threadpoolctl>=2.0.0
  Using cached threadpoolctl-2.1.0-py3-none-any.whl (12 kB)
Building wheels for collected packages: imbalanced-learn
  Building wheel for imbalanced-learn (setup.py) ... done
  Created wheel for imbalanced-learn: filename=imbalanced_learn-0.7.0-py3-none-any.whl size=167059 sha256=918ab70c37d7e0c4bd3a323df08feaf631f0b375497d5c50f460166dea08e223
  Stored in directory: /home/glemaitre/.cache/pip/wheels/8e/c7/5f/c6068c06319a76fc0fd864c711c72f4d686a4b45a128df1d12
Successfully built imbalanced-learn
Installing collected packages: threadpoolctl, scikit-learn, imbalanced-learn
  Attempting uninstall: scikit-learn
    Found existing installation: scikit-learn 0.22.1
    Uninstalling scikit-learn-0.22.1:
      Successfully uninstalled scikit-learn-0.22.1
Successfully installed imbalanced-learn-0.7.0 scikit-learn-0.23.1 threadpoolctl-2.1.0
(test) ✔ ~/Documents/packages/imbalanced-learn [0.7.X L|✔] 
09:32 $ python
Python 3.7.7 (default, May  7 2020, 21:25:33) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sklearn
>>> sklearn.__version__
'0.23.1'

[chkoar]

@glemaitre and it seems to work:

Shouldn't we then document it?

[glemaitre]

I am still not sure if it was a pip issue. If it was a pip issue, we should probably report the problem instead of documenting a workaround, isn't it?

[AkhilMokkapati]

conda insall scikit-learn -c conda-forge
This has worked for you....thank you very much

Windows-10-10.0.19041-SP0
Python 3.7.7 (default, Apr 15 2020, 05:09:04) [MSC v.1916 64 bit (AMD64)]
NumPy 1.18.1
SciPy 1.4.1
Scikit-Learn 0.23.2
imblearn 0.7.0

[sridatascience]

conda install scikit-learn -c conda-forge stil throws the same problem my scikit-learn version is not 0.23.2
after creating the environment do we have to reinstall jupyter notebook

[sridatascience]

can someone list down the steps to be taken to avoid this error
AttributeError: 'SMOTE' object has no attribute '_validate_data'.
I get this error when using with Jupyternotebook , please list down exact steps to come up with a solution on jupyter notebook

[sridatascience]

This seems to be the solution as given :
can some one suggest what is meant by reinstalling the IDE:
conda create -n test python=3.7.6
conda activate test
pip install scikit-learn==0.23.1
pip install imbalanced-learn==0.7.0
Finally, you need to reinstall your IDE in the new virtual environment in order to use these packages.

However, once scikit-learn version 0.23.1 becomes available in Conda and there are no compatibility issues, you can install it in the base environment directly

[hayesall]

@sridatascience It looks like you're referring to this thread on StackOverflow.

IDE likely refers to an Integrated Development Environment (IDLE, PyCharm, etc.). If you're using Jupyter notebooks, this advice is unrelated.

Installing imbalanced-learn and jupyter should be sufficient.

[sridatascience]

But still my version of scikit-learn is not the updated-'0.22.1' whereas the imbalanced-learn is updated to '0.7.0'

[glemaitre]

The issue is only due to the fact that we need the latest scikit-learn with the latest imbalanced-learn.
I will close then.

[ViaKepesi]

This seems to be the solution as given :
can some one suggest what is meant by reinstalling the IDE:
conda create -n test python=3.7.6
conda activate test
pip install scikit-learn==0.23.1
pip install imbalanced-learn==0.7.0
Finally, you need to reinstall your IDE in the new virtual environment in order to use these packages.

However, once scikit-learn version 0.23.1 becomes available in Conda and there are no compatibility issues, you can install it in the base environment directly

This instruction helped me finally. But I have to mention that there were some access issues around as well. I had to give permission to my user on Anaconda3 folder under the Program Data folder...Without this step I was not able to install the correct version of sckit learn and imbalanced-learn.