Not able to import manifold if scikit-learn is installed directly from github in Google Colab.

[Santosh-Gupta]

I am unable to import manifold if scikit-learn is installed directly from github in Google Colaboratory.

Here is my code

#create seperate folder to install scikit-learn
if not os.path.exists('MODEL7'):
    os.makedirs('MODEL7')

#install scikit-learn directly from github directly to new folder
!pip install -t ./MODEL7 https://github.com/scikit-learn/scikit-learn/archive/master.zip

#reload and import TSNE specifically from new folder
import importlib
import MODEL7.sklearn.manifold
importlib.reload(MODEL7.sklearn.manifold)
from MODEL7.sklearn.manifold import TSNE

This is the error I get

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-58-d1b89c7ee7c9> in <module>()
      1 import importlib
----> 2 import MODEL6.sklearn.manifold
      3 importlib.reload(MODEL6.sklearn.manifold)
      4 from MODEL6.sklearn.manifold import TSNE

/content/MODEL6/sklearn/manifold/__init__.py in <module>()
      3 """
      4 
----> 5 from .locally_linear import locally_linear_embedding, LocallyLinearEmbedding
      6 from .isomap import Isomap
      7 from .mds import MDS, smacof

/content/MODEL6/sklearn/manifold/locally_linear.py in <module>()
     15 from ..utils.validation import check_is_fitted
     16 from ..utils.validation import FLOAT_DTYPES
---> 17 from ..neighbors import NearestNeighbors
     18 
     19 

/content/MODEL6/sklearn/neighbors/__init__.py in <module>()
      7 from .kd_tree import KDTree
      8 from .dist_metrics import DistanceMetric
----> 9 from .graph import kneighbors_graph, radius_neighbors_graph
     10 from .unsupervised import NearestNeighbors
     11 from .classification import KNeighborsClassifier, RadiusNeighborsClassifier

/content/MODEL6/sklearn/neighbors/graph.py in <module>()
      5 # License: BSD 3 clause (C) INRIA, University of Amsterdam
      6 
----> 7 from .base import KNeighborsMixin, RadiusNeighborsMixin
      8 from .unsupervised import NearestNeighbors
      9 

/content/MODEL6/sklearn/neighbors/base.py in <module>()
     19 from .kd_tree import KDTree
     20 from ..base import BaseEstimator
---> 21 from ..metrics import pairwise_distances_chunked
     22 from ..metrics.pairwise import PAIRWISE_DISTANCE_FUNCTIONS
     23 from ..utils import check_X_y, check_array, _get_n_jobs, gen_even_slices

/content/MODEL6/sklearn/metrics/__init__.py in <module>()
      5 
      6 
----> 7 from .ranking import auc
      8 from .ranking import average_precision_score
      9 from .ranking import coverage_error

/content/MODEL6/sklearn/metrics/ranking.py in <module>()
     34 from ..utils.sparsefuncs import count_nonzero
     35 from ..exceptions import UndefinedMetricWarning
---> 36 from ..preprocessing import label_binarize
     37 
     38 from .base import _average_binary_score

/content/MODEL6/sklearn/preprocessing/__init__.py in <module>()
      4 """
      5 
----> 6 from ._function_transformer import FunctionTransformer
      7 
      8 from .data import Binarizer

/content/MODEL6/sklearn/preprocessing/_function_transformer.py in <module>()
      3 from ..base import BaseEstimator, TransformerMixin
      4 from ..utils import check_array
----> 5 from ..utils.testing import assert_allclose_dense_sparse
      6 from ..externals.six import string_types
      7 

/content/MODEL6/sklearn/utils/testing.py in <module>()
     48 from sklearn.externals import joblib
     49 from sklearn.utils.fixes import signature
---> 50 from sklearn.utils import deprecated, IS_PYPY
     51 
     52 

ImportError: cannot import name 'IS_PYPY'

---------------------------------------------------------------------------
NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.
---------------------------------------------------------------------------

If I use the 0.19.X version instead of master, it imports manifold just fine with no errors.

ie:

!pip install -t ./MODEL8 https://github.com/Santosh-Gupta/scikit-learn/archive/0.19.X.zip

[amueller]

There's probably a mixture of versions installed right now. make sure sklearn is not installed before installing from github.

[Santosh-Gupta]

I forgot to mention an important detail: I'm trying to install this in Google Colaboratory. I updated the title and original comment.

I inserted the line !pip uninstall scikit-learn -y but it is still not able to import manifold, though I am getting a different error message. Before the main error message was " ImportError: cannot import name 'IS_PYPY' " , now the main error message is "ModuleNotFoundError: No module named 'sklearn'"

I am also using a fresh runtime. You can do this by switching from GPU mode to CPU mode (some how just selecting runtime->restart keeps previous pip installations).

Here is the full code I last tried

#Google Colab pip is not up to date so updating just in case
!pip install --upgrade pip
!pip uninstall scikit-learn -y 
!pip install Cython
import os

#create seperate folder to install scikit-learn
if not os.path.exists('sklearnFolder'):
    os.makedirs('sklearnFolder')

#install scikit-learn directly from github directly to new folder
!pip install -t ./sklearnFolder https://github.com/scikit-learn/scikit-learn/archive/master.zip

#reload and import TSNE specifically from new folder
import importlib
import sklearnFolder.sklearn.manifold
importlib.reload(sklearnFolder.sklearn.manifold)
from sklearnFolder.sklearn.manifold import TSNE

Here is the output/error message

Collecting pip
  Downloading https://files.pythonhosted.org/packages/5f/25/e52d3f31441505a5f3af41213346e5b6c221c9e086a166f3703d2ddaf940/pip-18.0-py2.py3-none-any.whl (1.3MB)
    100% |████████████████████████████████| 1.3MB 6.8MB/s 
Installing collected packages: pip
  Found existing installation: pip 10.0.1
    Uninstalling pip-10.0.1:
      Successfully uninstalled pip-10.0.1
Successfully installed pip-18.0
Uninstalling scikit-learn-0.19.2:
  Successfully uninstalled scikit-learn-0.19.2
Collecting Cython
  Downloading https://files.pythonhosted.org/packages/1c/10/ffebdb9faa901c83b69ab7040a1f5f3b2c71899be141752a6d466718c491/Cython-0.28.4-cp36-cp36m-manylinux1_x86_64.whl (3.4MB)
    100% |████████████████████████████████| 3.4MB 8.2MB/s 
Installing collected packages: Cython
Successfully installed Cython-0.28.4
Collecting https://github.com/scikit-learn/scikit-learn/archive/master.zip
  Downloading https://github.com/scikit-learn/scikit-learn/archive/master.zip
     - 6.6MB 21.4MB/s
Collecting numpy>=1.8.2 (from scikit-learn==0.20.dev0)
  Downloading https://files.pythonhosted.org/packages/88/29/f4c845648ed23264e986cdc5fbab5f8eace1be5e62144ef69ccc7189461d/numpy-1.15.0-cp36-cp36m-manylinux1_x86_64.whl (13.9MB)
    100% |████████████████████████████████| 13.9MB 2.9MB/s 
Collecting scipy>=0.13.3 (from scikit-learn==0.20.dev0)
  Downloading https://files.pythonhosted.org/packages/a8/0b/f163da98d3a01b3e0ef1cab8dd2123c34aee2bafbb1c5bffa354cc8a1730/scipy-1.1.0-cp36-cp36m-manylinux1_x86_64.whl (31.2MB)
    100% |████████████████████████████████| 31.2MB 1.5MB/s 
Building wheels for collected packages: scikit-learn
  Running setup.py bdist_wheel for scikit-learn ... done
  Stored in directory: /tmp/pip-ephem-wheel-cache-1aiq5q0b/wheels/9e/4b/7e/56798a5efcb7101fb6518d859ba612b4c38ff8be4898bdfb6f
Successfully built scikit-learn
Installing collected packages: numpy, scipy, scikit-learn
Successfully installed numpy-1.15.0 scikit-learn-0.20.dev0 scipy-1.1.0
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-1-d14f1b24ef90> in <module>()
     12 #reload and import TSNE specifically from new folder
     13 import importlib
---> 14 import sklearnFolder.sklearn.manifold
     15 importlib.reload(sklearnFolder.sklearn.manifold)
     16 from sklearnFolder.sklearn.manifold import TSNE

/content/sklearnFolder/sklearn/manifold/__init__.py in <module>()
      3 """
      4 
----> 5 from .locally_linear import locally_linear_embedding, LocallyLinearEmbedding
      6 from .isomap import Isomap
      7 from .mds import MDS, smacof

/content/sklearnFolder/sklearn/manifold/locally_linear.py in <module>()
     15 from ..utils.validation import check_is_fitted
     16 from ..utils.validation import FLOAT_DTYPES
---> 17 from ..neighbors import NearestNeighbors
     18 
     19 

/content/sklearnFolder/sklearn/neighbors/__init__.py in <module>()
      4 """
      5 
----> 6 from .ball_tree import BallTree
      7 from .kd_tree import KDTree
      8 from .dist_metrics import DistanceMetric

sklearn/neighbors/dist_metrics.pxd in init sklearn.neighbors.ball_tree()

ModuleNotFoundError: No module named 'sklearn'

---------------------------------------------------------------------------
NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.
---------------------------------------------------------------------------

I noticed that the 0.19.2 version was uninstalled from Colab, so perhaps Colab is somehow setup only for 0.19.2 since that version is pre-installed.

[rth]

Even without google collab, I can reproduce the import error when trying to install sklearn in some specific folder with pip install -t sklearnFolder then importing it as sklearnFolder.sklearn.manifold.

That is not a supported way of installing / using scikit-lean. One reason why it wouldn't work is that a few modules in scikit-learn use absolute imports (e.g. from sklearn import something) which will fail with such setup.

You should either install it with pip install <github-url> or if you want to have it in some specific folder, clone the repo to that folder, then run pip install -e <folder-name>, in both cases it will be imported as sklearn.

Closing as a usage question, please comment if you disagree.

[amueller]

We shouldn't use absolute imports though, right? I thought we only did that in tests?

[rth]

We shouldn't use absolute imports though, right? I thought we only did that in tests?

We do use them in tests, but also in a few python files and a number of cython files. Is there a reason we don't like absolute imports in sklearn (cf related discussion in https://stackoverflow.com/a/16748366/1791279) ? There are also some suspicious looking issues about relative cimport in Cython.

I would be +1 to make all imports (excluding tests) relative for the sake of consistency (made a PR for the Python files in #11665), but I still don't think that the we want to support pip install --target some_folder sklearn; import some_folder.sklearn in general.