loading dask_ml gives error contextualversionconflict with sklearn

[iamyihwa]

Getting error "ContextualVersionConflict: (scikit-learn 0.22.1 (/anaconda/envs/azureml_py38/lib/python3.8/site-packages), Requirement.parse('scikit-learn>=1.2.0'), {'dask-ml'})"

Have updated the setuptools following this issue.

sklearn is also upto date (version = 1.2.2)

Minimal Complete Verifiable Example:

# Put your MCVE code here

Anything else we need to know?:

Environment:

• Dask version: 2021.06.0
• Python version: 3.8
• Operating System: linux
• Install method (conda, pip, source): pip

[TomAugspurger]

Hmm, two things are confusing me.

1. Why does it say sckit-learn 0.22.1 in that traceback?
2. Exactly whose code is running there? Can you post the full traceback (ideally the text, not a screenshot)?

I'm also surprised to see dask-ml using pkg_resources in https://github.com/dask/dask-ml/blob/main/dask_ml/__init__.py. I don't think that's the recommended way to do things with setuptools-scm these days (https://pypi.org/project/setuptools-scm/).

A PR to update the packaging stuff would be most welcome!

[iamyihwa]

@TomAugspurger

1. Why does it say sckit-learn 0.22.1 in that traceback?

Good question. Not sure ..

Exactly whose code is running there? Can you post the full traceback (ideally the text, not a screenshot)?
Sure

BTW, I have reinstalled the dask to the most up to date version, and now the error is different.
dask version = 2023.5.0

import dask_ml

Whole error message below:

ImportError                               Traceback (most recent call last)
Cell In[9], line 1
----> 1 import dask_ml

File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/dask_ml/__init__.py:4
      1 from pkg_resources import DistributionNotFound, get_distribution
      3 # Ensure we always register tokenizers
----> 4 from dask_ml.model_selection import _normalize
      6 __all__ = []
      8 try:

File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/dask_ml/model_selection/__init__.py:6
      1 """Utilities for hyperparameter optimization.
      2 
      3 These estimators will operate in parallel. Their scalability depends
      4 on the underlying estimators being used.
      5 """
----> 6 from ._hyperband import HyperbandSearchCV
      7 from ._incremental import IncrementalSearchCV, InverseDecaySearchCV
      8 from ._search import GridSearchCV, RandomizedSearchCV, check_cv, compute_n_splits

File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/dask_ml/model_selection/_hyperband.py:12
      9 import numpy as np
     10 from sklearn.utils import check_random_state
---> 12 from ._incremental import BaseIncrementalSearchCV
     13 from ._successive_halving import SuccessiveHalvingSearchCV
     15 logger = logging.getLogger(__name__)

File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/dask_ml/model_selection/_incremental.py:19
     17 import scipy.stats
     18 import toolz
---> 19 from dask.distributed import Future, default_client, futures_of, wait
     20 from distributed.utils import log_errors
     21 from sklearn.base import BaseEstimator, clone

File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/dask/distributed.py:11
      3 _import_error_message = (
      4     "dask.distributed is not installed.\n\n"
      5     "Please either conda or pip install distributed:\n\n"
      6     "  conda install dask distributed             # either conda install\n"
      7     '  python -m pip install "dask[distributed]" --upgrade    # or pip install'
      8 )
     10 try:
---> 11     from distributed import *
     12 except ImportError as e:
     13     if e.msg == "No module named 'distributed'":

File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/distributed/__init__.py:7
      4 from dask.utils import import_required
      6 from ._version import get_versions
----> 7 from .actor import Actor, ActorFuture
      8 from .client import (
      9     Client,
     10     CompatibleExecutor,
   (...)
     20     wait,
     21 )
     22 from .core import Status, connect, rpc

File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/distributed/actor.py:6
      3 import threading
      4 from queue import Queue
----> 6 from .client import Future, default_client
      7 from .protocol import to_serialize
      8 from .utils import iscoroutinefunction, sync, thread_state

File /anaconda/envs/azureml_py38/lib/python3.8/site-packages/distributed/client.py:30
     28 import dask
     29 from dask.base import collections_to_dsk, normalize_token, tokenize
---> 30 from dask.compatibility import apply
     31 from dask.core import flatten
     32 from dask.highlevelgraph import HighLevelGraph

ImportError: cannot import name 'apply' from 'dask.compatibility' (/anaconda/envs/azureml_py38/lib/python3.8/site-packages/dask/compatibility.py)

[TomAugspurger]

For that new error message, you'll want to ensure that your versions of dask and distributed match.

[iamyihwa]

Thanks @TomAugspurger
Indeed upgrading dask[distributed] using python -m pip install "dask[distributed]" --upgrade as was mentioned in the error message did solve the issue!
Thanks for the help!!!