New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Joblib 0.12.2 #11741

Merged
merged 3 commits into from Aug 3, 2018

Conversation

Projects
None yet
6 participants
@ogrisel
Member

ogrisel commented Aug 3, 2018

Synchronize to the latest version of joblib.

  • includes loky: it fixes the intermittent crashes between OpenMP / Accelerate and GridSearchCV / RandomizedSearchCV;

  • makes it possible to better negotiate backends when using dask

  • better detection of the number of available CPUs and size of /dev/shm when running in containers such as docker

  • makes it possible to use LZ4 fast compression in load/dump

  • many other improvements and bug fixes.

@ogrisel ogrisel referenced this pull request Aug 3, 2018

Closed

[WIP] joblib 0.12 integration #9486

1 of 2 tasks complete
@ogrisel

This comment has been minimized.

Member

ogrisel commented Aug 3, 2018

Ok this is good to go. Merging.

@ogrisel ogrisel merged commit 40e6c43 into scikit-learn:master Aug 3, 2018

5 of 7 checks passed

codecov/patch 55.18% of diff hit (target 95.3%)
Details
codecov/project 93.08% (-2.23%) compared to eb453b6
Details
ci/circleci: deploy Your tests passed on CircleCI!
Details
ci/circleci: python2 Your tests passed on CircleCI!
Details
ci/circleci: python3 Your tests passed on CircleCI!
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@ogrisel ogrisel deleted the ogrisel:joblib-0.12.2 branch Aug 3, 2018

@qinhanmin2014

This comment has been minimized.

Member

qinhanmin2014 commented Aug 3, 2018

@ogrisel
(1) Circle CI is failing

Unexpected failing examples:
/home/circleci/project/examples/cluster/plot_feature_agglomeration_vs_univariate_selection.py failed leaving traceback:
Traceback (most recent call last):
  File "/home/circleci/project/examples/cluster/plot_feature_agglomeration_vs_univariate_selection.py", line 75, in <module>
    clf.fit(X, y)  # set the best parameters
  File "/home/circleci/project/sklearn/model_selection/_search.py", line 663, in fit
    cv.split(X, y, groups)))
  File "/home/circleci/project/sklearn/externals/joblib/parallel.py", line 981, in __call__
    if self.dispatch_one_batch(iterator):
  File "/home/circleci/project/sklearn/externals/joblib/parallel.py", line 818, in dispatch_one_batch
    self._pickle_cache)
  File "/home/circleci/project/sklearn/externals/joblib/parallel.py", line 253, in __init__
    self.items = list(iterator_slice)
  File "/home/circleci/project/sklearn/model_selection/_search.py", line 662, in <genexpr>
    for parameters, (train, test) in product(candidate_params,
  File "/home/circleci/project/sklearn/base.py", line 62, in clone
    new_object_params[name] = clone(param, safe=False)
  File "/home/circleci/project/sklearn/base.py", line 50, in clone
    return estimator_type([clone(e, safe=safe) for e in estimator])
  File "/home/circleci/project/sklearn/base.py", line 50, in <listcomp>
    return estimator_type([clone(e, safe=safe) for e in estimator])
  File "/home/circleci/project/sklearn/base.py", line 50, in clone
    return estimator_type([clone(e, safe=safe) for e in estimator])
  File "/home/circleci/project/sklearn/base.py", line 50, in <listcomp>
    return estimator_type([clone(e, safe=safe) for e in estimator])
  File "/home/circleci/project/sklearn/base.py", line 62, in clone
    new_object_params[name] = clone(param, safe=False)
  File "/home/circleci/project/sklearn/base.py", line 53, in clone
    return copy.deepcopy(estimator)
  File "/home/circleci/miniconda/envs/testenv/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/circleci/miniconda/envs/testenv/lib/python3.6/copy.py", line 274, in _reconstruct
    y = func(*args)
  File "/home/circleci/project/sklearn/externals/joblib/memory.py", line 830, in __init__
    location, cachedir))
ValueError: You set both "location='/tmp/tmp1v2mgts1' and "cachedir=False". 'cachedir' has been deprecated in version 0.12 and will be removed in version 0.14.
Please only set "location='/tmp/tmp1v2mgts1'"


/home/circleci/project/examples/compose/plot_compare_reduction.py failed leaving traceback:
Traceback (most recent call last):
  File "/home/circleci/project/examples/compose/plot_compare_reduction.py", line 119, in <module>
    grid.fit(digits.data, digits.target)
  File "/home/circleci/project/sklearn/model_selection/_search.py", line 663, in fit
    cv.split(X, y, groups)))
  File "/home/circleci/project/sklearn/externals/joblib/parallel.py", line 981, in __call__
    if self.dispatch_one_batch(iterator):
  File "/home/circleci/project/sklearn/externals/joblib/parallel.py", line 818, in dispatch_one_batch
    self._pickle_cache)
  File "/home/circleci/project/sklearn/externals/joblib/parallel.py", line 253, in __init__
    self.items = list(iterator_slice)
  File "/home/circleci/project/sklearn/model_selection/_search.py", line 662, in <genexpr>
    for parameters, (train, test) in product(candidate_params,
  File "/home/circleci/project/sklearn/base.py", line 62, in clone
    new_object_params[name] = clone(param, safe=False)
  File "/home/circleci/project/sklearn/base.py", line 53, in clone
    return copy.deepcopy(estimator)
  File "/home/circleci/miniconda/envs/testenv/lib/python3.6/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/circleci/miniconda/envs/testenv/lib/python3.6/copy.py", line 274, in _reconstruct
    y = func(*args)
  File "/home/circleci/project/sklearn/externals/joblib/memory.py", line 830, in __init__
    location, cachedir))
ValueError: You set both "location='/tmp/tmpa408j1cz' and "cachedir=False". 'cachedir' has been deprecated in version 0.12 and will be removed in version 0.14.
Please only set "location='/tmp/tmpa408j1cz'"

(2) We get many extra warnings in the examples
e.g., UserWarning: 'n_jobs' > 1 does not have any effect when 'solver' is set to 'liblinear'. Got 'n_jobs' = None.
(3) In the doc, we still have things like n_jobs : int, optional (default=1)
(4) What'e the difference between n_jobs=1 and n_jobs=None? Maybe add some explanations?

@js3711

This comment has been minimized.

js3711 commented Aug 3, 2018

Thanks @ogrisel. Is there way to request a scikit-learn release candidate version through conda-forge (when it is ready)?

I'm excited about how dask will make use of these changes and I'd like to get a build for multiple platforms.

@jakirkham

This comment has been minimized.

Contributor

jakirkham commented Aug 3, 2018

We have built RCs for scikit-learn in the past. If scikit-learn plans to make an RC, we'd be happy to work with them to get it released. RCs for various packages are available via conda-forge/label/rc.

@jnothman

This comment has been minimized.

Member

jnothman commented Aug 5, 2018

@rth

This comment has been minimized.

Member

rth commented Aug 13, 2018

Discussion about the reported issues continued in #11771

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment