[WIP] joblib 0.12 integration

[ogrisel]

This PR is not mean to be merged as this. This vendors the development branches of loky and joblib, notably the following PRs:

• [MRG+1] Integrate loky as default backend joblib/joblib#516 (in joblib) [ready for merge]
• https://github.com/tomMoral/loky/pull/83 (in loky) [merged]

The goal is to have the full scikit-learn CI test suite run on this PRs to ensure that we can merge them and release them without causing a regression in scikit-learn master.

TODO for this PR:

• fix the coverage bug upstream and backport a fix via a monkeypatch or other in our CI
• decide whether we want to keep on vendoring everything or not

This is part of #7650: A plan to safely support OpenMP in our Cython code base.

[amueller]

Or we could stop vendoring and declare our dependencies?

[amueller]

(though that would still require a PR with the dependency changed to test the integration)

[amueller]

wait joblib vendors loky?!? I'm against integrating loky either as vendored in joblib or as vendored in sklearn. Scikit-learn is not a package manager.

[ogrisel]

I agree that nowadays dependency management works much better that it used to do in the past so that we can decide to stop vendoring stuff. Personally I am fine with both approaches. All those packages (joblib, loky and cloudpickle) have universal wheels, there is no compiled extensions in them.

[ogrisel]

This PR has revealed that the new combinations of the long running threads and coverage can trigger the following bug in the coverage package:

https://bitbucket.org/ned/coveragepy/issues/581/44b1-44-breaking-in-ci

We did not trigger this has part of the joblib and loky CI setup for some reason.

[ogrisel]

The appveyor tests of sklearn have revealed a packaging issue. I am on it but I have to go now. I will fix it tomorrow.

[ogrisel]

@tomMoral the appveyor failure (Python 2.7 / 64 bit) seems to reveal a real bug in loky:

Exception in thread QueueManager:
Traceback (most recent call last):
  File "C:\Python27-x64\lib\threading.py", line 801, in __bootstrap_inner
    self.run()
  File "C:\Python27-x64\lib\threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "C:\Python27-x64\lib\site-packages\sklearn\externals\joblib\externals\loky\process_executor.py", line 500, in _queue_management_worker
    timeout=_poll_timeout)
  File "C:\Python27-x64\lib\site-packages\sklearn\externals\joblib\externals\loky\backend\_win_wait.py", line 52, in wait
    elif h.poll(0):
AttributeError: 'long' object has no attribute 'poll'

I wonder why we did not get that bug from the loky appveyor CI.

[ogrisel]

@tomMoral as shown in the second commit of tomMoral/loky#87 we did not test 64 bit windows as we thought we were. Will read up about tox and appveyor.

[ogrisel]

The failures with 64 bit Python under windows have been fixed. CI is red because of the aforementioned coverage bug.

[ogrisel]

\o/ the coverage fix works. I hope my PR will get merged upstream in the coverage project ;)

[ogrisel]

@tomMoral I am currently rebasing this PR on top of the current master (and upgrading to joblib 0.12.1).

[ogrisel]

Closing in favor of #11741.