[MRG+2] parallelized VotingClassifier

[olologin]

First version, looks like it's working :)
Also, i added sample_weight parameter into fit method, don't know if it's appropriate.

[olologin]

If someone wants to test it with nested multithreading, e.g. in the code below we call version of VotingClassifier from cross_val_score.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import GaussianNB
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import VotingClassifier
from sklearn import datasets
from sklearn.model_selection import cross_val_score

# Load the iris dataset and randomly permute it
iris = datasets.load_iris()
X, y = iris.data[:, 1:3], iris.target


"""Check classification by majority label on dataset iris."""
clf1 = LogisticRegression(random_state=123)
clf2 = RandomForestClassifier(random_state=123)
clf3 = GaussianNB()
eclf = VotingClassifier(estimators=[
    ('lr', clf1), ('rf', clf2), ('gnb', clf3)],
    voting='hard', n_jobs=-1, backend='multiprocessing')
scores = cross_val_score(eclf, X, y, cv=5, scoring='accuracy', n_jobs=-1)

With backend='multiprocessing' it shows message UserWarning: Multiprocessing-backed parallel loops cannot be nested, setting n_jobs=1, so it just falls back to -1 threads in cross_val_score, but VotingClassifier entirely works in 1 thread. With backend='threading' it doesn't show any warning.

In both cases everything works fine and provides correct results.

[giorgiop]

In both cases everything works fine and provides correct results.

Could you please add tests showing that? Same for the sample_weight

[olologin]

Could you please add tests showing that? Same for the sample_weight

Ok, i'll add couple of tests with multithreading/processing. I've tested it already on existing tests from test_voting_classifier, with different default threading/processing parameters.

[betatim]

two empty lines between functions please (PEP8)

[betatim]

There are a few more pep8 related things to fix (param = 42 should be param=42, etc) in the code. If you install pep8 to have it tell you about them.

[MechCoder]

I don't think we should allow the user to choose this. AFAIK, the threading backend is useful only if the code is written in a nogil block and the user is highly unlikely to know in which estimators that this is being done.

We should maybe just allow the default "multiprocessing" backend just like in cross_val_score and in other places.

[MechCoder]

@olologin Could you rebase?

[MechCoder]

Is this test not redundant? Isn't the previous test sufficient?

[olologin]

It tests "hard" voting stability (So that 1threaded and 2threaded version will give same results) on iris dataset, while previous one tests 'soft' version.
Anyway first test is fast (Small artificial dataset), so it shouldn't increase testing time noticeably.

[MechCoder]

LGTM pending nitpick @olologin

cc: @agramfort | @TomDLT for a quick second pass.

[MechCoder]

And sorry for the delays..

[olologin]

@MechCoder , Actually I almost forgot about this PR, thought it was someone else’s :)

Could you take a look at this one too #6116 ? I think it's good PR too.

[MechCoder]

I can try after the sem breaks, if no one gets to it by then.

[jnothman]

A few cosmetic issues here.

1. Let's avoid this level of nesting.
2. closing parentheses conventionally appear right after what they're closing, not on a new line.

[jnothman]

could you sort this minor issue out

[jnothman]

For many estimators (e.g. linear models), predicting in parallel will degrade performance, particularly with the multiprocessing backend. Perhaps we should be using the threading backend at predict time, for which overhead is much smaller, or remove it altogether. You're welcome to benchmark, but I'm not sure how you'd come up with a realistic set of voters in the ensemble.

[olologin]

@jnothman, Yes, you are right about this. I made some benchmarks with this code:

import xgboost as xgb
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier, ExtraTreesClassifier
from sklearn.svm import SVC
from sklearn.ensemble import VotingClassifier
from sklearn.datasets import load_digits

digits = load_digits()

lr = LogisticRegression(C=1000)
svc = SVC(gamma=.001, C=100, kernel='rbf', probability=True)
rf= RandomForestClassifier(n_estimators=400)
extraGini = ExtraTreesClassifier(n_estimators=400, criterion='gini')
extraEntropy = ExtraTreesClassifier(n_estimators=400, criterion='entropy')
xgb_model = xgb.XGBClassifier(nthread=2, silent=True, colsample_bytree=.4, learning_rate=0.05, max_depth=4, gamma=0, n_estimators=800)

#Setup the ensemble classifier
eclf = VotingClassifier(estimators=[
    ('lr', lr),
    ('svc', svc),
    ('rf', rf),
    ('extraGini', extraGini),
    ('extraEntropy', extraEntropy),
    ('xgb_model', xgb_model)
], voting='soft', n_jobs=4)

eclf.fit(digits.data, digits.target)
%timeit eclf.predict(digits.data)
%timeit eclf.predict_proba(digits.data)

Version with multiprocessing backend takes ~2.1sec, threading backend ~1.3sec, without any parallelization ~1.9sec for predict_proba and predict.

I think I could revert parallelization in predict methods completely (though threading backend is faster, difference is not so big), or change it to threading backend. What do you think is best solution?

[jnothman]

I don't feel like I have the expertise to judge this, but would lean conservatively (i.e. no parallelism) since we can't know the makeup of the ensemble, and prediction time is small relative to fit time.

[jnothman]

Also, all prediction can be parallelised over samples, so slow predictors can deal with that, I supose.

[MechCoder]

We can merge after rebase..

[jnothman]

Uh, @MechCoder I've not actually given this my +1.

[jnothman]

(nor has anyone but you)

[jnothman]

I don't get why there's an _ after sample_weight here.

[jnothman]

PEP8 requires newline at end of file

[jnothman]

LGTM!

[olologin]

@jnothman, thanks. Sorry for having so many style problems in so small PR :) I'm working with C++ at work, so sometimes different code-style kicks in unintentionally.

[jnothman]

No big deal. A commit hook running flake8 on changed files can help...

[jnothman]

Thanks!