Very weird behavior from RandomForest with warm start

[joaquinvanschoren]

According to OpenML, I'm getting a really strange performance boost when using warm start RandomForests:

Using OpenML:

from sklearn.ensemble import RandomForestClassifier
from openml import tasks,runs,datasets

task = tasks.get_task(145677)
clfs = [RandomForestClassifier(n_estimators=64,warm_start=True,n_jobs=-1),
        RandomForestClassifier(n_estimators=64,warm_start=False,n_jobs=-1)]

for clf in clfs:
    run = runs.run_task(task, clf)
    p = run.publish()
    print("Uploaded run 1 with id %s. Check it at www.openml.org/r/%s" %(run.run_id,run.run_id))

Output:

Uploaded run 1 with id 1852507. Check it at www.openml.org/r/1852507
Uploaded run 1 with id 1852508. Check it at www.openml.org/r/1852508

Looking up the results on OpenML, the first has an AUC of 0.9959, the second has AUC 0.8764.
This is a huge performance gap. Moreover, the classifier was not trained incrementally, so why does the warm start make a difference?

Just to check whether this is OpenML-specific, here is the same experiment doing 10xCV locally:

task = tasks.get_task(145677)
dataset = task.get_dataset()
X, y, categorical = dataset.get_data(target=dataset.default_target_attribute,return_categorical_indicator=True)

clfs = [RandomForestClassifier(n_estimators=64,warm_start=True,n_jobs=-1),
        RandomForestClassifier(n_estimators=64,warm_start=False,n_jobs=-1)]

for clf in clfs:    
    scores = cross_val_score(clf,X,y,cv=10,scoring="roc_auc",n_jobs=-1)
    print("10xCV Score (AUC): %s" %(np.mean(scores)))

Output:

10xCV Score (AUC): 0.87225053074
10xCV Score (AUC): 0.87225053074

Any clue about what's going on? Are we somehow maintaining information from classifiers trained in previous runs? Even then, a score of 0.9959 would be unexpected. Is there some form of information leakage going on?

Seems like a case for @amueller @mfeurer @janvanrijn ;)

[joaquinvanschoren]

If you look at the predictions of https://www.openml.org/r/1852504
You can see that only the first fold has errors, the other 9 have perfect performance. Maybe some information leaks between the folds??

[joaquinvanschoren]

Here's a simulation of what happens if you don't properly reinitialize the classifiers in a CV:

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import cross_val_score

X, y = load_breast_cancer(return_X_y = True)

clf = RandomForestClassifier(n_estimators=64,warm_start=True,n_jobs=-1)
scores=[]
test_size = int(len(X)/10)
for i in range(0,10):
  X_test = X[i*test_size:(i+1)*test_size]
  y_test = y[i*test_size:(i+1)*test_size]
  X_train = np.concatenate((X[:i*test_size],X[(i+1)*test_size:]))
  y_train = np.concatenate((y[:i*test_size],y[(i+1)*test_size:]))
  fit = clf.fit(X_train,y_train)
  score = clf.score(X_test,y_test)
  scores.append(score)
print('Scores',scores)
print('Mean',np.mean(scores))

clf = RandomForestClassifier(n_estimators=64,warm_start=True,n_jobs=-1)
scores = cross_val_score(clf, X, y, cv=10)
print('CV Mean',np.mean(scores))

Output:

Scores [0.9464285714285714, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
Mean 0.994642857143
CV Mean 0.964998703656

This is exactly what we observe in the warm start runs on OpenML.
If you reinitialize the random forest within the for loop, you get this:

Scores [0.9107142857142857, 0.9464285714285714, 0.8928571428571429, 1.0, 1.0, 0.9821428571428571, 0.9642857142857143, 0.9464285714285714, 0.9821428571428571, 1.0]
Mean 0.9625
CV Mean 0.963183821623

Much more consistent with sklearn's CV estimate

[joaquinvanschoren]

And thanks to Janos Szedelényi for providing the example.

[janvanrijn]

If this is the case we might need to delete all sklearn runs that are on live server right now.

…

On 16 Mar 2017 01:57, "Joaquin Vanschoren" ***@***.***> wrote: And thanks to Janos Szedelényi for providing the example. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#210 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACL7-qMUuF6TM59ZQQVV262p5Ne7k036ks5rmIjagaJpZM4Mb7R4> .

[joaquinvanschoren]

Not really, only the ones that use warm starting? On Thu, Mar 16, 2017 at 11:03 AM janvanrijn <notifications@github.com> wrote:

If this is the case we might need to delete all sklearn runs that are on live server right now. On 16 Mar 2017 01:57, "Joaquin Vanschoren" ***@***.***> wrote: > And thanks to Janos Szedelényi for providing the example. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > < #210 (comment) >, > or mute the thread > < https://github.com/notifications/unsubscribe-auth/ACL7-qMUuF6TM59ZQQVV262p5Ne7k036ks5rmIjagaJpZM4Mb7R4 > > . > — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#210 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABpQVz5dmxO92EBhEEIO1bSZWkhrP2i4ks5rmQkAgaJpZM4Mb7R4> .

-- Thank you, Joaquin

[janvanrijn]

Please thank your student. I don't think we would have noticed this easily.

Fix in branch fix210. I will merge