Setting random_state and np.random.seed does not ensure reproducibility

[maxnoe]

I think it would be great and make things a lot easier, if there would be a top level API for scikit-learn

scikit-learn.set_random_seed

This would help a lot for reproducibility as one would not have to remember setting random states for each algorithm that is called. This has to deal with multiprocessing though I guess.

[rth]

This was previously requested in #5781 and the solution (i.e. using numpy global random seed) is documented in the FAQ.

[maxnoe]

I'm asking, because right now I have problems with reproducibility. I set the np.random.seed as well as each algorithms random state, however the results are still a bit different each time a run the scripts.

[maxnoe]

See for example https://github.com/fact-project/classifier-tools/blob/random_seed/klaas/scripts/train_separation_model.py

[rth]

See for example https://github.com/fact-project/classifier-tools/blob/random_seed/klaas/scripts/train_separation_model.py

Hmm, could you please provide a minimal example together with a sample dataset, that wouldn't require installing all the imported dependencies?

[maxnoe]

Yes, I provided it here:

https://factdata.app.tu-dortmund.de/sklearn_example

When I run it three times, I always get slightly different roc aucs:

$ python example.py 
0.868710206499
$ python example.py 
0.868723414987
$ python example.py 
0.868695037329

[maxnoe]

import pandas as pd
import numpy as np

from sklearn.metrics import roc_auc_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import StratifiedKFold

seed = 0

columns = [
    'conc_core',
    'concentration_one_pixel',
    'concentration_two_pixel',
    'leakage',
    'leakage2',
    'size',
    'width',
    'length',
    'conc_cog',
    'm3l',
    'm3t',
    'num_islands',
    'num_pixel_in_shower',
    'ph_charge_shower_max',
    'ph_charge_shower_mean',
    'ph_charge_shower_min',
    'ph_charge_shower_variance',
]


if __name__ == '__main__':
    np.random.seed(seed)

    gammas = pd.read_hdf('gammas.hdf5')
    protons = pd.read_hdf('protons.hdf5')

    gammas['label'] = 1
    protons['label'] = 0

    df = pd.concat([gammas, protons], ignore_index=True)

    X = df[columns].values
    y = df['label'].values

    stratified_kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=seed)

    classifier = RandomForestClassifier(n_estimators=100, n_jobs=-1)
    classifier.random_state = seed  # set it here to be compatible to the original script

    aucs = []
    for fold, (train, test) in enumerate(stratified_kfold.split(X, y)):
        # select data
        xtrain, xtest = X[train], X[test]
        ytrain, ytest = y[train], y[test]

        # fit and predict
        classifier.fit(xtrain, ytrain)

        y_probas = classifier.predict_proba(xtest)[:, 1]
        aucs.append(roc_auc_score(ytest, y_probas))

    print(np.mean(aucs))

[maxnoe]

This looks like a multiprocessing issue. When I run this with n_jobs=1 It seems that I always get the same result.

[rth]

Thanks. Could you please provide the data as well? The tu-dortmund link asks me for authentification...

[maxnoe]

Sorry, I forgot to remove the passwordprotection. Should be public now.

[jnothman]

@maxnoe: When you submitted your issue, you were asked to report what version of scikit-learn you are using. If it is version 0.19.0, and not 0.19.1, I'm guessing this was fixed by #9830, and you should get yourself the most recent release.