TPOTClassifier parameter scoring bug

[PracticalDataSc]

I set scoring to "roc_auc" among others parameters:

tpot = TPOTClassifier(scoring='roc_auc', max_time_mins=120, verbosity=3, random_state=RANDOM_STATE, n_jobs=-1)
print(tpot)

but when i print the object configuration, the scoring is None:

30 operators have been imported by TPOT.
TPOTClassifier(config_dict={'sklearn.naive_bayes.GaussianNB': {}, 'sklearn.naive_bayes.BernoulliNB': {'alpha': [0.001, 0.01, 0.1, 1.0, 10.0, 100.0], 'fit_prior': [True, False]}, 'sklearn.naive_bayes.MultinomialNB': {'alpha': [0.001, 0.01, 0.1, 1.0, 10.0, 100.0], 'fit_prior': [True, False]}, 'sklearn.tree.DecisionT....3 , 0.35, 0.4 , 0.45, 0.5 , 0.55,
0.6 , 0.65, 0.7 , 0.75, 0.8 , 0.85, 0.9 , 0.95, 1. ])}}}},
crossover_rate=0.1, cv=5, disable_update_check=False,
early_stop=None, generations=1000000, max_eval_time_mins=5,
max_time_mins=120, memory=None, mutation_rate=0.9, n_jobs=4,
offspring_size=100, periodic_checkpoint_folder=None,
population_size=100, random_state=42, scoring=None, subsample=1.0,
verbosity=3, warm_start=False)

i've also tried with others estimators from: https://epistasislab.github.io/tpot/api/ such as: 'balanced_accuracy', 'f1' and the result is still None.

Context of the issue

I'm running the code in:

• conda version 4.4.8
• python v 3.6.5
• tpot v 0.9.3
• sklearn v 0.19.1

I had also replicated the process in google colab (python 3) and the same issue is happening.

Process to reproduce the issue

1. User creates TPOT instance and change the parameter "scoring" from default
2. User print TPOT object

Expected result

To change the object default configuration. In the previous example:

30 operators have been imported by TPOT.
TPOTClassifier(config_dict={'sklearn.naive_bayes.GaussianNB': {}, 'sklearn.naive_bayes.BernoulliNB': {'alpha': [0.001, 0.01, 0.1, 1.0, 10.0, 100.0], 'fit_prior': [True, False]}, 'sklearn.naive_bayes.MultinomialNB': {'alpha': [0.001, 0.01, 0.1, 1.0, 10.0, 100.0], 'fit_prior': [True, False]}, 'sklearn.tree.DecisionT....3 , 0.35, 0.4 , 0.45, 0.5 , 0.55,
0.6 , 0.65, 0.7 , 0.75, 0.8 , 0.85, 0.9 , 0.95, 1. ])}}}},
crossover_rate=0.1, cv=5, disable_update_check=False,
early_stop=None, generations=1000000, max_eval_time_mins=5,
max_time_mins=120, memory=None, mutation_rate=0.9, n_jobs=4,
offspring_size=100, periodic_checkpoint_folder=None,
population_size=100, random_state=42, scoring=roc_auc, subsample=1.0,
verbosity=3, warm_start=False)

[weixuanfu]

Thank you for reporting this issue. TPOT internally didn't use self.scoring but uses self.scoring_function generated by _setup_scoring_function and scoring parameter in pipeline evaluation, so I think TPOT uses the right scoring function as expected during optimization. But, indeed, this is a API bug related to #739. I think I fixed this API bug in PR #740 and we will fixed this issue in the next version of TPOT.

[weixuanfu]

This issue should be fixed in TPOT 0.9.4. Please feel free to reopen this issue if you have any questions.