fit with buildin metrics error

[shm007g]

automl.fit(X_train, y_train, metric=autosklearn.metrics.f1) failed.

[WARNING] [2018-02-05 15:25:37,152:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] Error loading predictions_ensemble_1_00001.npy: <class 'ValueError'> - Target is multiclass but average='binary'. Please choose another average setting.
[ERROR] [2018-02-05 15:25:37,154:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] Caught ValueError: Target is multiclass but average='binary'. Please choose another average setting.
[WARNING] [2018-02-05 15:25:37,619:smac.intensification.intensification.Intensifier] Challenger was the same as the current incumbent; Skipping challenger
[WARNING] [2018-02-05 15:25:37,619:smac.intensification.intensification.Intensifier] Challenger was the same as the current incumbent; Skipping challenger
---------------------------------------------------------------------------
KeyError   
...
...
KeyError: None

In [13]: [WARNING] [2018-02-05 15:25:39,164:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] Error loading predictions_ens.
[ERROR] [2018-02-05 15:25:39,170:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] Caught ValueError: Target is multiclass .
[WARNING] [2018-02-05 15:25:41,180:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] Error loading predictions_ensemble_1_0.
[ERROR] [2018-02-05 15:25:41,185:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] Caught ValueError: Target is multiclass .
[WARNING] [2018-02-05 15:25:43,204:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] Error loading predictions_ensemble_1_0.
[ERROR] [2018-02-05 15:25:43,209:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] Caught ValueError: Target is multiclass .
[WARNING] [2018-02-05 15:25:45,215:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] Error loading predictions_ensemble_1_0.
[ERROR] [2018-02-05 15:25:45,218:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] Caught ValueError: Target is multiclass .
[WARNING] [2018-02-05 15:25:47,239:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] Error loading predictions_ensemble_1_0.
[ERROR]

[shm007g]

also test precision and roc, all fail with same problem.
seems only support default accuracy.

[mfeurer]

I must agree that our metrics aren't document well enough and we really need to improve on that. Basically, the issue boils down to f1 not being defined for multiclass. In order to apply f1 to multiclass, you need to define an averaging mechanism as explained in the scikit-learn API. Here are all possible metrics which can be used in Auto-sklearn, and you can follow this example:

import sklearn.model_selection
import sklearn.datasets
import sklearn.metrics

import autosklearn.classification
import autosklearn.metrics


def main():
    X, y = sklearn.datasets.load_digits(return_X_y=True)
    X_train, X_test, y_train, y_test = \
        sklearn.model_selection.train_test_split(X, y, random_state=1)

    automl = autosklearn.classification.AutoSklearnClassifier(
        time_left_for_this_task=120,
        per_run_time_limit=30,
        tmp_folder='/tmp/autosklearn_holdout_example_tmp',
        output_folder='/tmp/autosklearn_holdout_example_out',
        disable_evaluator_output=False,
    )
    automl.fit(
        X_train,
        y_train,
        dataset_name='digits',
        metric=autosklearn.metrics.f1_macro
    )

    # Print the final ensemble constructed by auto-sklearn.
    print(automl.show_models())
    predictions = automl.predict(X_test)
    # Print statistics about the auto-sklearn run such as number of
    # iterations, number of models failed with a time out.
    print(automl.sprint_statistics())
    print("Accuracy score", sklearn.metrics.f1_score(
        y_test, predictions, average='macro'
    ))


if __name__ == '__main__':
    main()

[kumarishikha1]

Is this issue already fixed? I am working on a binary classification task and I get the same error.