Feature: Seperate Cost Function and Test Metric for Runhistory #981
Description
Hello,
I'm looking to implement a custom loss/score function to train models on but to also have a separate metric that assess the model during the run, specifically the data available accessible through model.automl_.runhistory_.data and used in the plotting example show here. I'm not sure if this is possible through a custom SMAC configuration but a high level Python interface could be quite helpful .
Use Case:
I'd like to use autosklearn's runhistory to be able to plot its score on the test set provided by model.fit(X, y, X_Test, y_test) but have this score metric be separate from the metric used to train and find suitable models.
At the moment, I am using autosklearn to predict a multiclass classification vector. For test scoring, the model must only get one of the positive labels correct to be sufficient i.e.
import numpy as np
test_scorer = lambda y, y_pred: np.any(y & y_pred).astype(int)
label = np.asarray([0, 1, 1, 0])
pred = np.asarray([0, 1, 0, 0])
test_scorer(label, pred) # == 1If I were to use this test_scorer as training metric, a model could achieve 100% of the possible score by simply predicting [1,1,1,1] for all samples.
However, in training, I would still like the model to be trained using the default f1_macro or using the current method for provided a custom_scorer as documented here.
from autosklearn.metrics import f1_macro
f1_macro(label, pred) # == 0.733...Apologies if this is possible but I could not find how in either the documentation or the source code, feel free to close this if it is already possible.