Tuner has no scores attribute until record is run once

[jcmgray]

• BTB version: master
• Python version: 3.7
• Operating System: linux

Description

The BaseTuner has no scores attribute (just raw_scores) until record is run at least once. This means, for example, that without modifcation the readme snippet errors when run. Is this intentional? Obviously handling the first few trials manually is only mildly inconvenient (gettattr(tuner, 'scores', [])) but would be easy to fix if desired!

What I Did

(from readme)

from btb.tuning import Tunable
from btb.tuning.tuners import GPTuner
from btb.tuning.hyperparams import IntHyperParam
hyperparams = {
    'n_estimators': IntHyperParam(min=10, max=500),
    'max_depth': IntHyperParam(min=10, max=500),
}
tunable = Tunable(hyperparams)
tuner = GPTuner(tunable)

Set-up selector.

from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from btb.selection import UCB1
from btb.tuning.hyperparams import FloatHyperParam
models = {
    'RF': RandomForestClassifier,
    'SVC': SVC
}
selector = UCB1(['RF', 'SVC'])
rf_hyperparams = {
    'n_estimators': IntHyperParam(min=10, max=500),
    'max_depth': IntHyperParam(min=3, max=20)
}
rf_tunable = Tunable(rf_hyperparams)
svc_hyperparams = {
    'C': FloatHyperParam(min=0.01, max=10.0),
    'gamma': FloatHyperParam(0.000000001, 0.0000001)
}
svc_tunable = Tunable(svc_hyperparams)
tuners = {
    'RF': GPTuner(rf_tunable),
    'SVC': GPTuner(svc_tunable)
}

Try and get first candidate:

next_choice = selector.select({
    'RF': tuners['RF'].scores,
    'SVC': tuners['SVC'].scores
})
# AttributeError: 'GPTuner' object has no attribute 'scores'

[pvk-developer]

Hello @jcmgray ,

Thank you for detecting and reporting the issue! It is not intended to be like this, and during the latest changes and because of some naming discussions we completely forgot about scores.

Just if you are wondering: raw_scores (back in the day named scores) was used to save the scores that the user gives as an input and then we had another _scores which then had the normalized score (in case your problem is maximize or minimize).

I'm fixing this right now, and thank you once more.

[jcmgray]

Great, thanks for the prompt fix!

[pvk-developer]

Great, thanks for the prompt fix!

It will be fixed on pipy with the following release.