Calculate multiple metrics

[rabsr]

This change will provide users an option scoring_functions to calculate list of metrics for each pipeline. The calculated metrics will be available from attribute cv_results_.
This will provide users with more insights for each pipeline executed.

Usage:

automl = autosklearn.classification.AutoSklearnClassifier(
    time_left_for_this_task=30,
    per_run_time_limit=10,
    scoring_functions=['accuracy', 'balanced_accuracy', 'log_loss'],)
automl.fit(X_train,y_train)
print(automl.cv_results_)

[mfeurer]

This implements #981 I guess?

[mfeurer]

Thanks a lot for the PR! I left some initial comments you could address, and then it would be good to incrementally work on the following things:

1. rebase on the development branch to fix the current issues CI
2. add an example about tracking auxiliary metrics
3. think about additional unit tests to test this new functionality

Looking forward to the changes

[rabsr]

@mfeurer

This implements #981 I guess?

Yeah. This PR implements #981

Made changes based on review comments and rebased with latest development.

Todo

• additional unit tests to test this new functionality
• add an example about tracking auxiliary metrics

I will be working on these items.

[mfeurer]

This looks great, looking forward to the example and unit tests.

[mfeurer]

A few updates from my side:

• I cannot reproduce the failure for the pandas example locally, so I think everything is fine here. Nevertheless, I restarted the tests on travis-ci.
• The code looks good and I would merge it as is, therefore, you could now move to adding unit tests (I've been thinking about one for the score calculation function, and making the tests in test_train_evaluator stricter by checking the entries in the additional run info, please let me know if you have any questions regarding that) and an example on how to use this new feature.

I'm really looking forward to this new feature as I actually also just found a good use case within our research :)

[codecov]

Codecov Report

Merging #985 (d281763) into development (3743b25) will increase coverage by 0.03%.
The diff coverage is 84.37%.

@@               Coverage Diff               @@
##           development     #985      +/-   ##
===============================================
+ Coverage        85.34%   85.37%   +0.03%     
===============================================
  Files              125      125              
  Lines             9858     9881      +23     
===============================================
+ Hits              8413     8436      +23     
  Misses            1445     1445              

Impacted Files	Coverage Δ
autosklearn/ensemble_builder.py	76.43% <ø> (ø)
autosklearn/ensembles/ensemble_selection.py	68.42% <ø> (ø)
autosklearn/evaluation/test_evaluator.py	90.90% <ø> (ø)
autosklearn/evaluation/train_evaluator.py	72.74% <50.00%> (ø)
autosklearn/evaluation/abstract_evaluator.py	88.42% <80.00%> (ø)
autosklearn/automl.py	84.68% <85.00%> (+0.01%)	⬆️
autosklearn/estimators.py	93.02% <100.00%> (+0.10%)	⬆️
autosklearn/evaluation/__init__.py	82.62% <100.00%> (ø)
autosklearn/smbo.py	82.29% <100.00%> (+0.06%)	⬆️
autosklearn/util/logging_.py	94.30% <0.00%> (-1.63%)	⬇️
... and 4 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3743b25...d281763. Read the comment docs.

[rabsr]

@mfeurer Test cases and example will take some time. Currently I am on vacation for 2 weeks. I will start working on test cases and example once I am back.

I actually also just found a good use case within our research

If possible, can you highlight what is the use case you are looking forward to?

[mfeurer]

Thanks for the heads up @rabsr enjoy your vacation

[mfeurer]

Thanks a lot for the tests! I just made a PR which makes the examples work. How about I merge this one and you start working on a new PR updating the example for using different metrics?

If possible, can you highlight what is the use case you are looking forward to?

I'm planning to use it to generate meta-data for the Auto-sklearn 2.0 project.

[rabsr]

How about I merge this one and you start working on a new PR updating the example for using different metrics?

@mfeurer That works for me. You can merge this PR. I will open a new PR for example in couple of days.