AutoML: Provide leaderboard extension with complexity info: model training time, model scoring time

[exalate-issue-sync]

To help users to assess the complexity of models produced by {{AutoML}}, we can provide a leaderboard extension (not rendered by default) with the following information:

• model training time (in seconds): excludes CV training time.
• model average scoring time per row (in millis), based on prediction of leaderboard frame if provided otherwise on the entire training frame (requires additional computation, so only executed on demand):

In a second time (TBD):

• model size (in bytes)

h2.
Final client API

h3. Python

{noformat}lb_all = h2o.automl.get_leaderboard(aml, extra_columns = 'ALL')
lb_custom = h2o.automl.get_leaderboard(aml, ['predict_time_per_row_ms', 'training_time_ms'])
lb_custom_sorted = lb_custom.sort(by='predict_time_per_row_ms')
{noformat}

h3. R

{noformat}lb_all <- h2o.get_leaderboard(object = aml, extra_columns = 'ALL')
lb_custom <- h2o.get_leaderboard(aml, c('predict_time_per_row_ms', 'training_time_ms'))
{noformat}

[exalate-issue-sync]

Erin LeDell commented: Hey [~accountid:557058:9328661f-241f-4a0f-9d9a-d4e78ef05ba0] thought you may be interested in this, recently added to 3.28.0.1.

[exalate-issue-sync]

Ruslan Dautkhanov commented: Thank you Erin - that’s definitely very interesting. Didn’t know 3.28 is out.

[h2o-ops]

JIRA Issue Migration Info

Jira Issue: PUBDEV-7074
Assignee: Sebastien Poirier
Reporter: Sebastien Poirier
State: Closed
Fix Version: 3.28.0.1
Attachments: N/A
Development PRs: Available

Linked PRs from JIRA

#4124