AutoML: Change default leaderboard sorting in regression to RMSE

[exalate-issue-sync]

This is frankly the best choice for a default regression metric and we probably should have been using all along. However, we decided to go with mean residual deviance since that’s the default early stopping metric in regression. In most cases, mean residual deviance is equivalent to MSE anyway, so it doesn’t change the rankings of the models on the leaderboard. This will result in the rmse column becoming the first column on the leaderboard in regression problems.

And it also solves an issue that we just found when adding the {{distribution}} parameter. In the case where we expose the {{distribution}} param in AutoML and allow the user to use Tweedie, on a few but not all models, then we have an issue that arises with using “mean residual deviance” as the default sort metric on the leaderboard. The two types of models (Tweedie vs Gaussian) will have different scales for their deviance and therefore should not be compared.

!Screen Shot 2022-01-17 at 14.44.23.png|width=1009,height=251!

[h2o-ops-ro]

JIRA Issue Details

Jira Issue: PUBDEV-8528
Assignee: Tomas Fryda
Reporter: Erin LeDell
State: Resolved
Fix Version: 3.36.1.1
Attachments: Available (Count: 1)
Development PRs: Available

[h2o-ops-ro]

Attachments From Jira

Attachment Name: Screen Shot 2022-01-17 at 14.44.23.png
Attached By: Erin LeDell
File Link:https://h2o-3-jira-github-migration.s3.amazonaws.com/PUBDEV-8528/Screen Shot 2022-01-17 at 14.44.23.png

[h2o-ops-ro]

Linked PRs from JIRA

#6050