Enable sorting by custom metrics in AutoML leaderboard #6779
Assignees
Milestone
Comments
|
Wendy Wong commented: customer support ticket: https://support.h2o.ai/a/tickets/104467 |
|
JIRA Issue Details Jira Issue: PUBDEV-9015 |
This was referenced Jun 8, 2023
tomasfryda
added a commit
that referenced
this issue
Jul 13, 2023
* Add custom metric to leaderboard * Verify sorting by custom metric * Make sure the custom metrics share the same names * Fix custom metric calculation in CV (#15576) * GH-15565: Add custom metric to automl [nocheck] (#15577) * Fix custom metric calculation in CV (prototype) * Allow custom_metric_func in automl * Return NaN for models that don't support custom metric * Revert change used to ensure existing metric * GH-15559: Add custom metric to SE [nocheck] (#15579) * Fix custom metric calculation in CV (prototype) * Allow custom_metric_func in automl * Return NaN for models that don't support custom metric * Revert change used to ensure existing metric * Add custom metric to SE * Mention SE support of the custom metric func in docs * Use directly model metrics in leaderboard This change is motivated by an issue with SE level-one validation frame that is ephemeral and can not be checksummed.
maurever
pushed a commit
that referenced
this issue
Sep 6, 2023
* Add custom metric to leaderboard * Verify sorting by custom metric * Make sure the custom metrics share the same names * Fix custom metric calculation in CV (#15576) * GH-15565: Add custom metric to automl [nocheck] (#15577) * Fix custom metric calculation in CV (prototype) * Allow custom_metric_func in automl * Return NaN for models that don't support custom metric * Revert change used to ensure existing metric * GH-15559: Add custom metric to SE [nocheck] (#15579) * Fix custom metric calculation in CV (prototype) * Allow custom_metric_func in automl * Return NaN for models that don't support custom metric * Revert change used to ensure existing metric * Add custom metric to SE * Mention SE support of the custom metric func in docs * Use directly model metrics in leaderboard This change is motivated by an issue with SE level-one validation frame that is ephemeral and can not be checksummed.
maurever
pushed a commit
that referenced
this issue
Sep 6, 2023
* Add custom metric to leaderboard * Verify sorting by custom metric * Make sure the custom metrics share the same names * Fix custom metric calculation in CV (#15576) * GH-15565: Add custom metric to automl [nocheck] (#15577) * Fix custom metric calculation in CV (prototype) * Allow custom_metric_func in automl * Return NaN for models that don't support custom metric * Revert change used to ensure existing metric * GH-15559: Add custom metric to SE [nocheck] (#15579) * Fix custom metric calculation in CV (prototype) * Allow custom_metric_func in automl * Return NaN for models that don't support custom metric * Revert change used to ensure existing metric * Add custom metric to SE * Mention SE support of the custom metric func in docs * Use directly model metrics in leaderboard This change is motivated by an issue with SE level-one validation frame that is ephemeral and can not be checksummed.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
No description provided.
The text was updated successfully, but these errors were encountered: