AutoMM early stopping with a threshold #2633
Conversation
|
Job PR-2633-3712bb1 is done. |
| @@ -1261,6 +1261,7 @@ def _fit( | |||
| monitor=task.validation_metric_name, | |||
| patience=config.optimization.patience, | |||
| mode=minmax_mode, | |||
| stopping_threshold=1 - 1e-6 if minmax_mode == MAX else 1e-6, | |||
There was a problem hiding this comment.
Are we assuming all positive-is-better metrics have an optimal score of 1? I guess in general this is true, but it is a pretty flimsy assumption especially if a user can provide a custom metric. Ditto for assuming optimal score of 0 for negative-is-better metrics.
These situations are handled in my autogluon.core.metrics logic, where each metric has a defined optimum value.
There was a problem hiding this comment.
Beyond the above point, the fact we need an if here seems like unnecessary code bloat and increases risk of bugs. It would be better if the optimum value / minmax_mode was simply provided by the metric itself, rather than hard-coded into this line.
There was a problem hiding this comment.
for example:
stopping_threshold = metric.optimum - metric.sign * epsilon
which translates to:
stopping_threshold = 1 - (1 * 1e-6)
I think this would be a lot cleaner.
Both of these things are provided in autogluon.core.metrics.
from autogluon.core.metrics import roc_auc
epsilon = 1e-6
stopping_threshold = roc_auc.optimum - roc_auc._sign * epsilon
print(stopping_threshold)0.999999
There was a problem hiding this comment.
Good suggestion. Thanks.
|
Job PR-2633-fd4f17c is done. |
Issue #, if available:
#2555 #2629
Description of changes:
Early stopping if reaching a perfect score. Can't use exactly 1 or 0 due to the precision issues. Tested it with the example in #2629.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.