Skip to content

Conversation

tveasey
Copy link
Contributor

@tveasey tveasey commented May 31, 2022

We currently apply a small penalty to prefer selecting small models when the validation loss is similar. This is based on the model size, but is also parameterised by the mean size of all models trained up to each point where we test. The mean size changes through the optimisation loop and means we don't apply a completely consistent penalty when comparing the candidate model with the current best model (whose penalty was calculated using earlier parameters). This change stores the best model size as well so we compute penalties using the same parameters.

Copy link
Contributor

@valeriy42 valeriy42 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. I have a single comment. If this was a copy/paste error, you could just fix it. If not, would you add a quick explanation, please? No need for me to review it again.

@tveasey tveasey merged commit 8825530 into elastic:main Jun 2, 2022
@tveasey tveasey deleted the best-model-selection branch June 2, 2022 16:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants