Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Minimum improvement early stopping callback #51

Open
GilesStrong opened this issue Jun 5, 2020 · 0 comments
Open

Minimum improvement early stopping callback #51

GilesStrong opened this issue Jun 5, 2020 · 0 comments
Labels
enhancement New feature or request good first issue Good for newcomers low priority Not urgent and won't degrade with time

Comments

@GilesStrong
Copy link
Owner

Current state

Main training function fold_train_ensemble uses early stopping by default: if a number of subepochs are passed (patience) without an improvement in the validation loss, then training stops.

Problem

Sometime the validation loss will move to a plateau with a shallow slope; the validation loss continues to decrease, but at rate which minimal impact on performance.

Suggestion

A new callback is introduced to to stop training if the validation loss doesn't decrease by a certain amount (or fraction) after a certain number of epochs. This should ideally automatically scale to the typical loss values for each particular training and should accurately detect plateaus without fine tuning by the user. It could, for instance, monitor the rate of change of loss. MetricLogger already does something similar by monitoring the loss velocity. It should also be resistant to fluctuations in the loss, which may occur, particularity in heavily weighted data.

@GilesStrong GilesStrong added enhancement New feature or request good first issue Good for newcomers low priority Not urgent and won't degrade with time labels Jun 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers low priority Not urgent and won't degrade with time
Projects
None yet
Development

No branches or pull requests

1 participant