Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement StopAfterThresholdReached hook #92

Merged
merged 1 commit into from Nov 12, 2015

Conversation

iver56
Copy link
Contributor

@iver56 iver56 commented Nov 8, 2015

This hook stops the training when a certain accuracy, for example 97 %, is reached. Since it might be useful for others, I thought I'd share it.

Example usage:

trainer.add_hook(bs.hooks.StopAfterThresholdReached('validation.Accuracy',
                                                    threshold=0.97,
                                                    criterion='at_least'))

This needs a test and some documentation. If there would be documentation and tests for the existing EarlyStopper, then I could write similar docs and tests for this hook.

@flukeskywalker
Copy link
Collaborator

Thanks! Here are my thoughts:

  1. The behavior for 'min' and 'max' criteria is contrary to EarlyStopper, and I believe incorrect.
  2. 'at_least' and 'at_most' seem confusing (the second one more so). It's probably best to simply use 'min' and 'max', which should be easier to explain in the docs: The hook stops training if the minimum/maximum value reaches the threshold.
  3. threshold should not have a default value.
  4. Can you please add some doc and example usages? :)

@iver56
Copy link
Contributor Author

iver56 commented Nov 8, 2015

I can fix those things. Re "at_least": My thinking went something like "stop when accuracy is at least 97 %".

When it comes to documentation, I think I'll wait until there's documentation with examples for EarlyStopper. Then I can copy, paste and adapt so the documentation style becomes consistent.

@flukeskywalker
Copy link
Collaborator

Yes, 'at least' parses fine, but saying stop when loss is at most 0.1 doesn't really work well.

@landscape-bot
Copy link

Code Health
Repository health decreased by 0.02% when pulling 286a20b on iver56:threshold-hook into 8ab60fa on IDSIA:master.

@landscape-bot
Copy link

Code Health
Repository health decreased by 0.02% when pulling d2a92f4 on iver56:threshold-hook into f80250c on IDSIA:master.

@landscape-bot
Copy link

Code Health
Repository health decreased by 0.04% when pulling 404f97a on iver56:threshold-hook into f80250c on IDSIA:master.

@iver56
Copy link
Contributor Author

iver56 commented Nov 12, 2015

@flukeskywalker I've fixed the things you mentioned

@landscape-bot
Copy link

Code Health
Repository health decreased by 0.04% when pulling 81be287 on iver56:threshold-hook into f80250c on IDSIA:master.

flukeskywalker added a commit that referenced this pull request Nov 12, 2015
Implement StopAfterThresholdReached hook
@flukeskywalker flukeskywalker merged commit f273421 into IDSIA:master Nov 12, 2015
@flukeskywalker
Copy link
Collaborator

Thanks!

@iver56 iver56 deleted the threshold-hook branch November 12, 2015 15:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants