-
Notifications
You must be signed in to change notification settings - Fork 187
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[WIP] Drop last in train #411
[WIP] Drop last in train #411
Conversation
Codecov Report
@@ Coverage Diff @@
## master #411 +/- ##
==========================================
+ Coverage 84.17% 84.18% +0.01%
==========================================
Files 46 46
Lines 3601 3617 +16
==========================================
+ Hits 3031 3045 +14
- Misses 570 572 +2 |
what experiment on your end did motivate you change this default? did you observe issues without? |
So actually when writing original HBM paper (https://onlinelibrary.wiley.com/doi/full/10.1002/hbm.23730), this issue caused the longest "bughunt" I ever had (~about 2 weeks). One subject had a dataset size that led to the final training batch having size 1, which would cause the loss to randomly diverge after some random number of epochs (let's say 200 epochs everything fine, then suddenly diverge). So should have maybe done this much earlier, but it is what it is. |
Ok I am convinced :)
… Message ID: ***@***.***>
|
ok then just still to consider whether to catch the case with batch_size < first batch size, so case where this would mean no training. In general I think in this case anyways one should be made aware that the requested batch size is impossible as dataset too small just question how to solve this nicely |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
All changes seems ok to me. No change in logic. Robin, can you update the whats new file?
We were getting this in our experiments if you remember well @agramfort :) |
so with regard to giving the user some error/warning in case batch size < min batch size, should we do this? I think it is not super trivial, probably on call to fit(), one would need to check the lengths of the train loader iterator and then give some warning (or error?) if it is 0... should we still do this now or defer to 0.8 even? Skorch doesn't raise an error in that case and runs without training and without computing training metrics... |
It looks like this is more unexpected behaviour for the skorch team than it does for us. For me, we don't do anything in version 0.7 and try to solve it together (braindecode and skorch) for version 0.8 with a warning. |
Can we merge @robintibor? |
d206042
to
cf962d8
Compare
yes feel free to merge then @bruAristimunha |
Closes #16
This PR would by default drop the last batch of the training set during training if it is smaller than the given batch size (happens if #examples in training set is not exactly divisible by the batch size). This would affect existing training code, i.e., might change results (hence tests had to be updated), but I think it is such a sensible default it should be on by default...
If you give a batch size that is larger than the entire training set, the behavior might be somewhat unexpected (training loop is simply skipped in skorch, but it does not crash, just no training happens)... But might be such a rare case we can live with it? otherwise we would have to catch that on call of fit I guess..
What do you think @bruAristimunha @agramfort @gemeinl @sliwy ?