-
-
Notifications
You must be signed in to change notification settings - Fork 657
Checkpoint: fix returning worst model of the saved models. #745
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
@chm90 thanks for the PR! I need to recheck that. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@chm90 could you please add an explicit test for checking that. Thanks!
- minor improvement of typing
@chm90 as it is a bug, I added tests by myself. So, thanks again for catching it ! |
Thanks @vfdev-5, I didn't have time to implement tests yet. Thank's for implementing |
@vfdev-5, do I need to do anything else for you to be able to merge the pr? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@chm90 thanks again for the catch!
Hi @vfdev-5 , Have you added this bug fix to nightly build? |
Hi @Nic-Ma This commit will be normally added to the nighlty build today (30/01/2020). |
Hi @vfdev-5 , Thanks for your information. Is it compatible with Python 3.5? |
@vfdev-5 , And BTW, I found some small bugs in current checkpoint handler:
For example, the saved model filename for epoch 2 will be: Could you please have a look? |
@Nic-Ma thanks for the feedback, seems like it is a regression we introduced with typing. Sorry about that. We will fix it ASAP. About the suffix, yes that's right, it should be related to another PR where such case was somehow out of scope. I'll take a look on that. Thanks again for reporting these things! |
@vfdev-5 , Cool, thanks for your quick fix. Anyway, we will still use ignite 0.3.0 until the 0.4.0 version is stable and officially released. |
@Nic-Ma probably this should be related with an issue and its fix here: Line 15 in b593817
So, I would say that in python 3.7 you most probably should not have the issue. But, it would depend on what kind of extension you made in torch.DataLoader .
Could you please open a separate issue and put a small snippet to reproduce the problem from our side. Or pointout an example if it is in your MONAI project. Thanks! |
Hi @vfdev-5 , Thanks for your tips. |
@Nic-Ma I see, thanks. If you could however share some more details on your extended DataLoader (e.g. a snippet or just a description of what kind of extension you made) such that we could create some tests for that and ensure that ignite works for various cases. Thanks
Certainly, we will release several 0.3.X versions before 0.4.0 |
Fixes #
returning the model with the lowest priority of the saved models
Description:
The
self._saved
list is always sorted in an ascending order(ensured by sorting it each time a new element is added). Hence, returningself._saved[0]
inlast_checkpoint
actually yields the oldest checkpoint of the saved ones, or the worst if a score function is provided. Changing the above toself._saved[-1]
yields the expected result, i.e, the model with the highest priority.Check list: