Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How do I alter the save-interval when training? #244

Closed
1 task done
initdebugs opened this issue Jan 11, 2023 · 6 comments
Closed
1 task done

How do I alter the save-interval when training? #244

initdebugs opened this issue Jan 11, 2023 · 6 comments
Labels
question Further information is requested

Comments

@initdebugs
Copy link

Search before asking

Question

When trying to train a Object Detection model. In the previous version of Yolo you could pass the --save-interval argument to tell the script after how many epochs to save the model.
How would I do that in Yolov8? Can't seem to find it.

Additional

No response

@initdebugs initdebugs added the question Further information is requested label Jan 11, 2023
@AyushExel
Copy link
Contributor

@initdebugs hey, right now the local weights best.pt and last.pt are updated after every epoch and artifacts are logged to supported logging platforms only after training finishes and models are stripped. Do you want a similar logging feature to log checkpoints in order to save progress?
We are working on better online logging support through HUB using which you can resume training from anywhere, even a different system. If your use case is different, please let us know so that we can support it better. thanks!

@initdebugs
Copy link
Author

Okay that seems odd. For me it saved a best.pt and a last.pt after the first epoch. After that it hasn't updated those files. It's currently at epoch 78 but hasn't saved it.

@AyushExel
Copy link
Contributor

@initdebugs okay that is not expected. It might be a bug. Can you confirm if the ckpt have not been updated by doing this.

import torch

ckpt = torch.load("last.pt")
print(ckpt["epoch"])

Thanks!

@initdebugs
Copy link
Author

It shows the correct epoch when doing that. Does that mean that it does save it? The 'last modified' datetime for the file hasn't changed.

@AyushExel
Copy link
Contributor

@initdebugs yeah it shows the epoch it was saved on. Maybe it's something with how OS's file managers check last changed? But it should not be something to worry about

@initdebugs
Copy link
Author

Thank you for the help. I appreciate the work you guys are putting into this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants