Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Save optimizer state along with model checkpoints #203

Closed
frostedoyster opened this issue May 27, 2024 · 1 comment · Fixed by #262
Closed

Save optimizer state along with model checkpoints #203

frostedoyster opened this issue May 27, 2024 · 1 comment · Fixed by #262
Assignees
@frostedoyster
Labels
Alchemical Model Alchemical model experimental architecture Discussion Issues to be discussed by the contributors Priority: High Critical issues needing immediate attention. SOAP BPNN SOAP BPNN experimental architecture

Comments

@frostedoyster
Copy link
Collaborator

Not saving the optimizer state leads to some big jumps when using Adam, undoing most of the optimization done until that point. This is important when continuing training with the exact same dataset. In other cases (e.g. fine tuning), I believe it would still be beneficial (although we could have a flag to disable it).
@frostedoyster frostedoyster added SOAP BPNN SOAP BPNN experimental architecture Alchemical Model Alchemical model experimental architecture labels May 27, 2024
@jwa7
Copy link
Member

jwa7 commented May 28, 2024

+1 but support loading state dicts of schedulers too

@PicoCentauri PicoCentauri added the Priority: High Critical issues needing immediate attention. label Jun 3, 2024
@frostedoyster frostedoyster self-assigned this Jun 3, 2024
@frostedoyster frostedoyster added the Discussion Issues to be discussed by the contributors label Jun 11, 2024
@frostedoyster frostedoyster mentioned this issue Jun 14, 2024
Merged
3 tasks
@frostedoyster frostedoyster linked a pull request Jun 14, 2024 that will close this issue
Move saving checkpoints from model to trainer #262
Merged
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@frostedoyster frostedoyster

Labels
Alchemical Model Alchemical model experimental architecture Discussion Issues to be discussed by the contributors Priority: High Critical issues needing immediate attention. SOAP BPNN SOAP BPNN experimental architecture
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Move saving checkpoints from model to trainer lab-cosmo/metatrain
3 participants
@PicoCentauri @jwa7 @frostedoyster