Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add checkpointing for training #188

Merged
merged 24 commits into from
Jan 22, 2020
Merged

Add checkpointing for training #188

merged 24 commits into from
Jan 22, 2020

Conversation

tanaysoni
Copy link
Contributor

No description provided.

tanaysoni and others added 17 commits January 17, 2020 18:03
@tanaysoni
Add save and load of checkpoints during training
e71cb9a
@tanaysoni
Refactor train checkpointing
e04c7be
@brandenchan @tanaysoni
Start implementation
8c8a551
@brandenchan @tanaysoni
latest hyperparams
bc1dc0c
@tanaysoni
Add dataset checkpointing
e49822b
@tanaysoni
Fix vocab_size initialization in BertConfig
d396e2d
@brandenchan @tanaysoni
Fix lm pretraining
5e87492
@tanaysoni
Remove in-place key pop for opts dict in scheduler
83a36b3
@tanaysoni
Add docs and clean-up code
1eb5e6a
@tanaysoni
Fix steps iteration in resuming train from a checkpoint
ade4722
@tanaysoni
Add calculation of global_step when resuming trainining from a checkp…
7266a95 
…oint
@tanaysoni
Add log_learning_rate param in the state_dict of Trainer
1a19b7e
@tanaysoni
Fix resume step when starting to train from a checkpoint
06b96d1
@tholor @tanaysoni
rename and update basic example
890cdae
@tanaysoni
Add additional states in train checkpoint
30999b3
@tanaysoni
Fix Trainer init arguments
9f11bc3
@tanaysoni
Set initial epoch and set to 0
0227a65
tholor
tholor requested changes Jan 22, 2020
View reviewed changes
Copy link
Member

@tholor tholor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only minor comments.

examples/train_from_scratch.py Outdated Show resolved Hide resolved
farm/modeling/language_model.py Outdated Show resolved Hide resolved
farm/train.py Outdated Show resolved Hide resolved
@tanaysoni tanaysoni merged commit 4d83f4a into master Jan 22, 2020
@tholor tholor mentioned this pull request Jan 22, 2020
Closed
@tholor tholor deleted the train_checkpoint branch April 28, 2020 07:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tholor tholor tholor requested changes

@brandenchan brandenchan Awaiting requested review from brandenchan

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@tanaysoni @tholor @brandenchan