support save snapshot by iteration #1204
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kan-bayashi
reviewed
Sep 23, 2019
espnet/asr/pytorch_backend/asr.py
Outdated
| @@ -428,11 +429,11 @@ def train(args): | |||
| # we used an empty collate function instead which returns list | |||
| train_iter = {'main': ChainerDataLoader( | |||
| dataset=TransformDataset(train, lambda data: converter([load_tr(data)])), | |||
| batch_size=1, num_workers=args.n_iter_processes, | |||
| batch_size=1, num_workers=args.n_iter_processes, pin_memory=True, | |||
There was a problem hiding this comment.
Might be better not to include this PR.
espnet/asr/pytorch_backend/asr.py
Outdated
| shuffle=not use_sortagrad, collate_fn=lambda x: x[0])} | ||
| valid_iter = {'main': ChainerDataLoader( | ||
| dataset=TransformDataset(valid, lambda data: converter([load_cv(data)])), | ||
| batch_size=1, shuffle=False, collate_fn=lambda x: x[0], | ||
| batch_size=1, pin_memory=True, shuffle=False, collate_fn=lambda x: x[0], |
espnet/asr/asr_utils.py
Outdated
| @@ -284,6 +284,21 @@ def torch_snapshot(trainer): | |||
| return torch_snapshot | |||
|
|
|||
|
|
|||
| def torch_snapshot_iter(savefun=torch.save, | |||
| filename='snapshot.iter.{.updater.iteration}'): | |||
There was a problem hiding this comment.
I think that it is not necessary to make a new function.
Just reuse torch_snapshot is fine.
espnet/asr/pytorch_backend/asr.py
Outdated
| @@ -490,6 +494,8 @@ def train(args): | |||
|
|
|||
| # save snapshot which contains model and optimizer states | |||
| trainer.extend(torch_snapshot(), trigger=(1, 'epoch')) | |||
| if args.save_interval_iters > 0: | |||
| trainer.extend(torch_snapshot_iter(), trigger=(args.save_interval_iters, 'iteration')) | |||
There was a problem hiding this comment.
Why don't you reuse torch_snapshot instead of torch_snapshot_iter?
Suggested change
| trainer.extend(torch_snapshot_iter(), trigger=(args.save_interval_iters, 'iteration')) | |
| trainer.extend(torch_snapshot(filename='snapshot.iter.{.updater.iteration}'), | |
| trigger=(args.save_interval_iters, 'iteration')) |
Codecov Report
@@ Coverage Diff @@
## v.0.6.0 #1204 +/- ##
===========================================
+ Coverage 78.32% 78.32% +<.01%
===========================================
Files 100 100
Lines 9295 9296 +1
===========================================
+ Hits 7280 7281 +1
Misses 2015 2015
Continue to review full report at Codecov.
|
sw005320
reviewed
Sep 23, 2019
sw005320
reviewed
Sep 23, 2019
|
Thanks a lot! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
the epoch num may not be very large on large dataset training because of time limit. And average_checkpoints is very important to achieve lower CER. So I suspect that it's necessary to support save snapshot when iterations(10000,20000 etc.).
Any idea will be appreciate, thanks