Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Deeplab resume bug: update path in checkpoint file #756

Merged
merged 2 commits into from Apr 10, 2019

Conversation

Projects
None yet
1 participant
@lewfish
Copy link
Contributor

lewfish commented Apr 9, 2019

This PR allows you to resume training using Deeplab. Previously, it would crash because the checkpoint file contained absolute paths to the model checkpoints which were specific to the temporary directory that was used when running remotely. I tested this by downloading the train directory from a remote run that did not complete, ran it locally, and saw that it resumed training from the latest checkpoint.

TODO: make an issue to make an issue to deeplab about this

Screen Shot 2019-04-09 at 5 57 34 PM

Update model_checkpoint_path in checkpoint file
This allows resume training functionality for Deeplab

@lewfish lewfish added the review label Apr 9, 2019

@lewfish lewfish changed the title Fix resume bug: update model_checkpoint_path in checkpoint file Fix Deeplab resume bug: update path in checkpoint file Apr 9, 2019

@lewfish lewfish merged commit b3478ec into master Apr 10, 2019

@lewfish lewfish removed the review label Apr 10, 2019

@lewfish lewfish referenced this pull request Apr 12, 2019

Merged

Fix checkpoint bug #763

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.