Merge pull request #756 from azavea/lf/deeplab-checkpoint

Fix Deeplab resume bug: update path in checkpoint file

docs/changelog.rst

@@ -38,6 +38,7 @@ Features
 
 Bug Fixes
 ^^^^^^^^^
+- Fix Deeplab resume bug: update path in checkpoint file `#756 <https://github.com/azavea/raster-vision/pull/756>`_
 - Allow Spaces in ``--channel-order`` Argument `#731 <https://github.com/azavea/raster-vision/pull/731>`_
 - Fix error when using predict packages with AOIs `#674 <https://github.com/azavea/raster-vision/pull/674>`_
 - Correct checkpoint name `#624 <https://github.com/azavea/raster-vision/pull/624>`_

rastervision/backend/tf_deeplab.py

@@ -620,6 +620,17 @@ def train(self, tmp_dir: str) -> None:
                 train_restart_dir
         ) > 0 and not self.backend_config.train_options.replace_model:
             sync_from_dir(train_restart_dir, train_logdir_local)
+
+            # Need to update model_checkpoint_path in the checkpoint file,
+            # since it has the absolute paths from the previous run which
+            # was using a different temporary directory on another machine.
+            # If Deeplab could save relative paths instead (like the Object
+            # Detection API does), then we wouldn't need to do this.
+            latest_checkpoint = get_latest_checkpoint(train_logdir_local)
+            checkpoint_path = join(train_logdir_local, 'checkpoint')
+            with open(checkpoint_path, 'w') as cf:
+                cf.write(
+                    'model_checkpoint_path: \"{}\"'.format(latest_checkpoint))
         else:
             if self.backend_config.train_options.replace_model:
                 if os.path.exists(train_logdir_local):