Could only save 5 checkpoints w/ model_main.py

[YijinLiu]

System information

• What is the top-level directory of the model you are using:
  research/object_detection
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
  No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
  Ubuntu 18.04
• TensorFlow installed from (source or binary):
  Source
• TensorFlow version (use command below):
  1.10
• Bazel version (if compiling from source):
  0.15.2
• CUDA/cuDNN version:
  9.2/7.1.4.18
• GPU model and memory:
  GTX 960/4G
• Exact command to reproduce:
  python tf_models/research/object_detection/model_main.py --alsologtostderr
  --pipeline_config_path=ssdlite_mobilenet_v2_coco.config --model_dir=XXX

Describe the problem

Looks like the new model_main.py hardcodes many parameters. It simply ignores keep_checkpoint_every_n_hours. In another bug, it only exports one image to tensorboard, ignoring
num_visualizations.

[irmowan]

https://www.tensorflow.org/api_docs/python/tf/train/Saver#__init__
you can use tf.train.Saver(max_to_keep=None) to save all checkpoints when you use legacy/trainer.py

[yu-jingrui]

@irmowan do you mean the one on line 408 in model_lib.py? Adding max_to_keep=None to this one has no effect.

[yu-jingrui]

@irmowan check issue #5139. With tf r1.10 it can be set in model_main.py

[lan2720]

@YijinLiu @fisheess Do you fix this issue? I have the same problem.

[lan2720]

@pkulzc Hi, how to set max_to_keep=None. I have changed all tf.train.Saver(max_to_keep=None) in object_detection project, but it's still not working.

[yu-jingrui]

@lan2720 It is solved. With the latest tensorflow you need to set the parameters in model_main.py. Check #5139. You need to add keep_checkpoint_max=None to line 59 config = tf.estimator.RunConfig(...)

[yu-jingrui]

@lan2720 you can check documents on tf.estimator.RunConfig to get more information.
@YijinLiu I think it is time to close this issue since it is already solved.

[lan2720]

@fisheess Thank you! It works.

[YijinLiu]

@fisheess 's method works for me.

[saeedarabi92]

Also check this: https://stackoverflow.com/a/51858873/8407621

[YijinLiu]

Sigh, it's broken again after I synced to the latest code...
It's caused by a1337e0#diff-76df8ca264a059e8a3003851fe4d7849R474
You'll need to add a max_to_keep parameter.