python object_detection/eval.py stuck at load .ckpt file

[civilman628]

Now i am using Ubuntu 16.04 and able to train object detection. however, after stop the training and run eval.py alone, it stucks at model .ckpt loading part: i do not understand why it loads .ckpt twice as the log below(Restoring parameters from). I have 3 titan x on my machine.

scopeserver@scopephotos:~/RaidDisk/DeepLearning/mwang/models/object_detection$ python eval.py --logtostderr --pipeline_config_path=/home/scopeserver/RaidDisk/DeepLearning/mwang/models/object_detection/samples/configs/faster_rcnn_resnet101_fashion.config --checkpoint_dir=/home/scopeserver/RaidDisk/DeepLearning/mwang/models/object_detection/checkpoint --eval_dir=/home/scopeserver/RaidDisk/DeepLearning/mwang/models/object_detection/eval/
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
2017-06-23 15:19:27.029181: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-23 15:19:27.029220: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-23 15:19:27.029235: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-06-23 15:19:27.029247: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-23 15:19:27.029259: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2017-06-23 15:19:27.466931: I tensorflow/core/common_runtime/gpu/gpu_device.cc:887] Found device 0 with properties:
name: GeForce GTX TITAN X
major: 5 minor: 2 memoryClockRate (GHz) 1.2155
pciBusID 0000:0a:00.0
Total memory: 11.92GiB
Free memory: 11.81GiB
2017-06-23 15:19:27.467000: W tensorflow/stream_executor/cuda/cuda_driver.cc:485] creating context when one is currently active; existing: 0x6f46570
2017-06-23 15:19:27.738230: I tensorflow/core/common_runtime/gpu/gpu_device.cc:887] Found device 1 with properties:
name: GeForce GTX TITAN X
major: 5 minor: 2 memoryClockRate (GHz) 1.2155
pciBusID 0000:09:00.0
Total memory: 11.92GiB
Free memory: 11.81GiB
2017-06-23 15:19:27.738311: W tensorflow/stream_executor/cuda/cuda_driver.cc:485] creating context when one is currently active; existing: 0x7b90680
2017-06-23 15:19:28.006602: I tensorflow/core/common_runtime/gpu/gpu_device.cc:887] Found device 2 with properties:
name: GeForce GTX TITAN X
major: 5 minor: 2 memoryClockRate (GHz) 1.2155
pciBusID 0000:05:00.0
Total memory: 11.92GiB
Free memory: 11.53GiB
2017-06-23 15:19:28.008184: I tensorflow/core/common_runtime/gpu/gpu_device.cc:908] DMA: 0 1 2
2017-06-23 15:19:28.008198: I tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 0: Y Y Y
2017-06-23 15:19:28.008205: I tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 1: Y Y Y
2017-06-23 15:19:28.008211: I tensorflow/core/common_runtime/gpu/gpu_device.cc:918] 2: Y Y Y
2017-06-23 15:19:28.008226: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:0a:00.0)
2017-06-23 15:19:28.008234: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:1) -> (device: 1, name: GeForce GTX TITAN X, pci bus id: 0000:09:00.0)
2017-06-23 15:19:28.008241: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977] Creating TensorFlow device (/gpu:2) -> (device: 2, name: GeForce GTX TITAN X, pci bus id: 0000:05:00.0)
INFO:tensorflow:Restoring parameters from /home/scopeserver/RaidDisk/DeepLearning/mwang/models/object_detection/checkpoint/model.ckpt-17287
INFO:tensorflow:Restoring parameters from /home/scopeserver/RaidDisk/DeepLearning/mwang/models/object_detection/checkpoint/model.ckpt-17287

[civilman628]

oh, i think it is because it is running, but not like tf-rcnn.

[bw4sz]

@civilman628 can you explain what you mean. I am also confused by this "error" in the sense that it starts at some checkpoint and does not seem to progress alongside the train model.

so note that those images are at step 0, but the training is already many steps along.

the eval never seems to grab the new checkpoints. What am I not understanding?