mAP decreases as training steps increase in faster rcnn tensorflow object detection api. It is a weird way of training progress for me. what do you think the problem will be? I have used faster rccn inception v2 pretrained model to finetune it on my custom dataset. the config file is almost similar with the original version. batch size 24.

[Fetulhak]

Evaluate annotation type bbox
DONE (t=4.24s).
Accumulating evaluation results...
DONE (t=0.07s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.314
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.844
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.155
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.316
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.046
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.342
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.503
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.505
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
INFO:tensorflow:Finished evaluation at 2020-08-31-07:22:41
I0831 07:22:41.378933 140529569548160 evaluation.py:275] Finished evaluation at 2020-08-31-07:22:41
INFO:tensorflow:Saving dict for global step 3366: DetectionBoxes_Precision/mAP = 0.31436983, DetectionBoxes_Precision/mAP (large) = -1.0, DetectionBoxes_Precision/mAP (medium) = 0.31567088, DetectionBoxes_Precision/mAP (small) = 0.0, DetectionBoxes_Precision/mAP@.50IOU = 0.8435796, DetectionBoxes_Precision/mAP@.75IOU = 0.15527642, DetectionBoxes_Recall/AR@1 = 0.0456, DetectionBoxes_Recall/AR@10 = 0.3421, DetectionBoxes_Recall/AR@100 = 0.5034, DetectionBoxes_Recall/AR@100 (large) = -1.0, DetectionBoxes_Recall/AR@100 (medium) = 0.50491476, DetectionBoxes_Recall/AR@100 (small) = 0.0, Loss/BoxClassifierLoss/classification_loss = 0.24066836, Loss/BoxClassifierLoss/localization_loss = 0.2751926, Loss/RPNLoss/localization_loss = 0.066240005, Loss/RPNLoss/objectness_loss = 0.013297786, Loss/total_loss = 0.5953988, global_step = 3366, learning_rate = 0.001, loss = 0.5953988
I0831 07:22:41.379251 140529569548160 estimator.py:2049] Saving dict for global step 3366: DetectionBoxes_Precision/mAP = 0.31436983, DetectionBoxes_Precision/mAP (large) = -1.0, DetectionBoxes_Precision/mAP (medium) = 0.31567088, DetectionBoxes_Precision/mAP (small) = 0.0, DetectionBoxes_Precision/mAP@.50IOU = 0.8435796, DetectionBoxes_Precision/mAP@.75IOU = 0.15527642, DetectionBoxes_Recall/AR@1 = 0.0456, DetectionBoxes_Recall/AR@10 = 0.3421, DetectionBoxes_Recall/AR@100 = 0.5034, DetectionBoxes_Recall/AR@100 (large) = -1.0, DetectionBoxes_Recall/AR@100 (medium) = 0.50491476, DetectionBoxes_Recall/AR@100 (small) = 0.0, Loss/BoxClassifierLoss/classification_loss = 0.24066836, Loss/BoxClassifierLoss/localization_loss = 0.2751926, Loss/RPNLoss/localization_loss = 0.066240005, Loss/RPNLoss/objectness_loss = 0.013297786, Loss/total_loss = 0.5953988, global_step = 3366, learning_rate = 0.001, loss = 0.5953988
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 3366: training/model.ckpt-3366
I0831 07:22:41.393022 140529569548160 estimator.py:2109] Saving 'checkpoint_path' summary for global step 3366: training/model.ckpt-3366
INFO:tensorflow:global_step/sec: 0.456421
I0831 07:22:49.051671 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.456421
INFO:tensorflow:loss = 0.5314422, step = 3369 (219.095 sec)
I0831 07:22:49.052883 140529569548160 basic_session_run_hooks.py:260] loss = 0.5314422, step = 3369 (219.095 sec)
INFO:tensorflow:global_step/sec: 0.529406
I0831 07:25:57.942767 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.529406
INFO:tensorflow:loss = 0.5429549, step = 3469 (188.891 sec)
I0831 07:25:57.944226 140529569548160 basic_session_run_hooks.py:260] loss = 0.5429549, step = 3469 (188.891 sec)
INFO:tensorflow:global_step/sec: 0.526695
I0831 07:29:07.805795 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.526695
INFO:tensorflow:loss = 0.55023456, step = 3569 (189.863 sec)
I0831 07:29:07.807425 140529569548160 basic_session_run_hooks.py:260] loss = 0.55023456, step = 3569 (189.863 sec)
INFO:tensorflow:Saving checkpoints for 3668 into training/model.ckpt.
I0831 07:32:14.385575 140529569548160 basic_session_run_hooks.py:606] Saving checkpoints for 3668 into training/model.ckpt.
INFO:tensorflow:Calling model_fn.
I0831 07:32:17.689545 140529569548160 estimator.py:1148] Calling model_fn.
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 07:32:19.105491 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 07:32:19.118811 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:depth of additional conv before box predictor: 0
I0831 07:32:19.119150 140529569548160 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 07:32:20.175370 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 07:32:20.192636 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Done calling model_fn.
I0831 07:32:21.930514 140529569548160 estimator.py:1150] Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2020-08-31T07:32:21Z
I0831 07:32:21.946928 140529569548160 evaluation.py:255] Starting evaluation at 2020-08-31T07:32:21Z
INFO:tensorflow:Graph was finalized.
I0831 07:32:22.385655 140529569548160 monitored_session.py:240] Graph was finalized.
2020-08-31 07:32:22.386379: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 07:32:22.386697: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:00:04.0
2020-08-31 07:32:22.386823: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-08-31 07:32:22.386855: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-08-31 07:32:22.386881: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-08-31 07:32:22.386901: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-08-31 07:32:22.386938: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-08-31 07:32:22.386961: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-08-31 07:32:22.386989: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-08-31 07:32:22.387076: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 07:32:22.387378: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 07:32:22.387638: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-08-31 07:32:22.387682: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-31 07:32:22.387696: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2020-08-31 07:32:22.387706: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2020-08-31 07:32:22.387813: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 07:32:22.388110: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 07:32:22.388368: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14221 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5)
INFO:tensorflow:Restoring parameters from training/model.ckpt-3668
I0831 07:32:22.390268 140529569548160 saver.py:1284] Restoring parameters from training/model.ckpt-3668
INFO:tensorflow:Running local_init_op.
I0831 07:32:23.428871 140529569548160 session_manager.py:500] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I0831 07:32:23.588097 140529569548160 session_manager.py:502] Done running local_init_op.
INFO:tensorflow:Performing evaluation on 102 images.
I0831 07:32:38.782745 140526982002432 coco_evaluation.py:205] Performing evaluation on 102 images.
creating index...
index created!
INFO:tensorflow:Loading and preparing annotation results...
I0831 07:32:38.784129 140526982002432 coco_tools.py:115] Loading and preparing annotation results...
INFO:tensorflow:DONE (t=0.02s)
I0831 07:32:38.802798 140526982002432 coco_tools.py:137] DONE (t=0.02s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type bbox
DONE (t=4.18s).
Accumulating evaluation results...
DONE (t=0.06s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.308
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.838
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.161
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.310
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.045
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.340
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.511
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.513
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
INFO:tensorflow:Finished evaluation at 2020-08-31-07:32:43
I0831 07:32:43.164965 140529569548160 evaluation.py:275] Finished evaluation at 2020-08-31-07:32:43
INFO:tensorflow:Saving dict for global step 3668: DetectionBoxes_Precision/mAP = 0.3079519, DetectionBoxes_Precision/mAP (large) = -1.0, DetectionBoxes_Precision/mAP (medium) = 0.3095433, DetectionBoxes_Precision/mAP (small) = 0.0, DetectionBoxes_Precision/mAP@.50IOU = 0.83835673, DetectionBoxes_Precision/mAP@.75IOU = 0.16109954, DetectionBoxes_Recall/AR@1 = 0.0454, DetectionBoxes_Recall/AR@10 = 0.3405, DetectionBoxes_Recall/AR@100 = 0.5111, DetectionBoxes_Recall/AR@100 (large) = -1.0, DetectionBoxes_Recall/AR@100 (medium) = 0.5126379, DetectionBoxes_Recall/AR@100 (small) = 0.0, Loss/BoxClassifierLoss/classification_loss = 0.24741209, Loss/BoxClassifierLoss/localization_loss = 0.27934107, Loss/RPNLoss/localization_loss = 0.06676239, Loss/RPNLoss/objectness_loss = 0.014064799, Loss/total_loss = 0.6075804, global_step = 3668, learning_rate = 0.001, loss = 0.6075804
I0831 07:32:43.165262 140529569548160 estimator.py:2049] Saving dict for global step 3668: DetectionBoxes_Precision/mAP = 0.3079519, DetectionBoxes_Precision/mAP (large) = -1.0, DetectionBoxes_Precision/mAP (medium) = 0.3095433, DetectionBoxes_Precision/mAP (small) = 0.0, DetectionBoxes_Precision/mAP@.50IOU = 0.83835673, DetectionBoxes_Precision/mAP@.75IOU = 0.16109954, DetectionBoxes_Recall/AR@1 = 0.0454, DetectionBoxes_Recall/AR@10 = 0.3405, DetectionBoxes_Recall/AR@100 = 0.5111, DetectionBoxes_Recall/AR@100 (large) = -1.0, DetectionBoxes_Recall/AR@100 (medium) = 0.5126379, DetectionBoxes_Recall/AR@100 (small) = 0.0, Loss/BoxClassifierLoss/classification_loss = 0.24741209, Loss/BoxClassifierLoss/localization_loss = 0.27934107, Loss/RPNLoss/localization_loss = 0.06676239, Loss/RPNLoss/objectness_loss = 0.014064799, Loss/total_loss = 0.6075804, global_step = 3668, learning_rate = 0.001, loss = 0.6075804
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 3668: training/model.ckpt-3668
I0831 07:32:43.177099 140529569548160 estimator.py:2109] Saving 'checkpoint_path' summary for global step 3668: training/model.ckpt-3668
INFO:tensorflow:global_step/sec: 0.456204
I0831 07:32:47.005799 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.456204
INFO:tensorflow:loss = 0.4690953, step = 3669 (219.200 sec)
I0831 07:32:47.006930 140529569548160 basic_session_run_hooks.py:260] loss = 0.4690953, step = 3669 (219.200 sec)
INFO:tensorflow:global_step/sec: 0.53203
I0831 07:35:54.964972 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.53203
INFO:tensorflow:loss = 0.4143501, step = 3769 (187.959 sec)
I0831 07:35:54.966166 140529569548160 basic_session_run_hooks.py:260] loss = 0.4143501, step = 3769 (187.959 sec)
INFO:tensorflow:global_step/sec: 0.526431
I0831 07:39:04.923553 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.526431
INFO:tensorflow:loss = 0.49879405, step = 3869 (189.959 sec)
I0831 07:39:04.925200 140529569548160 basic_session_run_hooks.py:260] loss = 0.49879405, step = 3869 (189.959 sec)
INFO:tensorflow:Saving checkpoints for 3970 into training/model.ckpt.
I0831 07:42:15.309181 140529569548160 basic_session_run_hooks.py:606] Saving checkpoints for 3970 into training/model.ckpt.
INFO:tensorflow:Calling model_fn.
I0831 07:42:18.419227 140529569548160 estimator.py:1148] Calling model_fn.
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 07:42:19.822912 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 07:42:19.835297 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:depth of additional conv before box predictor: 0
I0831 07:42:19.835624 140529569548160 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 07:42:21.342495 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 07:42:21.358611 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Done calling model_fn.
I0831 07:42:23.143416 140529569548160 estimator.py:1150] Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2020-08-31T07:42:23Z
I0831 07:42:23.162491 140529569548160 evaluation.py:255] Starting evaluation at 2020-08-31T07:42:23Z
INFO:tensorflow:Graph was finalized.
I0831 07:42:23.599226 140529569548160 monitored_session.py:240] Graph was finalized.
2020-08-31 07:42:23.599888: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 07:42:23.600200: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:00:04.0
2020-08-31 07:42:23.600302: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-08-31 07:42:23.600331: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-08-31 07:42:23.600356: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-08-31 07:42:23.600380: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-08-31 07:42:23.600399: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-08-31 07:42:23.600418: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-08-31 07:42:23.600439: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-08-31 07:42:23.600521: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 07:42:23.600835: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 07:42:23.601074: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-08-31 07:42:23.601119: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-31 07:42:23.601132: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2020-08-31 07:42:23.601141: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2020-08-31 07:42:23.601241: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 07:42:23.601519: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 07:42:23.601751: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14221 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5)
INFO:tensorflow:Restoring parameters from training/model.ckpt-3970
I0831 07:42:23.603839 140529569548160 saver.py:1284] Restoring parameters from training/model.ckpt-3970
INFO:tensorflow:Running local_init_op.
I0831 07:42:24.696616 140529569548160 session_manager.py:500] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I0831 07:42:24.879029 140529569548160 session_manager.py:502] Done running local_init_op.
INFO:tensorflow:Performing evaluation on 102 images.
I0831 07:42:39.873946 140526990395136 coco_evaluation.py:205] Performing evaluation on 102 images.
creating index...
index created!
INFO:tensorflow:Loading and preparing annotation results...
I0831 07:42:39.875130 140526990395136 coco_tools.py:115] Loading and preparing annotation results...
INFO:tensorflow:DONE (t=0.02s)
I0831 07:42:39.898843 140526990395136 coco_tools.py:137] DONE (t=0.02s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type bbox
DONE (t=4.11s).
Accumulating evaluation results...
DONE (t=0.06s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.305
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.834
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.144
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.306
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.042
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.343
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.506
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.508
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
INFO:tensorflow:Finished evaluation at 2020-08-31-07:42:44
I0831 07:42:44.210580 140529569548160 evaluation.py:275] Finished evaluation at 2020-08-31-07:42:44
INFO:tensorflow:Saving dict for global step 3970: DetectionBoxes_Precision/mAP = 0.30534187, DetectionBoxes_Precision/mAP (large) = -1.0, DetectionBoxes_Precision/mAP (medium) = 0.3064494, DetectionBoxes_Precision/mAP (small) = 0.0, DetectionBoxes_Precision/mAP@.50IOU = 0.83381045, DetectionBoxes_Precision/mAP@.75IOU = 0.14356509, DetectionBoxes_Recall/AR@1 = 0.0423, DetectionBoxes_Recall/AR@10 = 0.3434, DetectionBoxes_Recall/AR@100 = 0.5065, DetectionBoxes_Recall/AR@100 (large) = -1.0, DetectionBoxes_Recall/AR@100 (medium) = 0.5080241, DetectionBoxes_Recall/AR@100 (small) = 0.0, Loss/BoxClassifierLoss/classification_loss = 0.2550419, Loss/BoxClassifierLoss/localization_loss = 0.29824752, Loss/RPNLoss/localization_loss = 0.06717979, Loss/RPNLoss/objectness_loss = 0.013152282, Loss/total_loss = 0.63362145, global_step = 3970, learning_rate = 0.001, loss = 0.63362145
I0831 07:42:44.210865 140529569548160 estimator.py:2049] Saving dict for global step 3970: DetectionBoxes_Precision/mAP = 0.30534187, DetectionBoxes_Precision/mAP (large) = -1.0, DetectionBoxes_Precision/mAP (medium) = 0.3064494, DetectionBoxes_Precision/mAP (small) = 0.0, DetectionBoxes_Precision/mAP@.50IOU = 0.83381045, DetectionBoxes_Precision/mAP@.75IOU = 0.14356509, DetectionBoxes_Recall/AR@1 = 0.0423, DetectionBoxes_Recall/AR@10 = 0.3434, DetectionBoxes_Recall/AR@100 = 0.5065, DetectionBoxes_Recall/AR@100 (large) = -1.0, DetectionBoxes_Recall/AR@100 (medium) = 0.5080241, DetectionBoxes_Recall/AR@100 (small) = 0.0, Loss/BoxClassifierLoss/classification_loss = 0.2550419, Loss/BoxClassifierLoss/localization_loss = 0.29824752, Loss/RPNLoss/localization_loss = 0.06717979, Loss/RPNLoss/objectness_loss = 0.013152282, Loss/total_loss = 0.63362145, global_step = 3970, learning_rate = 0.001, loss = 0.63362145
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 3970: training/model.ckpt-3970
I0831 07:42:44.222068 140529569548160 estimator.py:2109] Saving 'checkpoint_path' summary for global step 3970: training/model.ckpt-3970
INFO:tensorflow:global_step/sec: 0.455995
I0831 07:42:44.223970 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.455995
INFO:tensorflow:loss = 0.5270295, step = 3969 (219.300 sec)
I0831 07:42:44.224892 140529569548160 basic_session_run_hooks.py:260] loss = 0.5270295, step = 3969 (219.300 sec)
INFO:tensorflow:global_step/sec: 0.527989
I0831 07:45:53.621862 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.527989
INFO:tensorflow:loss = 0.50037175, step = 4069 (189.399 sec)
I0831 07:45:53.623555 140529569548160 basic_session_run_hooks.py:260] loss = 0.50037175, step = 4069 (189.399 sec)
INFO:tensorflow:global_step/sec: 0.522826
I0831 07:49:04.890216 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.522826
INFO:tensorflow:loss = 0.52528, step = 4169 (191.268 sec)
I0831 07:49:04.891823 140529569548160 basic_session_run_hooks.py:260] loss = 0.52528, step = 4169 (191.268 sec)
INFO:tensorflow:global_step/sec: 0.526333
I0831 07:52:14.884055 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.526333
INFO:tensorflow:loss = 0.47124624, step = 4269 (189.994 sec)
I0831 07:52:14.885528 140529569548160 basic_session_run_hooks.py:260] loss = 0.47124624, step = 4269 (189.994 sec)
INFO:tensorflow:Saving checkpoints for 4271 into training/model.ckpt.
I0831 07:52:16.755300 140529569548160 basic_session_run_hooks.py:606] Saving checkpoints for 4271 into training/model.ckpt.
INFO:tensorflow:Calling model_fn.
I0831 07:52:20.224051 140529569548160 estimator.py:1148] Calling model_fn.
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 07:52:21.619853 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 07:52:21.632409 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:depth of additional conv before box predictor: 0
I0831 07:52:21.632741 140529569548160 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 07:52:22.708257 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 07:52:22.729282 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Done calling model_fn.
I0831 07:52:24.549369 140529569548160 estimator.py:1150] Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2020-08-31T07:52:24Z
I0831 07:52:24.564951 140529569548160 evaluation.py:255] Starting evaluation at 2020-08-31T07:52:24Z
INFO:tensorflow:Graph was finalized.
I0831 07:52:25.000237 140529569548160 monitored_session.py:240] Graph was finalized.
2020-08-31 07:52:25.001031: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 07:52:25.001356: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:00:04.0
2020-08-31 07:52:25.001448: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-08-31 07:52:25.001479: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-08-31 07:52:25.001503: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-08-31 07:52:25.001530: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-08-31 07:52:25.001554: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-08-31 07:52:25.001575: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-08-31 07:52:25.001597: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-08-31 07:52:25.001684: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 07:52:25.002077: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 07:52:25.002366: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-08-31 07:52:25.002426: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-31 07:52:25.002443: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2020-08-31 07:52:25.002455: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2020-08-31 07:52:25.002587: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 07:52:25.002997: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 07:52:25.003239: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14221 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5)
INFO:tensorflow:Restoring parameters from training/model.ckpt-4271
I0831 07:52:25.004953 140529569548160 saver.py:1284] Restoring parameters from training/model.ckpt-4271
INFO:tensorflow:Running local_init_op.
I0831 07:52:26.397088 140529569548160 session_manager.py:500] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I0831 07:52:26.602425 140529569548160 session_manager.py:502] Done running local_init_op.
INFO:tensorflow:Performing evaluation on 102 images.
I0831 07:52:41.601227 140526990395136 coco_evaluation.py:205] Performing evaluation on 102 images.
creating index...
index created!
INFO:tensorflow:Loading and preparing annotation results...
I0831 07:52:41.602574 140526990395136 coco_tools.py:115] Loading and preparing annotation results...
INFO:tensorflow:DONE (t=0.02s)
I0831 07:52:41.622398 140526990395136 coco_tools.py:137] DONE (t=0.02s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type bbox
DONE (t=4.10s).
Accumulating evaluation results...
DONE (t=0.06s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.285
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.805
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.135
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.286
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.045
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.330
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.495
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.496
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
INFO:tensorflow:Finished evaluation at 2020-08-31-07:52:45
I0831 07:52:45.912902 140529569548160 evaluation.py:275] Finished evaluation at 2020-08-31-07:52:45
INFO:tensorflow:Saving dict for global step 4271: DetectionBoxes_Precision/mAP = 0.28498507, DetectionBoxes_Precision/mAP (large) = -1.0, DetectionBoxes_Precision/mAP (medium) = 0.28630382, DetectionBoxes_Precision/mAP (small) = 0.0, DetectionBoxes_Precision/mAP@.50IOU = 0.8050973, DetectionBoxes_Precision/mAP@.75IOU = 0.13491808, DetectionBoxes_Recall/AR@1 = 0.0454, DetectionBoxes_Recall/AR@10 = 0.3295, DetectionBoxes_Recall/AR@100 = 0.4946, DetectionBoxes_Recall/AR@100 (large) = -1.0, DetectionBoxes_Recall/AR@100 (medium) = 0.49608827, DetectionBoxes_Recall/AR@100 (small) = 0.0, Loss/BoxClassifierLoss/classification_loss = 0.2520462, Loss/BoxClassifierLoss/localization_loss = 0.3052786, Loss/RPNLoss/localization_loss = 0.06822211, Loss/RPNLoss/objectness_loss = 0.014657295, Loss/total_loss = 0.6402043, global_step = 4271, learning_rate = 0.001, loss = 0.6402043
I0831 07:52:45.913205 140529569548160 estimator.py:2049] Saving dict for global step 4271: DetectionBoxes_Precision/mAP = 0.28498507, DetectionBoxes_Precision/mAP (large) = -1.0, DetectionBoxes_Precision/mAP (medium) = 0.28630382, DetectionBoxes_Precision/mAP (small) = 0.0, DetectionBoxes_Precision/mAP@.50IOU = 0.8050973, DetectionBoxes_Precision/mAP@.75IOU = 0.13491808, DetectionBoxes_Recall/AR@1 = 0.0454, DetectionBoxes_Recall/AR@10 = 0.3295, DetectionBoxes_Recall/AR@100 = 0.4946, DetectionBoxes_Recall/AR@100 (large) = -1.0, DetectionBoxes_Recall/AR@100 (medium) = 0.49608827, DetectionBoxes_Recall/AR@100 (small) = 0.0, Loss/BoxClassifierLoss/classification_loss = 0.2520462, Loss/BoxClassifierLoss/localization_loss = 0.3052786, Loss/RPNLoss/localization_loss = 0.06822211, Loss/RPNLoss/objectness_loss = 0.014657295, Loss/total_loss = 0.6402043, global_step = 4271, learning_rate = 0.001, loss = 0.6402043
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 4271: training/model.ckpt-4271
I0831 07:52:45.924731 140529569548160 estimator.py:2109] Saving 'checkpoint_path' summary for global step 4271: training/model.ckpt-4271
INFO:tensorflow:global_step/sec: 0.460804
I0831 07:55:51.896059 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.460804
INFO:tensorflow:loss = 0.38488027, step = 4369 (217.012 sec)
I0831 07:55:51.897870 140529569548160 basic_session_run_hooks.py:260] loss = 0.38488027, step = 4369 (217.012 sec)
INFO:tensorflow:global_step/sec: 0.527887
I0831 07:59:01.330513 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.527887
INFO:tensorflow:loss = 0.39409614, step = 4469 (189.434 sec)
I0831 07:59:01.332315 140529569548160 basic_session_run_hooks.py:260] loss = 0.39409614, step = 4469 (189.434 sec)
INFO:tensorflow:global_step/sec: 0.521504
I0831 08:02:13.083614 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.521504
INFO:tensorflow:loss = 0.50167036, step = 4569 (191.753 sec)
I0831 08:02:13.085605 140529569548160 basic_session_run_hooks.py:260] loss = 0.50167036, step = 4569 (191.753 sec)
INFO:tensorflow:Saving checkpoints for 4572 into training/model.ckpt.
I0831 08:02:17.270462 140529569548160 basic_session_run_hooks.py:606] Saving checkpoints for 4572 into training/model.ckpt.
INFO:tensorflow:Calling model_fn.
I0831 08:02:20.573987 140529569548160 estimator.py:1148] Calling model_fn.
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 08:02:22.028924 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 08:02:22.044717 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:depth of additional conv before box predictor: 0
I0831 08:02:22.045222 140529569548160 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 08:02:23.154609 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 08:02:23.171390 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Done calling model_fn.
I0831 08:02:25.103013 140529569548160 estimator.py:1150] Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2020-08-31T08:02:25Z
I0831 08:02:25.120003 140529569548160 evaluation.py:255] Starting evaluation at 2020-08-31T08:02:25Z
INFO:tensorflow:Graph was finalized.
I0831 08:02:25.581543 140529569548160 monitored_session.py:240] Graph was finalized.
2020-08-31 08:02:25.582306: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 08:02:25.582646: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:00:04.0
2020-08-31 08:02:25.582764: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-08-31 08:02:25.582823: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-08-31 08:02:25.582854: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-08-31 08:02:25.582882: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-08-31 08:02:25.582904: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-08-31 08:02:25.582928: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-08-31 08:02:25.582964: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-08-31 08:02:25.583056: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 08:02:25.583385: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 08:02:25.583651: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-08-31 08:02:25.583716: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-31 08:02:25.583729: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2020-08-31 08:02:25.583739: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2020-08-31 08:02:25.583853: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 08:02:25.584140: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 08:02:25.584399: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14221 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5)
INFO:tensorflow:Restoring parameters from training/model.ckpt-4572
I0831 08:02:25.586534 140529569548160 saver.py:1284] Restoring parameters from training/model.ckpt-4572
INFO:tensorflow:Running local_init_op.
I0831 08:02:26.744107 140529569548160 session_manager.py:500] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I0831 08:02:26.950001 140529569548160 session_manager.py:502] Done running local_init_op.
INFO:tensorflow:Performing evaluation on 102 images.
I0831 08:02:41.757927 140526990395136 coco_evaluation.py:205] Performing evaluation on 102 images.
creating index...
index created!
INFO:tensorflow:Loading and preparing annotation results...
I0831 08:02:41.759518 140526990395136 coco_tools.py:115] Loading and preparing annotation results...
INFO:tensorflow:DONE (t=0.02s)
I0831 08:02:41.779049 140526990395136 coco_tools.py:137] DONE (t=0.02s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type bbox
DONE (t=4.06s).
Accumulating evaluation results...
DONE (t=0.06s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.275
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.804
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.124
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.277
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.038
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.320
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.481
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.482
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
INFO:tensorflow:Finished evaluation at 2020-08-31-08:02:46
I0831 08:02:46.018646 140529569548160 evaluation.py:275] Finished evaluation at 2020-08-31-08:02:46
INFO:tensorflow:Saving dict for global step 4572: DetectionBoxes_Precision/mAP = 0.27549165, DetectionBoxes_Precision/mAP (large) = -1.0, DetectionBoxes_Precision/mAP (medium) = 0.27698833, DetectionBoxes_Precision/mAP (small) = 0.0, DetectionBoxes_Precision/mAP@.50IOU = 0.8040639, DetectionBoxes_Precision/mAP@.75IOU = 0.12408327, DetectionBoxes_Recall/AR@1 = 0.0379, DetectionBoxes_Recall/AR@10 = 0.3203, DetectionBoxes_Recall/AR@100 = 0.4809, DetectionBoxes_Recall/AR@100 (large) = -1.0, DetectionBoxes_Recall/AR@100 (medium) = 0.48234704, DetectionBoxes_Recall/AR@100 (small) = 0.0, Loss/BoxClassifierLoss/classification_loss = 0.27474567, Loss/BoxClassifierLoss/localization_loss = 0.32303578, Loss/RPNLoss/localization_loss = 0.06927199, Loss/RPNLoss/objectness_loss = 0.0134567795, Loss/total_loss = 0.68051034, global_step = 4572, learning_rate = 0.001, loss = 0.68051034
I0831 08:02:46.018949 140529569548160 estimator.py:2049] Saving dict for global step 4572: DetectionBoxes_Precision/mAP = 0.27549165, DetectionBoxes_Precision/mAP (large) = -1.0, DetectionBoxes_Precision/mAP (medium) = 0.27698833, DetectionBoxes_Precision/mAP (small) = 0.0, DetectionBoxes_Precision/mAP@.50IOU = 0.8040639, DetectionBoxes_Precision/mAP@.75IOU = 0.12408327, DetectionBoxes_Recall/AR@1 = 0.0379, DetectionBoxes_Recall/AR@10 = 0.3203, DetectionBoxes_Recall/AR@100 = 0.4809, DetectionBoxes_Recall/AR@100 (large) = -1.0, DetectionBoxes_Recall/AR@100 (medium) = 0.48234704, DetectionBoxes_Recall/AR@100 (small) = 0.0, Loss/BoxClassifierLoss/classification_loss = 0.27474567, Loss/BoxClassifierLoss/localization_loss = 0.32303578, Loss/RPNLoss/localization_loss = 0.06927199, Loss/RPNLoss/objectness_loss = 0.0134567795, Loss/total_loss = 0.68051034, global_step = 4572, learning_rate = 0.001, loss = 0.68051034
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 4572: training/model.ckpt-4572
I0831 08:02:46.030554 140529569548160 estimator.py:2109] Saving 'checkpoint_path' summary for global step 4572: training/model.ckpt-4572
INFO:tensorflow:global_step/sec: 0.455421
I0831 08:05:52.660928 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.455421
INFO:tensorflow:loss = 0.37305856, step = 4669 (219.577 sec)
I0831 08:05:52.662541 140529569548160 basic_session_run_hooks.py:260] loss = 0.37305856, step = 4669 (219.577 sec)
INFO:tensorflow:global_step/sec: 0.52417
I0831 08:09:03.438588 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.52417
INFO:tensorflow:loss = 0.3631875, step = 4769 (190.778 sec)
I0831 08:09:03.440209 140529569548160 basic_session_run_hooks.py:260] loss = 0.3631875, step = 4769 (190.778 sec)
INFO:tensorflow:global_step/sec: 0.526801
I0831 08:12:13.263454 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.526801
INFO:tensorflow:loss = 0.3741885, step = 4869 (189.825 sec)
I0831 08:12:13.265093 140529569548160 basic_session_run_hooks.py:260] loss = 0.3741885, step = 4869 (189.825 sec)
INFO:tensorflow:Saving checkpoints for 4873 into training/model.ckpt.
I0831 08:12:18.910395 140529569548160 basic_session_run_hooks.py:606] Saving checkpoints for 4873 into training/model.ckpt.
INFO:tensorflow:Calling model_fn.
I0831 08:12:22.216905 140529569548160 estimator.py:1148] Calling model_fn.
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 08:12:23.580093 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 08:12:23.593119 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:depth of additional conv before box predictor: 0
I0831 08:12:23.593468 140529569548160 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 08:12:24.650454 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
I0831 08:12:24.665254 140529569548160 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Done calling model_fn.
I0831 08:12:26.758405 140529569548160 estimator.py:1150] Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2020-08-31T08:12:26Z
I0831 08:12:26.773212 140529569548160 evaluation.py:255] Starting evaluation at 2020-08-31T08:12:26Z
INFO:tensorflow:Graph was finalized.
I0831 08:12:27.221370 140529569548160 monitored_session.py:240] Graph was finalized.
2020-08-31 08:12:27.222244: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 08:12:27.222666: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
pciBusID: 0000:00:04.0
2020-08-31 08:12:27.222759: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-08-31 08:12:27.222803: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-08-31 08:12:27.222828: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-08-31 08:12:27.222849: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-08-31 08:12:27.222877: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-08-31 08:12:27.222907: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-08-31 08:12:27.222938: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-08-31 08:12:27.223062: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 08:12:27.223472: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 08:12:27.223841: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-08-31 08:12:27.223910: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-31 08:12:27.223933: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2020-08-31 08:12:27.223947: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2020-08-31 08:12:27.224092: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 08:12:27.224509: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-31 08:12:27.224887: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14221 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5)
INFO:tensorflow:Restoring parameters from training/model.ckpt-4873
I0831 08:12:27.227638 140529569548160 saver.py:1284] Restoring parameters from training/model.ckpt-4873
INFO:tensorflow:Running local_init_op.
I0831 08:12:28.571145 140529569548160 session_manager.py:500] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I0831 08:12:28.713408 140529569548160 session_manager.py:502] Done running local_init_op.
INFO:tensorflow:Performing evaluation on 102 images.
I0831 08:12:42.829031 140526990395136 coco_evaluation.py:205] Performing evaluation on 102 images.
creating index...
index created!
INFO:tensorflow:Loading and preparing annotation results...
I0831 08:12:42.830227 140526990395136 coco_tools.py:115] Loading and preparing annotation results...
INFO:tensorflow:DONE (t=0.02s)
I0831 08:12:42.848253 140526990395136 coco_tools.py:137] DONE (t=0.02s)
creating index...
index created!
Running per image evaluation...
Evaluate annotation type bbox
DONE (t=3.97s).
Accumulating evaluation results...
DONE (t=0.06s).
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.273
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.777
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.129
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.275
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.037
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.311
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.486
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.487
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
INFO:tensorflow:Finished evaluation at 2020-08-31-08:12:47
I0831 08:12:47.004221 140529569548160 evaluation.py:275] Finished evaluation at 2020-08-31-08:12:47
INFO:tensorflow:Saving dict for global step 4873: DetectionBoxes_Precision/mAP = 0.27279815, DetectionBoxes_Precision/mAP (large) = -1.0, DetectionBoxes_Precision/mAP (medium) = 0.27472305, DetectionBoxes_Precision/mAP (small) = 0.0, DetectionBoxes_Precision/mAP@.50IOU = 0.7765998, DetectionBoxes_Precision/mAP@.75IOU = 0.12912713, DetectionBoxes_Recall/AR@1 = 0.037, DetectionBoxes_Recall/AR@10 = 0.3107, DetectionBoxes_Recall/AR@100 = 0.486, DetectionBoxes_Recall/AR@100 (large) = -1.0, DetectionBoxes_Recall/AR@100 (medium) = 0.4874624, DetectionBoxes_Recall/AR@100 (small) = 0.0, Loss/BoxClassifierLoss/classification_loss = 0.2877345, Loss/BoxClassifierLoss/localization_loss = 0.3372294, Loss/RPNLoss/localization_loss = 0.070613034, Loss/RPNLoss/objectness_loss = 0.0133400755, Loss/total_loss = 0.7089169, global_step = 4873, learning_rate = 0.001, loss = 0.7089169
I0831 08:12:47.004480 140529569548160 estimator.py:2049] Saving dict for global step 4873: DetectionBoxes_Precision/mAP = 0.27279815, DetectionBoxes_Precision/mAP (large) = -1.0, DetectionBoxes_Precision/mAP (medium) = 0.27472305, DetectionBoxes_Precision/mAP (small) = 0.0, DetectionBoxes_Precision/mAP@.50IOU = 0.7765998, DetectionBoxes_Precision/mAP@.75IOU = 0.12912713, DetectionBoxes_Recall/AR@1 = 0.037, DetectionBoxes_Recall/AR@10 = 0.3107, DetectionBoxes_Recall/AR@100 = 0.486, DetectionBoxes_Recall/AR@100 (large) = -1.0, DetectionBoxes_Recall/AR@100 (medium) = 0.4874624, DetectionBoxes_Recall/AR@100 (small) = 0.0, Loss/BoxClassifierLoss/classification_loss = 0.2877345, Loss/BoxClassifierLoss/localization_loss = 0.3372294, Loss/RPNLoss/localization_loss = 0.070613034, Loss/RPNLoss/objectness_loss = 0.0133400755, Loss/total_loss = 0.7089169, global_step = 4873, learning_rate = 0.001, loss = 0.7089169
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 4873: training/model.ckpt-4873
I0831 08:12:47.016038 140529569548160 estimator.py:2109] Saving 'checkpoint_path' summary for global step 4873: training/model.ckpt-4873
INFO:tensorflow:global_step/sec: 0.460294
I0831 08:15:50.515957 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.460294
INFO:tensorflow:loss = 0.36465797, step = 4969 (217.253 sec)
I0831 08:15:50.517859 140529569548160 basic_session_run_hooks.py:260] loss = 0.36465797, step = 4969 (217.253 sec)
INFO:tensorflow:global_step/sec: 0.528299
I0831 08:18:59.802522 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.528299
INFO:tensorflow:loss = 0.34068385, step = 5069 (189.286 sec)
I0831 08:18:59.804225 140529569548160 basic_session_run_hooks.py:260] loss = 0.34068385, step = 5069 (189.286 sec)
INFO:tensorflow:global_step/sec: 0.527435
I0831 08:22:09.399181 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.527435
INFO:tensorflow:loss = 0.3287485, step = 5169 (189.596 sec)
I0831 08:22:09.400691 140529569548160 basic_session_run_hooks.py:260] loss = 0.3287485, step = 5169 (189.596 sec)
INFO:tensorflow:Saving checkpoints for 5175 into training/model.ckpt.
I0831 08:22:18.978193 140529569548160 basic_session_run_hooks.py:606] Saving checkpoints for 5175 into training/model.ckpt.
INFO:tensorflow:Skip the current checkpoint eval due to throttle secs (600 secs).
I0831 08:22:21.311619 140529569548160 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs).
INFO:tensorflow:global_step/sec: 0.520463
I0831 08:25:21.535818 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.520463
INFO:tensorflow:loss = 0.25103626, step = 5269 (192.136 sec)
I0831 08:25:21.537050 140529569548160 basic_session_run_hooks.py:260] loss = 0.25103626, step = 5269 (192.136 sec)
INFO:tensorflow:global_step/sec: 0.529386
I0831 08:28:30.433928 140529569548160 basic_session_run_hooks.py:692] global_step/sec: 0.529386
INFO:tensorflow:loss = 0.28013855, step = 5369 (188.899 sec)
I0831 08:28:30.435732 140529569548160 basic_session_run_hooks.py:260] loss = 0.28013855, step = 5369 (188.899 sec)