Detection precision is worse after fine tune with pre-trained model

[rogercw]

System information

• What is the top-level directory of the model you are using: object_detection
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes, but not much
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): UbuntuMATE 16.04
• TensorFlow installed from (source or binary): binary
• TensorFlow version (use command below): 1.4.1
• Bazel version (if compiling from source):
• CUDA/cuDNN version: V8.0.61
• GPU model and memory: GeForce GTX 1080 Ti / 10.91GiB
• Exact command to reproduce:

Describe the problem

I downloaded the pre-trained model of ssd-inception_v2_coco from detection model zoo, and set the "fine_tune_checkpoint" in "ssd_inception_v2_coco.config" to it. I use script "train.py" following the instruction in tutorial to train the network. After 200000 steps of training, "export_inference_graph.py" is used to export the model. However, the detection precision is getting worse (not just slightly different) with the re-trained model in the evaluation, comparing with directly using pre-trained model. May I know whether it is expected or a bug? If it is not a bug, how could we re-train a model to have better or at least similar detection precision as pre-trained model? Thanks in advance.

[cy89]

@jch1 @tombstone is @rogercooper76 seeing expected behavior?

[rogercw]

I notice there are quite amount of difference in the frozen_inference_graph.pb files between the pre-trained model and the data from step 0 checkpoint in training as well (pre-trained file is re-exported with same version of script). However, they are too large to attach here. If they will be useful to root cause the issue, please let me know what will be a good way to share them. Thanks.

[rogercw]

May I know whether there is any update about this issue? The attached images are the precision and the corresponding total loss graphic outputs from my latest training, in which I set "load_all_detection_checkpoint_vars" in "train_config" to true. Therefore, the initial training checkpoint is 5013943.

• Precision (evaluating with all 5000 images in coco/val2017)
  
• Total loss
  

Based on the result, loss value seems not fully correlated with the precision. Is anything else I should change from the default setting to fix it? Also, may I know whether the pre-trained data from model zoo were trained with this object detection API? Thanks.

[Aurora11111]

I have the same problem as you ,have you work it out?

[bolshackdragon]

@rogercooper76 I have the exact same problem. I used the 5000 validation examples from the coco dataset for both training and evaluation (just as a sanity check). I saw the precision dropping with training.
Did you find a resolution to your issue?

[tensorflowbutler]

Hi There,
We are checking to see if you still need help on this, as this seems to be considerably old issue. Please update this issue with the latest information, code snippet to reproduce your issue and error you are seeing.
If we don't hear from you in the next 7 days, this issue will be closed automatically. If you don't need help on this issue any more, please consider closing this.

[smeyg]

Hi @rogercw,

This post was long time ago but I am still suffering from the same issue. May I ask if you managed to increase the precision after fine tuning with pre-trained model either in TF1+ object detection APO or with TF2+object detection API? If yes can you please describe how?

Thanks

[rogercw]

Hi @smeyg, after working with it for a while, unfortunately, we still could not solve the problem. Therefore, we moved on to try other models and implementations, and I have not got the chance to check out TF2 API though.