Object_Detection unable to finetune from non-detection checkpoint

[AdamSanderson93]

System information

• What is the top-level directory of the model you are using: Object_Detection
• Have I written custom code: Modified Config file
• OS Platform and Distribution: Linux Ubuntu 14.04
• TensorFlow installed from: binary
• TensorFlow version: 1.2.1
• CUDA/cuDNN version: CUDA 8 / cuDNN 5.1
• GPU model and memory: gtx 980 4gb
• Exact command to reproduce:

python models/object_detection/train.py --train_dir=Test --pipeline_config_path=faster_rcnn_resnet101_Button_Cell.config

Describe the problem

When trying to fine-tune from a non-detection checkpoint (from_detection_checkpoint: false in config) the model will not properly load the variables. Prints out a large amount of warnings in this form...

WARNING:root:Variable [resnet_v1_101/conv1/BatchNorm/moving_mean] not available in checkpoint
WARNING:root:Variable [resnet_v1_101/conv1/BatchNorm/moving_variance] not available in checkpoint
WARNING:root:Variable [resnet_v1_101/conv1/weights] not available in checkpoint

Followed by:

ValueError: No variables to save

If I turn from_detection_checkpoint to true everything loads and runs properly. I am using an unedited version of the api and a slightly modified version of the cocosample config. Just filled in where the checkpoints are. I have attempted this using the checkpoints provided on the model detection zoo. Specifically the rfcn_resnet101_coco, and faster_rcnn_resnet101_coco models.

I apologize if the formatting is improper I am rather new to github.
Please let me know if any other information is required.

[AdamSanderson93]

Here is the config file I am using. All I am doing to create the issue is turning "from_detection_checkpoint" parameter from true to false.

faster_rcnn_resnet101_Button_Cell.txt

[derekjchow]

@AdamSanderson93 could you describe what checkpoint you're trying to finetune from? Is it a tf.slim classification checkpoint?

[AdamSanderson93]

@derekjchow I tested this with two separate checkpoints both provided in the model detection zoo from the object detection API. The link to which I have attached below.

https://github.com/tensorflow/models/blob/master/object_detection/g3doc/detection_model_zoo.md

I tried to load both the rfcn_resnet101_coco, and faster_rcnn_resnet101_coco into a model with the config file provided previously. I got the same errors from both checkpoints when "from_detection_checkpoint" was set to false.

[derekjchow]

from_detection_checkpoint should be set to true for those check points (they are detection checkpoints). faster_rcnn_resnet101_coco should be the checkpoint to use.

[AdamSanderson93]

Is there any way to load in just the feature extractors from the detection checkpoints or do I have to get them from a different repository?

[derekjchow]

Setting the finetune_checkpoint variable will only load the feature extractors.

[AdamSanderson93]

So if I'm understanding you correctly. If I load a pretrained network from the model detection zoo and set "from_detection_checkpoint" to true it will only load the feature extractor layers (In my case resnet 101) and not the RPN, or second stage box predictor layers.

However if I set "from_detection_checkpoint" to false I need to use a checkpoint from the slim pretrained networks found here: https://github.com/tensorflow/models/tree/master/slim#Pretrained

[vikramg1]

@AdamSanderson93 @jch1 , It is not possible to use the released detection checkpoint from model zoo and set from_detection_chekpoint to false. I tried this and it throws errors for BatchNorm parameters (for Faster RCN and ResNet101).
Wondering if we wanted to start the training from a classification checkpoint, then should we load from detection checkpoint and re-initialize the RPN and rest of the network to initial values? Is there a classification checkpoint available we can use directly?

Figured this out: download classification models from https://github.com/tensorflow/models/tree/master/research/slim#pre-trained-models, then set from_detection_checkpoint = false, and point to the classification model, you should be able to train (for details look at the the documentation is g3doc folder).

[FightForCS]

Hi, everyone, I have already fine tune a classification model from slim, and I want to combine this fine tuned model with the RPN network from object detection model zoo(RPN will not be fine tuned, I want to just use the parameter of the object detection model), is there some ways to achieve this?

[vikramg1]

What type of classification model do you have? If it is one of the ones from the model zoo, then you should be able to add RPN to it.

…

On Nov 20, 2017 6:49 PM, "FightForCS" ***@***.***> wrote: Hi, everyone, I have already fine tune a classification model from slim, and I want to combine this fine tuned model with the RPN network from object detection model zoo(RPN will not be fine tuned, I want to just use the parameter of the object detection model), is there some ways to achieve this? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2025 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AdotlY1Olxt5nG_MGIGnzf0Ow2eCMzDjks5s4jo6gaJpZM4Ohg7E> .

[FightForCS]

@vikramg1 resnet 50, can you give me some instruction on how to add RPN,
I have googled a lot, but still did not get useful information, Thanks

[vikramg1]

So, the thing is that the Resnet-50 that you have will probably be different than the implementations released by Google. They have BN, and also the structure is slightly different w.r.t. the downsampling location.

Anyways, try it by setting the from_detection_checkpoint set to false in the config file for object detection API. Then set the model location to your trained model.
The one thing that you can try is use the resnet model released by google first [using the above settings]. that worked for me. Then you can try your own model. Hope this helps.

[FightForCS]

@vikramg1 now I want to load the resnet50 detection model (Say model A) from tensorflow detection model zoo, then I will load my already fine tuned slim resnet50 model(Say model B), and hopefully, the feature extract part of model B will overwrite the feature extract part of model A, which is what I want that combines RPN parameter from model A and classification head from model B.

now, my encountered problem is: I want to load all parameter from model B, including the fully connected part, setting from_detection_checkpoint=False is not enough (it only load the conv layer, I guess), is there a way to load the fc layer parameter?

Thanks for you previous reply.