-
Notifications
You must be signed in to change notification settings - Fork 2.5k
How to finetune from pretrained detectron models with different number of classes? #15
Comments
Hi, There currently isn't an off-the-shelf option in the config for that. I think we could provide a functionality to perform 1 for the users, given a cfg file and a path to a model weight. That could be a possible improvement on top of what we currently have. What do you think? |
@fmassa I think option 1 is more user-friendly. We can add a config option like PRETRAINED_DETECTRON_WEIGHTS and if it is given all the weights but those of the last layer would be loaded to initilize the model. |
Yeah, option 1 is definitely simpler for the user (even if there are only a few lines to change here and there ;-) ) I'll prepare a PR adding support for this functionality, but I'm not 100% sure of what the API should look like, nor the best fix for it. APIShould we have a function that acts on the weights and creates a new set of weights file? Or should we add an extra config argument, to make it a single step function? If we add an argument (which seems simpler for the user), would it be ambiguous? ImplementationFor the possible fixes, we could hard-code the possible names for the layers that shouldn't be loaded (as I mentioned before). But this is not super robust if the user changes their module names (which they can, if they want). Another possible implementation is to not load the weights for the entire Thoughts? |
I would prefer the former way. For possible module name changes by users, I think they should also be careful for weights loading, either by name remapping or random initialization. |
@wangg12 could you expand on why you'd prefer the first approach? I was actually leaning more towards the second one, as it is more robust, and we have a clear contract with the user when we add an option to the config: "load every weight possible, except those in the predictor". |
@fmassa There are two conditions where the first one may be more suitable.
|
So, I've discussed with a few people here and it seems that the best way of handling this would be to actually perform model surgery on the model files. For example, the best results on CityScapes come from taking a COCO trained detector, then remove most of the classification and mask weights, but retaining those that correspond to common categories between both COCO and CityScapes. |
Yes, this way is more general. |
"load the pre-trained files that you want to use, and delete from the state_dict"
Where are the pretrained files located? For example, I want to use pretrained net in imageset, wheere can we find those files and load them? |
By default, they are stored in |
I added this function to train_net.py with an additional input arg. Note, the loaded models had an additional "module." prefix that had to be removed. After I removed this it worked great. def _transfer_pretrained_weights(model, pretrained_model_pth):
pretrained_weights = torch.load(pretrained_model_pth)['model']
new_dict = {k.replace('module.',''):v for k, v in pretrained_weights.items()
if 'cls_score' not in k and 'bbox_pred' not in k}
this_state = model.state_dict()
this_state.update(new_dict)
model.load_state_dict(this_state)
return model I don't think this is the solution that @fmassa wants to implement but it'll work in a pinch for now. |
Hello @steve-goley @fmassa , I've tried to load the pretrained model in this way: however, an error occured: UnicodeDecodeError: 'ascii' codec can't decode byte 0xad in position 2: ordinal not in range(128) But it seems to be no "model" key in the dict, just "blobs" dict and I can't find 'cls_score' and 'bbox_pred'. Could you tell me how to overcome this issue? Thanks |
@antocapp the What I'd recommend doing is the following: Let me know if it doesn't work, I might have missed a step here. |
Hi @fmassa, thanks for your support. from maskrcnn_benchmark.config import cfg
from maskrcnn_benchmark.utils.c2_model_loading import load_c2_format
cfg.merge_from_file("configs/caffe2/e2e_mask_rcnn_X_101_32x8d_FPN_1x_caffe2.yaml")
path = '/home/antonio/.torch/models/X-101-32x8d.pkl'
_d = load_c2_format(cfg, path)
keys = [k for k in _d['model'].keys()]
print(sorted(keys)) But i can't find 'cls_score' and 'bbox_pred' in the keys. |
@antocapp you are loading the ImageNet-trained models ( |
Thanks @fmassa, so where I can find that model? When i performed inference with that model it works very well (I want just to fine tune it on a class on a specific dataset) but in .torch/models/ i see that only "X-101-32x8d.pkl" has been downloaded. Where i can find the detection model? Thanks for your help i really appreciate that EDIT: I launched again inference and it started downloading again the file 36761843/12_2017_baselines/e2e_mask_rcnn_X-101-32x8d-FPN_1x.yaml.06_35_59.RZotkLKI/output/train/coco_2014_train%3Acoco_2014_valminusminival/generalized_rcnn/model_final.pkl ; maybe I accidentally deleted the previous model from models/ folder. Thanks again! A last question: how should I organize my dataset in order to fine tune the model? |
Hi @antocapp,
Thank you very much |
@BelhalK the weights are inside |
Got it. So the working function should be
Where somethingelse should be different than cls_score and bbox_pred, right? |
Almost, you'll probably need to plug it somewhere in |
you may be right.
But it may be necessary in some other scripts |
I have been using the different tips and tricks of this thread to modify a pre-trained model.
How should I use Thanks a lot for your help! |
You can just save it using |
@jbitton addressed your question in #273 Also, given that the current issues were not enough to give you full context on how to add new datasets, could you perhaps improve a bit the documentation in https://github.com/facebookresearch/maskrcnn-benchmark/blob/master/maskrcnn_benchmark/data/README.md (maybe adding a link from the main README as well) with the points that were missing, and send a PR? It would be a very welcome contribution! |
@fmassa For sure! Do you mind if I get the PR out mid-next week? I'd like to first verify that I was able to go through the training/eval scripts successfully. |
@jbitton sure, no worries! thanks a lot! |
What's the meaning of %3A in the saved path? It's the HTML code for a colon, but why do we want it in a path? |
@mattans we don't necessarily want it in the path. But this might be specific to what Windows can have as characters in a path |
To summarize, I've created a script Then you can simply point the converted model path in the config file by changing |
@wangg12 could you maybe add a section in the TROUBLESHOOTING or in the README pointing to your snippet and send a PR? Thanks! |
I had a question about using trim_detectron_model.py. Is there any solution for this? |
@xiaohai12 I believe you can just replace the call to load_c2_format with a simple torch.load, but I have not tested. |
Thanks. I will try it. |
It worked in my case when I modified load_c2_format to torch.load and modified the the parameters in removekey from cls_score to roi_heads.box.predictor.cls_socre(same for other parameters). |
❓ Questions and Help
Is there a config option to load pretrained coco models for finetuning? The last layers where the number of classes may be different, so those weights should not be loaded.
The text was updated successfully, but these errors were encountered: