Simple way to extract features from faster r-cnn

[cdancette]

Currently it is complicated to extract the object features from the faster r-cnn model. It would be nice to have a simple function like model.features(batch), that would give the features in addition to the boxes and labels.

[fmassa]

The question is always: which features do you want to extract? Just after RoIAlign? The one before the last classifier? Something in between?

I don't think we should be extending the API of particular models for extracting intermediate activations. This can be very quickly complicated, and will not be generic enough.

One possible (and easy to do) solution is to just attach a forward hook in the modules where you want to get the output from.

But I do agree that having a way to extract intermediate activations is important. I've opened an issue in PyTorch in the past for that, see pytorch/pytorch#21064

GIven the above, I'd say that we should not try to make the API more complicated for some particular models (and not others?), and thus I'm closing this issue.
But please feel free to reopen it if you think otherwise.

[shankar-agarwal]

Hi, faster-rcnn (and mask-rcnn) gives the bounding box + classes as output. Is there a way to extract the feature vectors of these objects? For example, how can I get the activations before the last classifier?

Regards -Shankar

[dipam7]

You can use PyTorch hooks to get the input, weights, and outputs of any layer in the Fasterrcnn. For example, after defining the model if you do model.named_children() or model.named_modules() you can access these individual modules, and then you can attach a hook to them.

# initalize fasterrcnn
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained = True)
num_classes = 2 
in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

features = []
def save_features(mod, inp, outp):
    features.append(outp.data)

# you can also hook layers inside the roi_heads
layer_to_hook = 'roi_heads'
for name, layer in model.named_modules():
    if name == layer_to_hook:
        layer.register_forward_hook(save_features)

# train model and after training, you can see the output activations in the "features" list

[harshraj22]

@dipam7 Thank you for the code snippet.
Just curious, is there a way to get both bounding box and the corresponding features ?
Say the model has 4 detections, and I want the 4 tuples (bounding box and corresponding features).

[tejan-rgb]

https://medium.com/the-dl/how-to-use-pytorch-hooks-5041d777f904

This worked for me.

[abhi-glitchhg]

https://github.com/elliotwaite/pytorch-hooks-tutorial

This is a nice resource if someone wants to learn the basics of hooks.