Fbnet benchmark

[newstzpz]

FBNet benchmark

• Added some pre-trained models and benchmark results for FBNet.
• Added a timer to benchmark model inference time in addition to total run time.

We provided pre-trained models for selected FBNet models.

• All the models are trained from scratched with BN using the training schedule specified below.
• Evaluation is performed on a single NVIDIA V100 GPU with MODEL.RPN.POST_NMS_TOP_N_TEST set to 200.

The following inference time is reported:

• inference total batch=8: Total inference time including data loading, model inference and pre/post preprocessing using 8 images per batch.
• inference model batch=8: Model inference time only and using 8 images per batch.
• inference model batch=1: Model inference time only and using 1 image per batch.
• inferenee caffe2 batch=1: Model inference time for the model in Caffe2 format using 1 image per batch. The Caffe2 models fused the BN to Conv and purely run on C++/CUDA by using Caffe2 ops for rpn/detection post processing.

The pre-trained models are available in the link in the model id.

backbone	type	resolution	inference total batch=8 (s/im)	inference model batch=8 (s/im)	inference model batch=1 (s/im)	inference caffe2 batch=1 (s/im)	box AP	mask AP	model id
R-50-C4 (reference)	Fast	800	0.0875	0.0793	0.0831	0.0625	34.4	-	f35857197
fbnet_chamv1a	Fast	600	0.0315	0.0260	0.0376	0.0188	33.5	-	f100940543
fbnet_default	Fast	600	0.0316	0.0250	0.0297	0.0130	28.2	-	f101086388
R-50-C4 (reference)	Mask	800	0.0918	0.0848	0.0844	-	35.2	31.0	f35858791
fbnet_xirb16d	Mask	600	0.0386	0.0319	0.0356	-	30.7	26.9	f101086394
fbnet_default	Mask	600	0.0327	0.0269	0.0385	-	29.0	26.1	f101086385

[fmassa]

This looks awesome, thanks a lot!

I have some minor comments, and one remark about the timing of the forward during inference.
Note that adding a torch.cuda.synchronize() could potentially slow down a bit, but that should be verified.

[fmassa]

Could you expand on why those two cases fail?

[newstzpz]

It seemed to fail when no input to the keypoint head. The error message is follow:

  File "/Users/stzpz/code/newstzpz/forked/maskrcnn-benchmark/maskrcnn_benchmark/modeling/roi_heads/roi_heads.py", line 53, in forward
    x, detections, loss_keypoint = self.keypoint(keypoint_features, detections, targets)
  File "/anaconda3/envs/maskrcnn_benchmark/lib/python3.7/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/stzpz/code/newstzpz/forked/maskrcnn-benchmark/maskrcnn_benchmark/modeling/roi_heads/keypoint_head/keypoint_head.py", line 42, in forward
    result = self.post_processor(kp_logits, proposals)
  File "/anaconda3/envs/maskrcnn_benchmark/lib/python3.7/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/stzpz/code/newstzpz/forked/maskrcnn-benchmark/maskrcnn_benchmark/modeling/roi_heads/keypoint_head/inference.py", line 15, in forward
    mask_prob, scores = self.keypointer(x, boxes)
  File "/Users/stzpz/code/newstzpz/forked/maskrcnn-benchmark/maskrcnn_benchmark/modeling/roi_heads/keypoint_head/inference.py", line 114, in __call__
    assert len(boxes) == 1
AssertionError

[fmassa]

Ok, thanks!
I'll see if I fix this in the future

[rwwneu]

It seemed to fail when no input to the keypoint head. The error message is follow:

  File "/Users/stzpz/code/newstzpz/forked/maskrcnn-benchmark/maskrcnn_benchmark/modeling/roi_heads/roi_heads.py", line 53, in forward
    x, detections, loss_keypoint = self.keypoint(keypoint_features, detections, targets)
  File "/anaconda3/envs/maskrcnn_benchmark/lib/python3.7/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/stzpz/code/newstzpz/forked/maskrcnn-benchmark/maskrcnn_benchmark/modeling/roi_heads/keypoint_head/keypoint_head.py", line 42, in forward
    result = self.post_processor(kp_logits, proposals)
  File "/anaconda3/envs/maskrcnn_benchmark/lib/python3.7/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/stzpz/code/newstzpz/forked/maskrcnn-benchmark/maskrcnn_benchmark/modeling/roi_heads/keypoint_head/inference.py", line 15, in forward
    mask_prob, scores = self.keypointer(x, boxes)
  File "/Users/stzpz/code/newstzpz/forked/maskrcnn-benchmark/maskrcnn_benchmark/modeling/roi_heads/keypoint_head/inference.py", line 114, in __call__
    assert len(boxes) == 1
AssertionError

Can you tell me how to solve this problem? We look forward to your reply.thanks a lot

[zimenglan-sysu-512]

i have a suggestion: make a folder configs/fbnet and pull the corresponding yaml files into it.

[jario-jin]

I have a question, why does Fbnet not use FPN? Would it be ineffective to detect small objects?

[fmassa]

I've updated the README with the pre-trained models for Detection.
If you want to use it, just download it following the link on the MODEL_ZOO, and pass the MODEL.WEIGHT to point to the downloaded file.

Thanks a lot @newstzpz !