Add Combined NMS #6138

pooyadavoodi · 2019-02-01T21:50:08Z

CombinedNMS op was recently added to TensorFlow.
tensorflow/tensorflow#23567

This PR adds CombinedNMS to the object detection API in TensorFlow.

The main difference between this new op and the old ones is that it does all of NMS in one op.
It's much more efficient, and it's also easier to swap out with a more efficient version of it from TF custom backends such as TensorRT and XLA.

pooyadavoodi · 2019-02-01T21:50:52Z

+@azaks2 @aaroey

aaroey · 2019-02-01T21:53:43Z

+@pengchongjin

dcyoung · 2019-02-02T17:18:37Z

@pooyadavoodi do you happen to have any ballpark performance metrics comparing inference before and after the change?

pooyadavoodi · 2019-02-04T23:47:24Z

@dcyoung In our inference benchmarks, I can see from 0.8x to 1.2x speedup when we use CombinedNMS. The smaller the batch size, the bigger the speedup. Note that <1x means regression.

Note that CombinedNMS only works on CPU. Someone might implement a GPU version of it in the future to make it much faster.

TF-TRT will be able to optimize CombinedNMS (even the CPU version) to TensorRT NMS and with that I can see about 2x speedup across the board.

pkulzc · 2019-02-10T05:32:05Z

research/object_detection/core/post_processing.py

+  """Multi-class version of non maximum suppression that operates on a batch.
+
+  This op is similar to `multiclass_non_max_suppression` but operates on a batch
+  of boxes and scores. See documentation for `multiclass_non_max_suppression`


Are these all simply copy/pasted? You should add some comments about what this method really do here.

@pkulzc could you PTAL. I just added a better explanation of what the function does.

pkulzc · 2019-02-10T05:32:30Z

research/object_detection/core/post_processing.py

+  with tf.name_scope(scope, 'CombinedNonMaxSuppression'):
+    nmsed_boxes, nmsed_scores, nmsed_classes, num_detections = tf.image.combined_non_max_suppression(
+            boxes, scores, max_size_per_class, max_total_size, iou_thresh,
+            score_thresh, use_static_shapes)


combined_non_max_suppression doesn't really have a input argument like this, right?

Which particular input are you referring to? The CombinedNonMaxSuppression has 6 inputs as defined in https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/ops/image_ops.cc#L913

There are 7 input arguments here while CombinedNonMaxSuppression only takes 6.

Sorry I missed the last attribute - use_static_shapes is the bool argument pad_per_class.

Could we use keyword argument for combined_non_max_suppression here?

pkulzc · 2019-02-10T05:32:45Z

research/object_detection/core/post_processing.py

+            score_thresh, use_static_shapes)
+
+    batch_nmsed_masks = None
+    batch_nmsed_additional_fields = None


Why are these not handled?

Currently the CombinedNonMaxSuppression does not have support to handle masks or additional fields. We can handle this once the kernel is updated to use these fields.

Could you add a comment mentioning masks and other fields are not supported when using combined_nms?

pkulzc

Thanks Pooya, just a few more nits.

pkulzc · 2019-05-20T21:59:10Z

research/object_detection/builders/post_processing_builder.py

                     'max_total_detections.')

-  non_max_suppressor_fn = functools.partial(
+  if(nms_config.combined_nms):


Nit: space between if and '('.

pkulzc · 2019-05-20T22:00:50Z

research/object_detection/core/post_processing.py

+  with tf.name_scope(scope, 'CombinedNonMaxSuppression'):
+    nmsed_boxes, nmsed_scores, nmsed_classes, num_detections = tf.image.combined_non_max_suppression(
+            boxes, scores, max_size_per_class, max_total_size, iou_thresh,
+            score_thresh, use_static_shapes)


Could we use keyword argument for combined_non_max_suppression here?

pkulzc · 2019-05-20T22:01:15Z

research/object_detection/core/post_processing.py

+            score_thresh, use_static_shapes)
+
+    batch_nmsed_masks = None
+    batch_nmsed_additional_fields = None


Could you add a comment mentioning masks and other fields are not supported when using combined_nms?

pkulzc · 2019-05-20T22:01:29Z

research/object_detection/core/post_processing.py

+
+    batch_nmsed_masks = None
+    batch_nmsed_additional_fields = None
+    #TODO supriyar: Set static shapes here?


Remove this?

pkulzc · 2019-05-20T22:02:23Z

research/object_detection/protos/post_processing.proto

  optional bool use_static_shapes = 6 [default = false];
+
+  // Whether to use the Combined NMS op
+  optional bool combined_nms = 7 [default = false];


use_combined_nms

pkulzc · 2019-05-20T22:03:21Z

research/object_detection/protos/faster_rcnn.proto

  optional bool use_static_shapes_for_eval = 37 [default = false];
+
+  // Whether to use Combined NMS for first stage.
+  optional bool first_stage_combined_nms = 38 [default=false];


use_combined_nms_in_first_stage

pkulzc · 2019-05-20T22:05:23Z

research/object_detection/core/post_processing_test.py

+      self.assertAllClose(nmsed_boxes, exp_nms_corners)
+      self.assertAllClose(nmsed_scores, exp_nms_scores)
+      self.assertAllClose(nmsed_classes, exp_nms_classes)
+      self.assertAllClose(num_detections, [3, 3])


assertListEqual

Change url to https://github.com/pooyadavoodi/models.git until PR tensorflow/models#6138 gets merged.

* Change url of tftrt/examples/third_party/models Change url to https://github.com/pooyadavoodi/models.git until PR tensorflow/models#6138 gets merged. * Update submodule tftrt/examples/third_party/models

pkulzc · 2019-06-05T22:24:18Z

Pooya, we recently updated the post_processing files and there may have some conflicts with your PR. Could you please resolve them?

pooyadavoodi · 2019-06-08T00:21:55Z

Thanks @pkulzc
I will check. There is also some problem in the PR that causes accuracy regression on Faster-RCNN and Mask-RCNN (not in SSD). I am looking into that too.

googlebot · 2019-06-08T00:21:58Z

We found a Contributor License Agreement for you (the sender of this pull request), but were unable to find agreements for all the commit author(s) or Co-authors. If you authored these, maybe you used a different email address in the git commits than was used to sign the CLA (login here to double check)? If these were authored by someone else, then they will need to sign a CLA as well, and confirm that they're okay with these being contributed to Google.
In order to pass this check, please resolve this problem and have the pull request author add another comment and the bot will run again. If the bot doesn't comment, it means it doesn't think anything has changed.

ℹ️ Googlers: Go here for more info.

1. Adds a unit test to test post_processing python API 2. Currently sets clip_window to None as the kernel uses the default clip_window of [0,0,1,1] 3. Added use_static_shapes to the API. In old API if use_static_shapes is true, then it pads/clips outputs to max_total_size, if specified. If not specified, it pads to num_classes*max_size_per_class. If use_static_shapes is false, it always pads/clips to max_total_size. Update unit test to account for clipped bouding boxes Changed the name to CombinedNonMaxSuppression based on feedback from Google Added additional parameters to combinedNMS python function. They are currently unused and required for networks like FasterRCNN and MaskRCNN

Because it was removed from CombinedNMS recently in the PR.

Remove redundant arguments from combined_nms

…sion Also rename combined_nms to use_combined_nms

googlebot · 2019-09-04T20:23:37Z

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

…ork" This reverts commit 2a3cc51.

pooyadavoodi · 2019-09-18T20:57:11Z

@pkulzc I have made some restructuring in the code. Could you please review.

FasterRCNN doesn't work with combined_nms in this PR mainly because of unsupported arguments. I have put some checks in the code to avoid calling combined_nms with those arguments, but some of those arguments can't be checked during graph construction because they are TF tensors (for those I print warning messages instead). We can add TF conditionals to the graph to check those if you think that's necessary (assuming they don't hurt the performance).

pkulzc

Looks good. Just a few nits.

pkulzc · 2019-09-24T08:23:40Z

research/object_detection/builders/post_processing_builder.py

  if nms_config.soft_nms_sigma < 0.0:
    raise ValueError('soft_nms_sigma should be non-negative.')
+  if nms_config.use_combined_nms:
+    if nms_config.use_class_agnostic_nms:


Merge two if clauses?

pkulzc · 2019-09-24T08:27:16Z

research/object_detection/protos/post_processing.proto

  // Soft NMS sigma parameter; Bodla et al, https://arxiv.org/abs/1704.04503)
  optional float soft_nms_sigma = 9 [default = 0.0];
+
+  // Whether to use tf.image.combined_non_max_suppression


Please add period to the end of your comments (same for your other changes in this PR as well).

pooyadavoodi · 2019-10-01T17:00:39Z

@pkulzc I think I've answered all the comments.
Please let me know what you think.

pkulzc

LGTM

aaroey · 2019-10-08T04:17:29Z

@pkulzc @pengchongjin could you help to merge this PR? Thanks.

* Change url of tftrt/examples/third_party/models Change url to https://github.com/pooyadavoodi/models.git until PR tensorflow/models#6138 gets merged. * Update submodule tftrt/examples/third_party/models

pooyadavoodi requested review from derekjchow, dreamdragon, jch1 and pkulzc as code owners February 1, 2019 21:50

googlebot added the cla: yes label Feb 1, 2019

aaroey requested a review from pengchongjin February 4, 2019 19:06

pkulzc suggested changes Feb 10, 2019

View reviewed changes

pkulzc suggested changes May 20, 2019

View reviewed changes

aaroey self-requested a review May 21, 2019 17:08

pooyadavoodi pushed a commit to pooyadavoodi/tensorrt that referenced this pull request May 23, 2019

Change url of tftrt/examples/third_party/models

a8d6b30

Change url to https://github.com/pooyadavoodi/models.git until PR tensorflow/models#6138 gets merged.

pooyadavoodi mentioned this pull request May 28, 2019

TensorRT speed up ssd_mobilenet_v1_fpn_coco #6344

Closed

googlebot added cla: no and removed cla: yes labels Jun 8, 2019

pooyadavoodi force-pushed the combined_nms branch from 7a53520 to b7d2704 Compare August 7, 2019 17:26

pooyadavoodi changed the title ~~Add Combined NMS~~ WIP: Add Combined NMS Aug 7, 2019

Pooya Davoodi added 7 commits September 4, 2019 13:23

Delete selected_indices from API

2861b65

Because it was removed from CombinedNMS recently in the PR.

Improve doc of function combined_non_max_suppression

d30c80b

Enable CombinedNonMaxSuppression for first_stage_nms

099e4fb

fix bug

3636aa8

Ensure agnostic_nms is not used with combined_nms

282fbac

Remove redundant arguments from combined_nms

Fix pylint

0334774

Pooya Davoodi added 4 commits September 4, 2019 13:23

Add checks for unsupported args

c895719

Fix pylint

c399bc2

Move combined_non_max_suppression to batch_multiclass_non_max_suppres…

105c945

…sion Also rename combined_nms to use_combined_nms

Delete combined_nms for first_stage_nms because it does not work

2a3cc51

pooyadavoodi force-pushed the combined_nms branch from 2affe0c to 2a3cc51 Compare September 4, 2019 20:23

googlebot added cla: yes and removed cla: no labels Sep 4, 2019

Pooya Davoodi added 5 commits September 4, 2019 13:24

Revert "Delete combined_nms for first_stage_nms because it does not w…

3d83064

…ork" This reverts commit 2a3cc51.

Use nmsed_additional_fields.get to avoid error

4caaa55

Merge combined_non_max_suppression with main nms function

8d54e50

Rename combined_nms for first stage nms

9b1953c

Improve docs

1d7812d

pooyadavoodi changed the title ~~WIP: Add Combined NMS~~ Add Combined NMS Sep 18, 2019

Use assertListEqual for numpy arrays

eb7c356

pooyadavoodi requested a review from pkulzc September 18, 2019 20:53

Fix pylint errors

1f9a447

pkulzc reviewed Sep 24, 2019

View reviewed changes

End comments with period

b6e3fdc

pkulzc approved these changes Oct 2, 2019

View reviewed changes

aaroey approved these changes Oct 8, 2019

View reviewed changes

saberkun merged commit 3980d2a into tensorflow:master Oct 9, 2019

jllarraz mentioned this pull request Nov 6, 2019

Object detection AttributeError: use_combined_nms #7762

Open

Add Combined NMS #6138

Add Combined NMS #6138

Uh oh!

Conversation

pooyadavoodi commented Feb 1, 2019

Uh oh!

pooyadavoodi commented Feb 1, 2019

Uh oh!

aaroey commented Feb 1, 2019

Uh oh!

dcyoung commented Feb 2, 2019

Uh oh!

pooyadavoodi commented Feb 4, 2019

Uh oh!

pkulzc Feb 10, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pkulzc May 20, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pkulzc left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pkulzc May 20, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pkulzc commented Jun 5, 2019

Uh oh!

pooyadavoodi commented Jun 8, 2019

Uh oh!

googlebot commented Jun 8, 2019

Uh oh!

googlebot commented Sep 4, 2019

Uh oh!

pooyadavoodi commented Sep 18, 2019

Uh oh!

pkulzc left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pooyadavoodi commented Oct 1, 2019

Uh oh!

pkulzc Feb 10, 2019 •

edited

Loading

pkulzc May 20, 2019 •

edited

Loading

pkulzc May 20, 2019 •

edited

Loading