Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DNN: supports NonMaxSuppression operator from ONNX #22473

Draft
wants to merge 4 commits into
base: 4.x
Choose a base branch
from

Conversation

fengyuentau
Copy link
Member

@fengyuentau fengyuentau commented Sep 5, 2022

Fixes #19779. And one step forward to support the ssd-mobilenet from onnx model zoo.

opencv/opencv_extra#1005

TODO list:

  • Supports the case where exporting a PyTorch model with torchvision.ops.nms to ONNX.
  • Supports the case where exporting a PyTorch model with torchvision.ops.batched_nms to ONNX.

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

  • I agree to contribute to the project under Apache 2 License.
  • To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
  • The PR is proposed to the proper branch
  • There is a reference to the original bug report and related work
  • There is accuracy test, performance test and test data in opencv_extra repository, if applicable
    Patch to opencv_extra has the same branch name.
  • The feature is well documented and sample code can be built with the project CMake

NMSFast_(boxes, scores, score_threshold, iou_threshold, 1.0, top_k, keep_indices, rect2dOverlap, keep_top_k);

// Store to output
outputs[0].setTo(-1);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since dnn does not support dynamic shape for now, the number of output boxes have to be set fixed (650 is used in this implementation, but there will not be so many boxes in one single image typically speaking). So invalid indices have to be set to -1.

@rogday if the nms operator is ported from torchvision.ops.nms, the NonMaxSuppression operator is always followed by a Gather operator (see the visualization in opencv/opencv_extra#1005). I wonder if it is reasonable to drop those invalid indices in your pull request for the Gather operator.

@fengyuentau
Copy link
Member Author

I would like to postphone this pull request, since I cannot find a workaround for the dynamic output of nms. Will get back to this pull request when we have next gen dnn which supports dynamic shape.

@dkurt
Copy link
Member

dkurt commented Oct 24, 2022

Can this op fit to DetectionOutput layer?

@fengyuentau
Copy link
Member Author

Can this op fit to DetectionOutput layer?

Yes and no. They all do NMS but DetectionOutput layer tries to generate priors and decode bboxes as well, which are not needed in the NonMaxSuppression operator from ONNX. Another problem is DetectionOutput layer has topK and keepTopK to limit the number of output bboxes, but NonMaxSuppression does not. So the output is dynamic for NonMaxSuppression and therefore this PR has to be postponed util we have dynamic shape support in the next generation of DNN module.

@fengyuentau fengyuentau mentioned this pull request Mar 14, 2023
48 tasks
@dkurt
Copy link
Member

dkurt commented May 23, 2023

And one step forward to support the ssd-mobilenet from onnx model zoo.

I were not able to find NonMaxSuppression layers in SSD-MobilenetV1 and SSD-MobilenetV1-12 models.

@fengyuentau
Copy link
Member Author

I were not able to find NonMaxSuppression layers in SSD-MobilenetV1 and SSD-MobilenetV1-12 models.

It is in the subgraph of the second Loop operator in SSD-MobilenetV1-12 for example.

@dkurt
Copy link
Member

dkurt commented May 23, 2023

image

@fengyuentau, cannot find it, maybe model has been updated?

@fengyuentau
Copy link
Member Author

Click on Loop and on the pop-up attribute page you can see there is a body attribute. Click on the name of the body attribute and you will be redirected to the subgraph, where you can find the NonMaxSuppression operators.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category: dnn (onnx) ONNX suport issues in DNN module
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Any plans to officially support the 'NonMaxSuppression' operator from ONNX?
3 participants