How to use NMS with Pytorch model (that was converted to ONNX -> TensorRT) #795
Comments
|
You can use ONNX-GraphSurgeon to modify the ONNX model to include a plugin node. You can look at the onnx_packnet example for details. |
|
Wow, ONNX-GS, nice! And just to be 100% sure: it's possible to use ONNX-GS in the same way as the regular GS to use this CPP NMS plugin (https://github.com/NVIDIA/TensorRT/tree/master/plugin/batchedNMSPlugin), correct? |
|
Yes, correct. You can replace the old NMS subgraph/node with a new node whose Old-GS StyleIn the simplest case, you can create the node and insert it into the graph like old GS, e.g.: # Find tensors
tmap = graph.tensors()
boxes, scores, nms_out = tmap["boxes"], tmap["scores"], tmap["nms_out"]
# Disconnect old subgraph
boxes.outputs.clear()
scores.outputs.clear()
nms_out.inputs.clear()
attrs = {
"share_location": False,
"num_classes": 8,
# etc.
}
node = gs.Node(op="BatchedNMS_TRT", attrs=attrs,
inputs=[boxes, scores], outputs=[nms_out])
graph.nodes.append(node)Layer + Register APIIf you're reusing this across multiple models, or using the plugin multiple times in the model, then the layer/register API might simplify things. First you'd register a function to insert the plugin: @gs.Graph.register()
def trt_batched_nms(self, boxes_input, scores_input, nms_output,
share_location, num_classes): # and other attrs
boxes_input.outputs.clear()
scores_input.outputs.clear()
nms_output.inputs.clear()
attrs = {
"share_location": share_location,
"num_classes": num_classes,
# etc.
}
return self.layer(op="BatchedNMS_TRT", attrs=attrs,
inputs=[boxes_input, scores_input],
outputs=[nms_output])And then in your models, you can use it without all the boilerplate: tmap = graph.tensors()
# Can also get attributes from the original graph instead of hard-coding
graph.trt_batched_nms(tmap["boxes"], tmap["scores"],
tmap["nms_out"], share_location=True,
num_classes=81)
graph.trt_batched_nms(tmap["boxes2"], tmap["scores2"],
tmap["nms_out2"], share_location=False,
num_classes=4) |
|
Hi, @pranavm-nvidia, I tried using the ONNX-GS method as you proposed and I've successfully replaced the NMS operator with BatchedNMS_TRT using the following code: import onnx_graphsurgeon as gs
import onnx
import numpy as np
input_model_path = "model.onnx"
output_model_path = "model_gs.onnx"
@gs.Graph.register()
def trt_batched_nms(self, boxes_input, scores_input, nms_output,
share_location, num_classes):
boxes_input.outputs.clear()
scores_input.outputs.clear()
nms_output.inputs.clear()
attrs = {
"shareLocation": share_location,
"numClasses": num_classes,
"backgroundLabelId": 0,
"topK": 116740,
"keepTopK": 100,
"scoreThreshold": 0.3,
"iouThreshold": 0.6,
"isNormalized": True,
"clipBoxes": True
}
return self.layer(op="BatchedNMS_TRT", attrs=attrs,
inputs=[boxes_input, scores_input],
outputs=[nms_output])
graph = gs.import_onnx(onnx.load(input_model_path))
graph.inputs[0].shape=[1,1280,720,3]
print(graph.inputs[0].shape)
for inp in graph.inputs:
inp.dtype = np.int
input = graph.inputs[0]
tmap = graph.tensors()
graph.trt_batched_nms(tmap["Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/non_max_suppression_with_scores_2/NonMaxSuppressionV5__1712:0"],
tmap["Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/non_max_suppression_with_scores/NonMaxSuppressionV5__1761:0"],
tmap["NonMaxSuppression__1763:0"],
share_location=False,
num_classes=4)
graph.trt_batched_nms(tmap["Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/non_max_suppression_with_scores_2/NonMaxSuppressionV5__1712:0"],
tmap["Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/non_max_suppression_with_scores_1/NonMaxSuppressionV5__1737:0"],
tmap["NonMaxSuppression__1739:0"],
share_location=False,
num_classes=4)
graph.trt_batched_nms(tmap["Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/non_max_suppression_with_scores_2/NonMaxSuppressionV5__1712:0"],
tmap["Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/non_max_suppression_with_scores_2/NonMaxSuppressionV5__1713:0"],
tmap["NonMaxSuppression__1715:0"],
share_location=False,
num_classes=4)
graph.trt_batched_nms(tmap["Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/non_max_suppression_with_scores_2/NonMaxSuppressionV5__1712:0"],
tmap["Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/non_max_suppression_with_scores_3/NonMaxSuppressionV5__1689:0"],
tmap["NonMaxSuppression__1691:0"],
share_location=False,
num_classes=4)
# Remove unused nodes, and topologically sort the graph.
# graph.cleanup()
# graph.toposort()
# graph.fold_constants().cleanup()
# Export the ONNX graph from graphsurgeon
onnx.checker.check_model(gs.export_onnx(graph))
onnx.save_model(gs.export_onnx(graph), output_model_path)
print("Saving the ONNX model to {}".format(output_model_path))
The problem I'm currently facing is that onnx.onnx_cpp2py_export.checker.ValidationError:
Node (NonMaxSuppression__1763) has output size 0 not in range [min=1, max=1].
==> Context: Bad node spec:
input: "const_fold_opt__2119"
input: "Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/non_max_suppression_with_scores_3/iou_threshold:0"
input:"Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/non_max_suppression_with_scores_2/score_threshold:0"
name: "NonMaxSuppression__1763" op_type: "NonMaxSuppression"I think that the problem is in {
DEFINE_BUILTIN_OP_IMPORTER(NonMaxSuppression)
// NonMaxSuppression is not supported opset below 10.
ASSERT(ctx->getOpsetVersion() >= 10, ErrorCode::kUNSUPPORTED_NODE);
nvinfer1::ITensor* boxes_tensor = &convertToTensor(inputs.at(0), ctx);
nvinfer1::ITensor* scores_tensor = &convertToTensor(inputs.at(1), ctx);
const int numInputs = inputs.size();
LOG_ERROR("no of inputs are "<<numInputs);
LOG_ERROR("node outsize and op type are "<<node.output().size()<< " type " << node.op_type());
const auto scores_dims = scores_tensor->getDimensions();
const auto boxes_dims = boxes_tensor->getDimensions();
LOG_ERROR("boxes dims "<< boxes_dims.nbDims << " dim3 has size "<<boxes_dims.d[2]);
const std::string pluginName = "BatchedNMS_TRT";
const std::string pluginVersion = "1";
std::vector<nvinfer1::PluginField> f;
/*
bool share_location = true;
const bool is_normalized = true;
const bool clip_boxes = true;
int backgroundLabelId = 0;
// Initialize.
f.emplace_back("shareLocation", &share_location, nvinfer1::PluginFieldType::kINT8, 1);
f.emplace_back("isNormalized", &is_normalized, nvinfer1::PluginFieldType::kINT8, 1);
f.emplace_back("clipBoxes", &clip_boxes, nvinfer1::PluginFieldType::kINT8, 1);
f.emplace_back("backgroundLabelId", &backgroundLabelId, nvinfer1::PluginFieldType::kINT32, 1);
*/
// Create plugin from registry
//nvinfer1::IPluginV2* plugin = importPluginFromRegistry(ctx, pluginName, pluginVersion, node.name$
nvinfer1::IPluginV2* plugin = createPlugin(node.name(), importPluginCreator(pluginName, pluginVers$
ASSERT(plugin != nullptr && "NonMaxSuppression plugin was not found in the plugin registry!",
ErrorCode::kUNSUPPORTED_NODE);
std::vector<nvinfer1::ITensor*> nms_inputs ={boxes_tensor, scores_tensor};
RETURN_FIRST_OUTPUT(ctx->network()->addPluginV2(nms_inputs.data(), nms_inputs.size(), *plugin));
}Do you have an example of properly registering BatchedNMS_TRT as a plugin? |
|
@qraleq You should not need to modify the parser at all in TRT 7.1 and later (for any plugin, not just NMS). Modifying the node in the ONNX model should be enough. Since
I don't know of an end-to-end example that uses NMS specifically, but we do have an example of GroupNorm (onnx-packnet). However, I think the code you have right now should be nearly working if you uncomment these 2 lines: You can try the resulting ONNX model with |
|
@pranavm-nvidia Tnx for the fast response. I commented out the changes to the parser and rebuilt onnx-tensorrt. Now when I uncomment the two lines as you proposed, I get the following error when running with Do you have an idea what might be wrong? |
|
@qraleq Looks like your model is using dynamic shapes somewhere: Seems like the model input dimensions are fixed though, so it must be due to some intermediate layer whose dimensions can't be resolved at build time. The good news is that it does seem to be finding the plugin correctly at least: Also I think you'll probably need to populate the other parameters expected by the plugin (see here) or you might run into issues during inference. |
|
@pranavm-nvidia Yeah, I also figured that something "dynamic" is still leftover in the graph, but can't figure out what. I tried to use I'm using TensorRT 7.1.3.0-1+cuda10.2 on Jetson AGX. How can I make sure that dynamic NMS gets recognized by the parser? |
|
@qraleq The dynamic version was added in 7.2, so that would explain why it doesn't recognize it. Would you be able to share the verbose |
|
@pranavm-nvidia Please find the log attached. Can I send you the model directly somehow? |
|
@qraleq You can email me the model/link to the model at pranavm@nvidia.com Seems like it's this Resize that's introducing dynamic shapes: Note that the output of the previous layer has a fixed shape: Is it possible to express the resize using a constant |
|
@qraleq Please look at @pranavm-nvidia 's comment above to see if it fixes your problem. |
|
We discussed this offline, I believe @qraleq's issue is resolved now. |
|
@mk-nvidia Thank you for following up. @pranavm-nvidia was of great help and we resolved the issue! Best regards! |
|
@pranavm-nvidia or anyone, can you help me figure out the problem. I appended a BatchedNMS plugin to the end of an onnx file and can visually see the result in netron but I got segmentation fault in trtexec. Here is the trtexec output: Here are the code snippets to add the nms plugins : The output shapes of onnx file before appending the nms is like this:
|
|
@xonobo Can you try getting a backtrace with |
|
used gdb with trtexec_debug and got not much info verbose output states num_detections[Int32()]. |
|
@pranavm-nvidia I am having the same issue as @qraleq with regards to dynamic shapes. Could you please share your solution? |
|
@jonakola You should use the Since I'm using NVIDIA Jetsons which do not support this plugin at the moment, I ended up freezing the TensorFlow model to a fixed shape, and then everything worked out as supposed. |
|
Thanks @qraleq . I am also using NVIDIA Jetsons. How did you accomplish fixing the shape in TF in your case? I started off with a frozen_inference_graph.pb (exported after training with TF object detection api) which I would have thought already freezes all variables and shapes. |
|
@jonakola Explicitly define sizes when freezing the model from Take a look here: https://github.com/tensorflow/models/blob/master/research/object_detection/export_inference_graph.py and specifically: |
|
@xonobo It's weird that one dimension is being cut out from each of the output shapes. Is it possible for you to share your model? |
|
Here I provide two onnx files. I zero outed all weights in order to make them small. The smaller model passes trtexec but the bigger one fails as I stated above. I appended the nms plugin to the models using the same python script. |
|
@xonobo It looks like the |
|
Thanks for figuring out the problem but can you share your inspection way, I can not see any log report about the invalid parameter. If you found this case by using nvidia's internal tools can you add these invalid cases to public log reports in the next release. Thanks a lot for your help. |
|
Thank you for the previous help @qraleq. I am still unable to successfully convert my model from onnx to trt. Here are the specifics:
Looking at the batchedNMSPlugin.cpp file for TensorRT 7.1.3 tag, the error is raised in the BatchedNMSPlugin::getOutputDimensions function, specifically suggesting that the BatchedNMS plugin is not receiving an input with the shape it expects, which should be the detection boxes. I'm using TensorRT 7.1 on Jetson Nano. @pranavm-nvidia any ideas on what could be causing this? |
Hi @pranavm-nvidia, since the DS-triton with TRT 7.2.1 support was released I upgraded to 7.2.1 to optimize the model. The |
|
@vilmara The BatchedNMSDynamic plugin expects the input to have 4 dimensions: |
|
Hi @pranavm-nvidia,
Hi @pranavm-nvidia, I replaced the old NMS subgraph/node with new node using BatchedNMSDynamic_TRT plugin and maybe I left a dimension out, how can I check the model and see which one is missing? Here is how the output node looks after the conversion: |
|
@vilmara The issue is the previous layer that supplies the |
|
@pranavm-nvidia I am using the pre-trained model yolov3 from ONNX Model Zoo repo. I see this model only has 3 outputs What other object detection pre-trained models do you recommend? or what could be the workaround solution? |
|
@vilmara Have you looked at the YoloV3 example shipped with TensorRT? I think the issue with the model zoo model is that the ONNX spec expects a boxes input of shape: Also I'm not sure if the model is valid, since both TRT and ONNX-Runtime are having trouble with other layers in the model, e.g:
|
@pranavm-nvidia, I got it, I will explore different models/frameworks then. The issue with |
|
@pranavm-nvidia or @qraleq would you mind explaining a bit about the keys that you passed to the graph.tensor() as below I am using the sample that you have shared above to modify the onnx model which is SSD mobilenet V2 fpn lite created using tensorflow object detection API 2 and getting the below error File "modify_onnx.py", line 135, in main ==> Context: Bad node spec: input: "Unsqueeze__924:0" input: "Unsqueeze__959:0" input: "const_fold_opt__1412" input: "StatefulPartitionedCall/Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/non_max_suppression_with_scores/iou_threshold:0" input: "StatefulPartitionedCall/Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/non_max_suppression_with_scores_1/score_threshold:0" name: "NonMaxSuppression__961" op_type: "NonMaxSuppression" Am bit unsure how you passed keys to the graph.trt_batched_nms(). Below are the keys esp related to NMS when i print from graph.tensor() dict StatefulPartitionedCall/Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/non_max_suppression_with_scores/iou_threshold:0 |
|
I have used different approach to feed input as some thing like looping the below lists Looks like it is fine updating NMS with BatchedNMS_TRT based on netron output, attached for your reference But when i am converting from onnx to trt using the below command [04/20/2021-14:37:57] [V] [TRT] ModelImporter.cpp:90: Importing initializer: StatefulPartitionedCall/WeightSharedConvolutionalBoxPredictor/PredictionTower/conv2d_1/BatchNorm/feature_0/FusedBatchNormV3/ReadVariableOp_1:0 @pranavm-nvidia Anything that i missed or any inputs to further debug? |
|
@VeeranjaneyuluToka From the error output, it seems like it's failing on a Loop node, e.g. see the last So I'm not sure this is necessarily related to the BatchedNMS node. Maybe you could open a separate issue? |
Hi @MAhaitao999. A late reply but I guess you have to reshape the input blobs of the nms plugin as the one defined in the plugin documentation. In my case (shared prior boxes over classes) I need to reshape the boxes and update the inputs blobs of the plugin like this:
|
|
@pranavm-nvidia, would you mind sharing a reference or explaining a bit about topK, keeptopK parameters which are getting passed to self.layer in @qraleq nms plugin register sample? Do these parameters change from model to model? |
|
@VeeranjaneyuluToka The parameters are described in the plugin documentation here. The values you'd pass for them would be specific to your particular model/use-case. For
|
|
@pranavm-nvidia Thanks for reference and explanation about the requested parameters. I have debugged the tf model and looks like no.of boxes before NMS step is 51150, I have passed same thing while adding BatchedNMS_plugin, still it gives Segmentation fault as below (i was assuming that as i am passing wrong no boxes, it might be giving seg fault) but looks like that is not the case. [04/22/2021-07:42:04] [V] [TRT] ImporterContext.hpp:120: Registering tensor: map_while_map_while_loop_counter:0 for ONNX tensor: map_while_map_while_loop_counter:0 You have suggested me to raise a new issue on this, i did raise it #1205 but no reply yet. Could you please have a look and let me know how to analyze this issue further. |
Yes, exactly. We don't need to check 51150 boxes if you expect far fewer detections than that. In fact, the NMS plugin is currently limited to 4096 input boxes, so your |
hi, @@pranavm-nvidia, could you please help me with some problems about graph-surgeon? my onnx model is exported without nms, as the following code, so how can I add tmap['nms_out']? Thanks in advance. |
|
@Edwardmark You can create a new output tensor. Something like: import numpy as np
nms_out = gs.Variable(name="nms_out", dtype=np.float32, shape=(...)) # TODO: Fill out shape
graph.outputs = [nms_out]and then use that instead of |
|
Close since no activation for more than 3 weeks, please reopen if you still have question, thanks! |
Please was your model trained using TF2 api, how did you go about freezing the the shape. |
|
Why does BatchedNMSTRT using a different definition than standard onnx? onnx need input to be dimension: [num_batches, spatial_dimension, 4] what if I want graph surgeon to add a NonMaxSupression in onnx onset? How to expand that dimension? |
All right, so, I have a PyTorch detector SSD with MobileNet. Since I failed to convert model with NMS in it (to be more precise, I converted it, but TRT engine is built in a wrong way with that .onnx file), I decided to leave NMS part to TRT.
In general, there are several ways to add NMS in TRT:
But, I have a PyTorch model that I converted to onnx and then to TRT without any CPP code (Python only). My question is very simple: how can I combine my current pipeline with the CPP plugin for NMS?
The text was updated successfully, but these errors were encountered: