ssd_resnet50_v1_fpn_shared_box_predictor_640x640_coco

[PythonImageDeveloper]

Hi,
Have you experience the converting the ssd_resnet50_v1_fpn_shared_box_predictor_640x640_coco to uff and bin files?

[jkjung-avt]

No. I haven't tried. Maybe you could share your experience?

[PythonImageDeveloper]

Hi
If you know about this below definitions, please explain about this:

PriorBox = gs.create_plugin_node(name="GridAnchor", op="GridAnchor_TRT",
    minSize=0.2,
    maxSize=0.95,
    aspectRatios=[1.0, 2.0, 0.5, 3.0, 0.33],
    variance=[0.1,0.1,0.2,0.2],
    featureMapShapes=[19, 10, 5, 3, 2, 1], 
    numLayers=6
)
NMS = gs.create_plugin_node(
    name="NMS",
    op="NMS_TRT",
    shareLocation=1,
    varianceEncodedInTarget=0,
    backgroundLabelId=0,
    confidenceThreshold=1e-8,
    nmsThreshold=0.6,
    topK=100,
    keepTopK=100,
    numClasses=91,
    inputOrder=[1, 0, 2],
    confSigmoid=1,
    isNormalized=1,
    scoreConverter="SIGMOID"
)

I know these definitions are for creating a new node using graphsurgeon library, but I have some questions,
1- The name in this definition should be the name of the Tensorflow node name? and the op name is an arbitrary name or the supported name of graphsurgeon library?
2- If I want to create a new node using graphsurgeon library, where I know the specifications of this layer such as shareLocation=1, topK=100 and so on?

namespace_plugin_map = {
    "MultipleGridAnchorGenerator": PriorBox,
    "Postprocessor": NMS,
    "Preprocessor": Input,
    "ToFloat": Input,
    "image_tensor": Input,
    "MultipleGridAnchorGenerator/Concatenate": concat_priorbox,  # for 'ssd_mobilenet_v1_coco'
    "Concatenate": concat_priorbox,  # for other models
    "concat": concat_box_loc,
    "concat_1": concat_box_conf
}
graph.collapse_namespaces(namespace_plugin_map)

3- In the namespace_plugin_map dictionary, This means is that we want to change nodes from Tensorflow node like "Postprocessor" to graphsurgeon NMS node ?

[PythonImageDeveloper]

I want to convert ssd_resnet50_v1_fpn_shared_box_predictor_640x640_coco to uff file, I get this error:
[TensorRT] ERROR: UffParser: Parser error: **FeatureExtractor/resnet_v1_50/fpn/top_down/nearest_neighbor_upsampling/mul**: Invalid scale mode, nbWeights: 4
I check the FeatureExtractor/resnet_v1_50/fpn/top_down/nearest_neighbor_upsampling/mul inputs and output node:
inputs : [N,20,1,20,1,256] and [N,1,1,2,1,2,1]
output : [N,20,2,20,2,256]
Likely the tensorrt can't multiply two inputs tensors with those shape.I don't know this problem is solved in the new vesion of tensorrt.

similar errors:
https://devtalk.nvidia.com/default/topic/1063279/tensorrt/error-uffparser-parser-error-gender_resnet0_batchnorm0-mul-invalid-scale-mode-nbweights-3/post/5387684/#5387684

NVIDIA/TensorRT#104

[jkjung-avt]

It looks like this "FeatureExtractor/resnet_v1_50/fpn/top_down/nearest_neighbor_upsampling/mul" node is doing "elementwise product" of 2 tensors with broadcasting.

According to this table, IElementWiseLayer does support "broadcasting". So I think you could try to dig deeper:

• Does the "FeatureExtractor/resnet_v1_50/fpn/top_down/nearest_neighbor_upsampling/mul" get converted to an Elementwise Product OP in TensorRT engine?
• If not, try to use graphsurgeon to replace that node with an Elementwise Product OP.

[PythonImageDeveloper]

If it's doing "elementwise product" of 2 tensors with broadcasting, So the UFF library doesn't support this layer, right?
UFF Operations
1- As the UFF file correctly created, So the parser related to TensorRT, and this error happen in the parsing UFF to TensorRT, So TensorRT doesn't support this operation, right?
2- My TensorRT version is 5.x, likely this operation is supported in TensorRT 6.x, in your opinion, If I upgrade TensorRT version to 6.x, this problem will be resolved. right?
3- If I use gs.create_node or gs.create_plugin_node instead "FeatureExtractor/resnet_v1_50/fpn/top_down/nearest_neighbor_upsampling/mul" node, Is it possible to create Elementwise Product OP with this technic or need to use tensorrt.ElementWiseOperation ?

[jkjung-avt]

Interesting...

1. I don't find an UFF Operator (in the documentation link you've provided) which could perform elementwise product with broadcasting. So what UFF operator was written in the uff file for this layer? Was it a "binary" layer with "mul" operation?

   As you have said, TensorRT probably only supports such "mul" where the 2 input tensors are of the same shape.

2. I'm not sure if upgrading to Tensor 6 would solve the problem. But that's a possibility.

3. I was thinking about adding a couple of Tile Layers to convert the [N,1,1,2,1,2,1] tensor to [N,20,1,20,1,256] (matching the other one). But UFF doesn't seem to have such an operation.

   I don't have an answer for you right now. But as stated in my previous comment, TensorRT's Elementwise OP does support broadcasting. So this seems to be a limitation of UFF......

[PythonImageDeveloper]

1- Binary mul operation
If I want to use TensorRT 6.x, I must use Tensorflow-1.15 for install TensorRT 6.x, or can I use TensorRT.6.x... deb file for install? I installed Tensorflow-1.12.2.

[jkjung-avt]

I missed this question earlier. The UFF (I believe is 0.6.5) in TensorRT 6.x was tested against tensorflow-1.14.0 (according to UFF's own log). So tensorflow 1.14 or 1.15 is better. That being said, I think tensorflow-1.12.2 would likely work OK, too.

[mw3155]

Hey @PythonImageDeveloper ,
I also want to try to run a SSD model with FPN on Jetson.

Did you have any success?

[PythonImageDeveloper]

Hi
No, I currently working on another project, If you can run the model on jetson, please let me know.

[jkjung-avt]

I think I won't spend more time looking into this issue ("ssd_resnet50_v1_fpn_shared_box_predictor_640x640_coco")...