which network is suitalble for integrate tensorRT with tensorflow(TFTRT) optimiztion ？

[tilaba]

I‘m work with TFTRT for tensorflow speed-up on GPU. it works well in some classification networks, such as resnet50V2， inceptionV2，VGG19. However， when I try to accelerate the obeject dectetion network like SSD， faster rcnn and mask rcnn, the INT8 networks processed by TFTRT do not gain any advantages in speed over the original network. I doubt that the TFTRT do not favor obeject dectetion network well. And I have two additional questions. Is there some limitaions of the TFTRT? Is there difference in performing INT8 optimzation between tensorRT and TFTRT ? @aaroey

[aaroey]

@pooyadavoodi

[jillelajitta]

Inception V2 would be better.
Follow this link https://github.com/NVIDIA-AI-IOT/tf_trt_models

[aaroey]

For SSD, TF-TRT doesn't work well at the moment since it's very hard to pattern match the unrolled NMS ops and fuse them. We're working with NVIDIA to add native support for that and hopefully it'll make it much faster after that. For other networks, do you have a repro demonstrating that TF-TRT doesn't do a good job? I'll be happy to take a look if you have one.

Thanks.

[pooyadavoodi]

We are still working on supporting object detection models. SSD is in a better shape than Faster-RCNN and Mask-RCNN. This is mostly related to having better caching mechanism in the dynamic mode, and having a better support for dynamic input shapes.

As @aaroey said, NMS is also a problem currently. You can work around that by increasing the minimum_segment_size (to something like 50). This op would solve most of the problems: tensorflow/tensorflow#23567
We will also provide new APIs for this new NMS op in object detection of tensorflow/models.

Regarding precision, we are seeing relatively good speedup with FP16/Tensor Core (especially on SSD with inception), but we are still working on fixing some problems with INT8. There shouldn't be any difference between TensorRT and TF-TRT regarding INT8. If there is, we would consider that a bug and we are working on some bugs like that currently.

[inders]

I am trying with FasterRCNN and facing the following - Any pointers would help.

2018-12-13 09:49:37.098239: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:794] Failed to register segment graphdef as a function 20: Invalid argument: Node 'map_1/while/concat/axis': Unknown input node '^map_1/while/Identity
'
2018-12-13 09:49:37.105664: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2853] Segment @scope '', converted to graph
2018-12-13 09:49:37.161006: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:724] Can't determine the device, constructing an allocator at device 0
2018-12-13 09:49:37.182205: E tensorflow/contrib/tensorrt/log/trt_logger.cc:38] DefaultLogger Parameter check failed at: Network.cpp::addInput::281, condition: isIndexedCHW(dims) && volume(dims) < MAX_TENSOR_SIZE
2018-12-13 09:49:37.182317: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:857] Engine creation for segment 0, composed of 3 nodes failed: Invalid argument: Failed to create Input layer tensor InputPH_0 rank=-2. Skipping...
2018-12-13 09:49:37.182353: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:724] Can't determine the device, constructing an allocator at device 0
2018-12-13 09:49:37.184917: E tensorflow/contrib/tensorrt/log/trt_logger.cc:38] DefaultLogger Parameter check failed at: Network.cpp::addInput::281, condition: isIndexedCHW(dims) && volume(dims) < MAX_TENSOR_SIZE
2018-12-13 09:49:37.185008: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:857] Engine creation for segment 1, composed of 3 nodes failed: Invalid argument: Failed to create Input layer tensor InputPH_0 rank=-2. Skipping...
2018-12-13 09:49:37.185031: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:724] Can't determine the device, constructing an allocator at device 0
*** Error in `python3': munmap_chunk(): invalid pointer: 0x00007ffe6b0e1b10 ***

[Renart-fox]

@inders it seems like you have an issue with the Identity layer. I've read here that, I quote:

...the identity layer, used for classification with Mask R-CNN, is one of the non-supported layers when porting a TensorFlow-based neural network to the TensorRT runtime. This layer is responsible for moving data between GPU modules, among other tasks.