Recognize digits model (tensorflow version) on TensorRT

[sidgoyal78]

In order to compare performance of the C++ inference framework with that of TensorRT (#8671), we need to get the tensorflow model of recognize digits running with TensorRT. (As of now we need to use Tensorflow's model, as we don't have a Fluid to TensorRT converter yet.)

[sidgoyal78]

Update: I managed to get the Tensorflow model corresponding the recognize_digits example (by essentially using: @dzhwinter's example: https://github.com/dzhwinter/benchmark/blob/master/tensorflow/mnist.py), and successfully running with TensorRT.

I modified the sample Python examples are provided in the TensorRT container: https://devblogs.nvidia.com/tensorrt-container/ , to do the following:

• Train a recognize digits using TF in python
• Save the model
• Use the parser in TensorRT to convert this into a UFF format.
• Use python bindings of TensorRT to read and run this model.

I will put up the code somewhere (after figuring out the license/acknowledgement aspect, since i have modified the code which is classified as "commercial" by Nvidia).

Meanwhile, below are the results:

batch	time (ms)
1	0.084
2	0.114
8	0.087
32	0.131
128	0.145
256	0.148

The GPU that was used for profiling is a GeForce GTX 1080 Ti.

[Xreki]

Thanks.

The running time increases as the batch size. It is a little strange.

[sidgoyal78]

I think the reason for that is in the current benchmarking code, I have included time for memory allocation in the device and also the time taken to transfer the input array (from Python runtime) to device.

[sidgoyal78]

@Xreki : I have updated the timings. (I basically removed the cudaMalloc and memcopy parts, and just timed the execution part). Now I think the values make more sense.

[Xreki]

So, do you compare Fluid with TensorRT and what is the results?