TensorFlow Lite vs TensorFlow-TRT vs TesnorRT

Hello

I am doing a benchmark between the Jetson Family (TensorRT, TensorFlow-TRT) and Coral (TensorFlow Lite).

1. Can TensorFlow Lite work with any type of GPU? Or ONLY with GPUs of mobile and embedded devices?
   For example, the NVIDIA Tesla T4 has a precision of INT8 like the Coral TPU.

2. How does TensorFlow Lite optimize? Does it work like Tensorflow-TRT?
   https://medium.com/tensorflow/high-performance-inference-with-tensorrt-integration-c4d78795fbfe
   • Elimination of layers whose outputs are not used
   • Elimination of operations which are equivalent to no-op
   • The fusion of convolution, bias and ReLU operations
   • Aggregation of operations with sufficiently similar parameters and the same source tensor (for example, the 1x1 convolutions in GoogleNet v5’s inception module)
   • Merging of concatenation layers by directing layer outputs to the correct eventual destination.

Thank you

[newbie-gk]

Would love to see the benchmark results :)

[impjdi]

1. You might be able to run TFLite GPU on a desktop with OpenCL installed. One of our core engineers managed to do things on his desktop with an Nvidia graphics card, but it's not tuned for it. Our workgroup sizes are optimized for mobile GPUs primarily. Also, we can't currently offer support for TFLite GPU on the desktop; you're on your own.

2. TFLite GPU does some of those techniques you mention, but doesn't do all of them. For example, we may do fusion and merging of ops, but don't eliminate layers whose outputs are not used. It also does unique things that are very specific to GPUs, so you can't have a real apples to apples comparison.