Converting TFLite model to run entirely on the Edge TPU as much as possible

[gsirocco]

Description

I have converted a TF model to TFLite and then to Edge TPU using the edgetpu_compiler. I have 3 operations that it says can run on the Edge TPU and 3 that can't. Output is below. I'm wondering if there's anyway to get the remaining 3 operations that run on CPU to run on Edge TPU? From the operations compatibility table (https://coral.ai/docs/edgetpu/models-intro/#supported-operations), it appears that FLOOR_MOD and DEQUANTIZE are not supported. Is there anything that I can change in the model to use somet different operation to substitute for FLOOR_MOD in order to run on Edge TPU? In terms of QUANTIZE and DEQUANTIZE operations, are these referring to quantization of input data for the model and then dequantization of the output data from the model? So these maybe can't be avoided not running on the Edge TPU, although the supported operations list does mention QUANTIZE as a supported operation for Runtime >=13.

Click to expand!

Issue Type

Build/Install, Performance, Support

Operating System

Ubuntu

Coral Device

Dev Board

Other Devices

No response

Programming Language

C++

Relevant Log Output

$ edgetpu_compiler -s testmodel2.tflite
Edge TPU Compiler version 16.0.384591198
Started a compilation timeout timer of 180 seconds.

Model compiled successfully in 1908 ms.

Input model: testmodel2.tflite
Input size: 24.72MiB
Output model: testmodel2_edgetpu.tflite
Output size: 25.60MiB
On-chip memory used for caching model parameters: 0.00B
On-chip memory remaining for caching model parameters: 7.66MiB
Off-chip memory used for streaming uncached model parameters: 25.49MiB
Number of Edge TPU subgraphs: 1
Total number of operations: 6
Operation log: testmodel2_edgetpu.log

Model successfully compiled but not all operations are supported by the Edge TPU. A percentage of the model will instead run on the CPU, which is slower. If possible, consider updating your model to use only operations supported by the Edge TPU. For details, visit g.co/coral/model-reqs.
Number of operations that will run on Edge TPU: 3
Number of operations that will run on CPU: 3

Operator                       Count      Status

DEQUANTIZE                     1          Operation is working on an unsupported data type
QUANTIZE                       1          Operation is otherwise supported, but not mapped due to some unspecified limitation
FULLY_CONNECTED                1          Mapped to Edge TPU
FLOOR_MOD                      1          Operation is working on an unsupported data type
RESHAPE                        1          Mapped to Edge TPU
CONCATENATION                  1          Mapped to Edge TPU
Compilation child process completed within timeout period.
Compilation succeeded!

[hjonnala]

it appears that FLOOR_MOD and DEQUANTIZE are not supported. Is there anything that I can change in the model to use somet different operation to substitute for FLOOR_MOD in order to run on Edge TPU?

Yes, FLOOR_MOD and DEQUANTIZE are not supported. We don't have any alternative supported operation for FLLOR_MOD.

In terms of QUANTIZE and DEQUANTIZE operations, are these referring to quantization of input data for the model and then dequantization of the output data from the model? So these maybe can't be avoided not running on the Edge TPU, although the supported operations list does mention QUANTIZE as a supported operation for Runtime >=13.

Quanitze operation can be mapped to EdgeTPU if its input either uint8 or init8. Here is an example model: https://raw.githubusercontent.com/google-coral/test_data/master/efficientdet_lite0_320_ptq.tflite

https://colab.sandbox.google.com/github/google-coral/tutorials/blob/master/retrain_efficientdet_model_maker_tf2.ipynb#scrollTo=XtxiUeZEiXpt

[gsirocco]

Thanks for the response. We will look into the example.

[google-coral-bot]

Are you satisfied with the resolution of your issue?
Yes
No