[TF2 Object Detection] Converting SSD models into .tflite uint8 format

[SukyoungCho]

Hi, I was wondering if anyone could help how to convert and quantize SSD models on TF2 Object Detection Model Zoo.
It seems like there's a difference in converting to .tflite in TF1 and TF2. To the best of my knowledge, in TF1, we first frozen the model using exporter and then quantized and converted it into .tflite. And, I had no problem in doing it in TF1.
The models I tried was

SSD MobileNet V2 FPNLite 320x320	22	22.2	Boxes
SSD MobileNet V2 FPNLite 640x640	39	28.2	Boxes
SSD ResNet50 V1 FPN 640x640 (RetinaNet50)	46	34.3	Boxes

However, when I followed the guideline provided on the github repo 1.(https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/running_on_mobile_tf2.md#step-1-export-tflite-inference-graph) and 2. (https://www.tensorflow.org/lite/performance/post_training_quantization#full_integer_quantization). I was not able to convert them into .tflite.

Running "Step 1: Export TFLite inference graph", created saved_model.pb file in the given output dir {inside ./saved_model/}
However, it displayed the skeptic messages below while exporting them, and not sure if it's run properly.

Running "Step 2: Convert to TFLite", is the pain in the ass. I managed to convert the model generated in the step 1 into .tflite without any quantization following the given command, although I am not sure if it can be deployed on the mobile devices.

tflite_convert \
  --saved_model_dir=/tmp/mobilenet_saved_model \
  --output_file=/tmp/mobilenet.tflite

But, I am trying to deploy it on the board with the coral accelerator and need to conver the model into 'uint8' format. I thought the models provided on the model zoo are not QAT trained, and hence they require PTQ. Using the command line below,

import tensorflow as tf
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
def representative_dataset_gen():
  for _ in range(num_calibration_steps):
    # Get sample input data as a numpy array in a method of your choosing.
    yield [input]
converter.representative_dataset = representative_dataset_gen
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.int8  # or tf.uint8
converter.inference_output_type = tf.int8  # or tf.uint8
tflite_quant_model = converter.convert()

It shows the error message below, and i am not able to convert the model into .tflite format. I think the error occurs because something went wrong in the first step.

Below, I am attaching the sample script I used to run "Step 2". I have never train a model, and i am just trying to check if it is possible to convert SSD models on TF 2 OD API Model Zoo into Uint8 format .tflite. That is why, i dont have the sample data used to train the model, and just using MNIST data in Keras to save the time and cost to create data. (checkpoint CKPT = 0)

import tensorflow as tf

# Convert the model
converter = tf.lite.TFLiteConverter.from_saved_model("/root/ecomfort/tf2_model_zoo/tflite/ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8/saved_model")
converter.optimizations = [tf.lite.Optimize.DEFAULT]
#def representative_dataset_gen():
#  for _ in range(num_calibration_steps):
#    # Get sample input data as a numpy array in a method of your choosing.
#    yield [input]
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()
x_train = x_train.reshape(60000, 28, 28, 1).astype('float32') / 255
x_test = x_test.reshape(10000, 28,28, 1).astype('float32') / 255
y_train = y_train.astype('float32')
y_test = y_test.astype('float32')

def create_represent_data(data):
  def data_gen():
    for i in data:
      yield [list([i])]
  return data_gen
#converter.representative_dataset = representative_dataset_gen
converter.representative_dataset = create_represent_data(x_train[:5000])
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.uint8  # or tf.int8
converter.inference_output_type = tf.uint8  # or tf.int8
tflite_quant_model = converter.convert()

# Save the model.
with open('model.tflite', 'wb') as f:
  f.write(tflite_model)

The environment description.
CUDA = 10.1
Tensorflow = 2.3, 2.2 (both are tried)
TensorRT =
ii libnvinfer-plugin6 6.0.1-1+cuda10.1 amd64 TensorRT plugin libraries
ii libnvinfer6 6.0.1-1+cuda10.1 amd64 TensorRT runtime libraries

It would be appreciated if anyone could help to solve the issue, or provide a guideline. @srjoglekar246 Would you be able to provide the guideline or help me to convert models into uint8? Hope you documented while you were enabling SSD models to be converted into .tflite. Thank you so much.

**_

####Update####
I realized I dont need to do the first step, as I didnt do any training and the models are already in .pb file. But when I run the same python script I attached above, to convert to .tflite Uint8. It shows another error message below

_**

[albertoisorna]

Same problem here,

triying now with different models of the model ZOO, for the moment impossible for me to convert to TFLITE a mobilenet V2 with TF2 Object detection API.

[srjoglekar246]

You cannot directly convert the SavedModels from the detection zoo to TFLite, since it contains training parts of the graph that are not relevant to TFLite. Can you post the command you used to export the TFLite-friendly SavedModel using export_tflite_graph_tf2?

Running "Step 2: Convert to TFLite", is the pain in the ass. I managed to convert the model generated in the step 1 into .tflite without any quantization following the given command, although I am not sure if it can be deployed on the mobile devices.

Why unsure? What is the size of the converted model? You can use a tool like Netron to check if the .tflite model is valid.

For quantization, you need to implement representative dataset gen in such a way that it mimics how you would typically pass image tensors to TFLite models. It usually boils down to something like this:

def representative_dataset_gen():
  for i in range(100):
    image = tf.io.read_file(os.path.join(data_dir, image_files[i % NUM_FILES]))
    image = tf.compat.v1.image.decode_jpeg(image)
    image = preprocess(image)

    yield [image]

For the preprocess function, look at this implementation.

[SukyoungCho]

@srjoglekar246 Thanks for your reply.
I tried to export TFLite-friendly SavedModel using following command line. I am confused that how "TFLite-friendly SavedModel" is different from the SavedModel from Model Zoo as they seem almost same to me. In addition, although you are saying I cannot directly convert the SavedModels from the detection Zoo, It was converted to TFLite somehow... although im not sure why.

CUDA_VISIBLE_DEVICES=1 python object_detection/export_tflite_graph_tf2.py \
    --pipeline_config_path /root/ecomfort/tf2_model_zoo/ssd_mobilenet_v2_fpnlite_640x640_coco17_tpu-8/pipeline.config \
    --trained_checkpoint_dir /root/ecomfort/tf2_model_zoo/ssd_mobilenet_v2_fpnlite_640x640_coco17_tpu-8/checkpoint \
    --output_directory /root/ecomfort/tf2_model_zoo/tflite/ssd_mobilenet_v2_fpnlite_640x640_coco17_tpu-8

For quantization, I thought it could be any data as long as it is image tensors, and thats why I used MNIST data which does not need to be downloaded. By the way, I think I saw somewhere that "representative_dataset_gen()" or TF2 takes images as numpy array, so could you let me know which is correct? I will try to do quantization following your suggested command line. I will update it here. Thanks.

[SukyoungCho]

@srjoglekar246 Hi, I got this error while quantizing. Wonder if you are familiar with this kind of error.

[SukyoungCho]

@srjoglekar246 Hi, I tried to train the model from a model zoo and convert it to .tflite and quantize it. However, there were 2 critical issues.

1. During the training, at certain point, the loss becomes NaN.

2. Converting and quantizing the trained model, it still shows ValueError: Failed to parse the model: pybind11::init(): factory function returned nullptr. error

[srjoglekar246]

Not sure if there is a bug with training. The ODAPI folks might have a better idea of any issues that occur before using export_tflite_graph_tf2.py.

Can you show your exact code for tflite conversion? Though the model should probably be trained well first.

[SukyoungCho]

@srjoglekar246 This is how I ran the step 1

CUDA_VISIBLE_DEVICES=1 tflite_convert \
  --saved_model_dir=/root/ecomfort/tf2_model_zoo/tflite/ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8/saved_model \
  --output_file=/root/ecomfort/tf2_model_zoo/tflite/ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8/test.tflite

and this is how I ran the step 2
CUDA_VISIBLE_DEVICES=1 python demo_convert.py

and this is demo_convert

import tensorflow as tf

# Convert the model
converter = tf.lite.TFLiteConverter.from_saved_model("/root/ecomfort/tf2_model_zoo/tflite/ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8/saved_model")
converter.optimizations = [tf.lite.Optimize.DEFAULT]

def representative_dataset_gen():
  for i in range(100):
    #dir path where .jpg (image) files are
    image = tf.io.read_file(os.path.join("/root/ecomfort/data/valid_data/total/", image_files[i % NUM_FILES]))
    image = tf.compat.v1.image.decode_jpeg(image)
    image = preprocess(image)

    yield [image]

"""
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()
x_train = x_train.reshape(60000, 28, 28, 1).astype('float32') / 255
x_test = x_test.reshape(10000, 28,28, 1).astype('float32') / 255
y_train = y_train.astype('float32')
y_test = y_test.astype('float32')

def create_represent_data(data):
  def data_gen():
    for i in data:
      yield [list([i])]
  return data_gen
converter.representative_dataset = create_represent_data(x_train[:5000])
"""

converter.representative_dataset = representative_dataset_gen
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.uint8  # or tf.int8
converter.inference_output_type = tf.uint8  # or tf.int8
tflite_quant_model = converter.convert()

# Save the model.
with open('model.tflite', 'wb') as f:
  f.write(tflite_model)

[srjoglekar246]

@SukyoungCho How did you run export_tflite_graph_tf2.py? Note that the saved model directly from the model zoo will not convert to TFLite in a performant way (and won't quantize either), since it is not optimized for on-device inference.

In your demo_convert, how are you implementing preprocess?

[SukyoungCho]

@srjoglekar246 sorry for confusing. this is how i ran export_tflite_grpah_tf.py

CUDA_VISIBLE_DEVICES=1 python object_detection/export_tflite_graph_tf2.py \
    --pipeline_config_path /root/ecomfort/tf2_model_zoo/ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8/pipeline.config \
    --trained_checkpoint_dir /root/ecomfort/tf2_model_zoo/ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8/checkpoint \
    --output_directory /root/ecomfort/tf2_model_zoo/tflite/ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8

Yes, I ran about 12000 steps of training and tried converting it to tflite and quantize it.

For preprocessing, image = preprocess(image) this is all i have as I just copied and pasted the code you provided above. Should i modify it? thanks

[SukyoungCho]

@srjoglekar246 Should I implement preprocess within the script? like this?

import tensorflow as tf

# Convert the model
converter = tf.lite.TFLiteConverter.from_saved_model("/root/ecomfort/tf2_model_zoo/tflite/ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8/saved_model")
converter.optimizations = [tf.lite.Optimize.DEFAULT]

def preprocess(image,
                        height,
                        width,
                        central_fraction=0.875,
                        scope=None,
                        central_crop=True,
                        use_grayscale=False):
  with tf.name_scope(scope, 'eval_image', [image, height, width]):
    if image.dtype != tf.float32:
      image = tf.image.convert_image_dtype(image, dtype=tf.float32)
    if use_grayscale:
      image = tf.image.rgb_to_grayscale(image)
    # Crop the central region of the image with an area containing 87.5% of
    # the original image.
    if central_crop and central_fraction:
      image = tf.image.central_crop(image, central_fraction=central_fraction)

    if height and width:
      # Resize the image to the specified height and width.
      image = tf.expand_dims(image, 0)
      image = tf.image.resize_bilinear(image, [height, width],
                                       align_corners=False)
      image = tf.squeeze(image, [0])
    image = tf.subtract(image, 0.5)
    image = tf.multiply(image, 2.0)
    return image

def representative_dataset_gen():
  for i in range(100):
    #dir path where .jpg (image) files are
    image = tf.io.read_file(os.path.join("/root/ecomfort/data/valid_data/total/", image_files[i % NUM_FILES]))
    image = tf.compat.v1.image.decode_jpeg(image)
    image = preprocess(image)

    yield [image]

converter.representative_dataset = representative_dataset_gen
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.uint8  # or tf.int8
converter.inference_output_type = tf.uint8  # or tf.int8
tflite_quant_model = converter.convert()

# Save the model.
with open('model.tflite', 'wb') as f:
  f.write(tflite_model)

[srjoglekar246]

@SukyoungCho Yes.

[SukyoungCho]

@srjoglekar246 When I try to quantize the model, using the script above. It still gives me an error message below..

this is the exact script

import tensorflow as tf

# Convert the model
converter = tf.lite.TFLiteConverter.from_saved_model("/root/ecomfort/tf2_model_zoo/tflite/ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8/saved_model")
converter.optimizations = [tf.lite.Optimize.DEFAULT]

def preprocess(image,
                        height,
                        width,
                        central_fraction=0.875,
                        scope=None,
                        central_crop=True,
                        use_grayscale=False):
  with tf.name_scope(scope, 'eval_image', [image, height, width]):
    if image.dtype != tf.float32:
      image = tf.image.convert_image_dtype(image, dtype=tf.float32)
    if use_grayscale:
      image = tf.image.rgb_to_grayscale(image)
    # Crop the central region of the image with an area containing 87.5% of
    # the original image.
    if central_crop and central_fraction:
      image = tf.image.central_crop(image, central_fraction=central_fraction)

    if height and width:
      # Resize the image to the specified height and width.
      image = tf.expand_dims(image, 0)
      image = tf.image.resize_bilinear(image, [height, width],
                                       align_corners=False)
      image = tf.squeeze(image, [0])
    image = tf.subtract(image, 0.5)
    image = tf.multiply(image, 2.0)
    return image

def representative_dataset_gen():
  for i in range(100):
    #dir path where .jpg (image) files are
    image = tf.io.read_file(os.path.join("/root/ecomfort/data/valid_data/total/", image_files[i % NUM_FILES]))
    image = tf.compat.v1.image.decode_jpeg(image)
    image = preprocess(image)

    yield [image]

converter.representative_dataset = representative_dataset_gen
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.uint8  # or tf.int8
converter.inference_output_type = tf.uint8  # or tf.int8
tflite_quant_model = converter.convert()

# Save the model.
with open('model.tflite', 'wb') as f:
  f.write(tflite_model)

[srjoglekar246]

This seems like an error with post-training quantization. Adding @jianlijianli who might be able to help.

[srjoglekar246]

@SukyoungCho Can you confirm if conversion to float works?

[SukyoungCho]

@SukyoungCho Can you confirm if conversion to float works?

I remember it worked before. But, I will try it again and update it as soon as possible.

[SukyoungCho]

@srjoglekar246 Hi, when I try conversion to float 32 or float 16, the code does not show any error and gives out a .tflite model file, but it is suspicious. The model produced by step 1 was about 7mb. However, the .tflite file from step 2 is only about 560 bytes - less than 1 kb. Here I attach the model resulted from the step 1 (https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/running_on_mobile_tf2.md) and the result from the step 2 (.tflite conversion to float 32).

ssd_mobilenet_v2_fpnlite_320.zip

Step 1
Step 1: Export TFLite inference graph
This step generates an intermediate SavedModel that can be used with the TFLite Converter via commandline or Python API.

To use the script:

# From the tensorflow/models/research/ directory
python object_detection/export_tflite_graph_tf2.py \
    --pipeline_config_path path/to/ssd_model/pipeline.config \
    --trained_checkpoint_dir path/to/ssd_model/checkpoint \
    --output_directory path/to/exported_model_directory

[mpa74]

@srjoglekar246 Hi, when I try conversion to float 32 or float 16, the code does not show any error and gives out a .tflite model file, but it is suspicious. The model produced by step 1 was about 7mb. However, the .tflite file from step 2 is only about 560 bytes - less than 1 kb. Here I attach the model resulted from the step 1 (https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/running_on_mobile_tf2.md) and the result from the step 2 (.tflite conversion to float 32).

ssd_mobilenet_v2_fpnlite_320.zip

Step 1
Step 1: Export TFLite inference graph
This step generates an intermediate SavedModel that can be used with the TFLite Converter via commandline or Python API.

To use the script:

# From the tensorflow/models/research/ directory
python object_detection/export_tflite_graph_tf2.py \
    --pipeline_config_path path/to/ssd_model/pipeline.config \
    --trained_checkpoint_dir path/to/ssd_model/checkpoint \
    --output_directory path/to/exported_model_directory

Hi! Try this solution (#9394 (comment)) It solve problem for me (and converting to INT too)

[SukyoungCho]

@srjoglekar246 Hi, when I try conversion to float 32 or float 16, the code does not show any error and gives out a .tflite model file, but it is suspicious. The model produced by step 1 was about 7mb. However, the .tflite file from step 2 is only about 560 bytes - less than 1 kb. Here I attach the model resulted from the step 1 (https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/running_on_mobile_tf2.md) and the result from the step 2 (.tflite conversion to float 32).
ssd_mobilenet_v2_fpnlite_320.zip
Step 1
Step 1: Export TFLite inference graph
This step generates an intermediate SavedModel that can be used with the TFLite Converter via commandline or Python API.
To use the script:

# From the tensorflow/models/research/ directory
python object_detection/export_tflite_graph_tf2.py \
    --pipeline_config_path path/to/ssd_model/pipeline.config \
    --trained_checkpoint_dir path/to/ssd_model/checkpoint \
    --output_directory path/to/exported_model_directory

Hi! Try this solution (#9394 (comment)) It solve problem for me (and converting to INT too)

Hi @mpa74, thank you so much for your help. I have tried the solution you offered and I was able to get .tflite file with seemingly legit size. However, I was not able to quantize it into uint8 format.

In addition, when I perform step 1 and step 2, i am getting lots of error messages although it is giving out the converted file. So, I am not sure if they are converted properly. Could you let me know if you have experienced the same error message? Thank you so much.

##Error Message during step 1 (No Concrete functions found for untraced function)

W1022 17:38:15.026157 140288236451648 save.py:228] No concrete functions found for untraced function `projection_1_layer_call_and_return_conditional_losses` while saving. This function will not be callable after loading.
W1022 17:38:15.026237 140288236451648 save.py:228] No concrete functions found for untraced function `nearest_neighbor_upsampling_layer_call_fn` while saving. This function will not be callable after loading.
W1022 17:38:15.026307 140288236451648 save.py:228] No concrete functions found for untraced function `nearest_neighbor_upsampling_layer_call_and_return_conditional_losses` while saving. This function will not be callable after loading.
W1022 17:38:15.026376 140288236451648 save.py:228] No concrete functions found for untraced function `nearest_neighbor_upsampling_layer_call_and_return_conditional_losses` while saving. This function will not be callable after loading.
W1022 17:38:15.026456 140288236451648 save.py:228] No concrete functions found for untraced function `nearest_neighbor_upsampling_layer_call_fn` while saving. This function will not be callable after loading.
W1022 17:38:15.026524 140288236451648 save.py:228] No concrete functions found for untraced function `nearest_neighbor_upsampling_layer_call_and_return_conditional_losses` while saving. This function will not be callable after loading.
W1022 17:38:15.026593 140288236451648 save.py:228] No concrete functions found for untraced function `nearest_neighbor_upsampling_layer_call_and_return_conditional_losses` while saving. This function will not be callable after loading.
W1022 17:38:15.032178 140288236451648 save.py:228] No concrete functions found for untraced function `smoothing_2_layer_call_fn` while saving. This function will not be callable after loading.
W1022 17:38:15.032264 140288236451648 save.py:228] No concrete functions found for untraced function `smoothing_2_layer_call_and_return_conditional_losses` while saving. This function will not be callable after loading.
W1022 17:38:15.032336 140288236451648 save.py:228] No concrete functions found for untraced function `smoothing_2_layer_call_and_return_conditional_losses` while saving. This function will not be callable after loading.
W1022 17:38:15.037694 140288236451648 save.py:228] No concrete functions found for untraced function `smoothing_1_layer_call_fn` while saving. This function will not be callable after loading.
W1022 17:38:15.037776 140288236451648 save.py:228] No concrete functions found for untraced function `smoothing_1_layer_call_and_return_conditional_losses` while saving. This function will not be callable after loading.
W1022 17:38:15.037863 140288236451648 save.py:228] No concrete functions found for untraced function `smoothing_1_layer_call_and_return_conditional_losses` while saving. This function will not be callable after loading.
INFO:tensorflow:Unsupported signature for serialization: (([(<tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f9300ec47b8>, TensorSpec(shape=(None, 40, 40, 32), dtype=tf.float32, name='image_features/0/1')), (<tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f9300ec48d0>, TensorSpec(shape=(None, 20, 20, 96), dtype=tf.float32, name='image_features/1/1')), (<tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f9300ec45c0>, TensorSpec(shape=(None, 10, 10, 1280), dtype=tf.float32, name='image_features/2/1'))], False), {}).
I1022 17:38:18.807447 140288236451648 def_function.py:1170] Unsupported signature for serialization: (([(<tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f9300ec47b8>, TensorSpec(shape=(None, 40, 40, 32), dtype=tf.float32, name='image_features/0/1')), (<tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f9300ec48d0>, TensorSpec(shape=(None, 20, 20, 96), dtype=tf.float32, name='image_features/1/1')), (<tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f9300ec45c0>, TensorSpec(shape=(None, 10, 10, 1280), dtype=tf.float32, name='image_features/2/1'))], False), {}).
INFO:tensorflow:Unsupported signature for serialization: (([(<tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f9300f32cf8>, TensorSpec(shape=(None, 40, 40, 32), dtype=tf.float32, name='image_features/0/1')), (<tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f9300f32d30>, TensorSpec(shape=(None, 20, 20, 96), dtype=tf.float32, name='image_features/1/1')), (<tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f9300f32eb8>, TensorSpec(shape=(None, 10, 10, 1280), dtype=tf.float32, name='image_features/2/1'))], True), {}).

##Errors during the step 2

WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
2020-10-22 17:43:38.769464: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:316] Ignored output_format.
2020-10-22 17:43:38.769521: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:319] Ignored drop_control_dependency.
2020-10-22 17:43:38.769530: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:325] Ignored change_concat_input_ranges.
2020-10-22 17:43:38.770421: I tensorflow/cc/saved_model/reader.cc:32] Reading SavedModel from: ./saved_model
2020-10-22 17:43:38.838857: I tensorflow/cc/saved_model/reader.cc:55] Reading meta graph with tags { serve }
2020-10-22 17:43:38.838917: I tensorflow/cc/saved_model/reader.cc:93] Reading SavedModel debug info (if present) from: ./saved_model
2020-10-22 17:43:38.838986: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2020-10-22 17:43:38.839010: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-10-22 17:43:38.839019: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]
2020-10-22 17:43:39.042039: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2020-10-22 17:43:39.095908: I tensorflow/cc/saved_model/loader.cc:206] Restoring SavedModel bundle.
2020-10-22 17:43:39.154998: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 3591830000 Hz
2020-10-22 17:43:39.651706: I tensorflow/cc/saved_model/loader.cc:190] Running initialization op on SavedModel bundle at path: ./saved_model
2020-10-22 17:43:39.884801: I tensorflow/cc/saved_model/loader.cc:277] SavedModel load for tags { serve }; Status: success: OK. Took 1114382 microseconds.
2020-10-22 17:43:40.914707: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:194] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.
2020-10-22 17:43:41.582975: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2020-10-22 17:43:41.585994: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:0a:00.0 name: TITAN X (Pascal) computeCapability: 6.1
coreClock: 1.531GHz coreCount: 28 deviceMemorySize: 11.91GiB deviceMemoryBandwidth: 447.48GiB/s
2020-10-22 17:43:41.586061: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2020-10-22 17:43:41.586133: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2020-10-22 17:43:41.586188: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2020-10-22 17:43:41.586241: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2020-10-22 17:43:41.586310: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2020-10-22 17:43:41.586499: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2020-10-22 17:43:41.586580: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2020-10-22 17:43:41.586641: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2020-10-22 17:43:41.591077: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2020-10-22 17:43:41.591123: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-10-22 17:43:41.591139: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0
2020-10-22 17:43:41.591154: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N
2020-10-22 17:43:41.595432: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 11217 MB memory) -> physical GPU (device: 0, name: TITAN X (Pascal), pci bus id: 0000:0a:00.0, compute capability: 6.1)
Exception ignored in: <bound method Buckets.__del__ of <tensorflow.python.eager.monitoring.ExponentialBuckets object at 0x7f6e7542dc48>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/monitoring.py", line 407, in __del__
AttributeError: 'NoneType' object has no attribute 'TFE_MonitoringDeleteBuckets'
root@9e19458b0d02:/workspace#

[mpa74]

@srjoglekar246 Hi, when I try conversion to float 32 or float 16, the code does not show any error and gives out a .tflite model file, but it is suspicious. The model produced by step 1 was about 7mb. However, the .tflite file from step 2 is only about 560 bytes - less than 1 kb. Here I attach the model resulted from the step 1 (https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/running_on_mobile_tf2.md) and the result from the step 2 (.tflite conversion to float 32).
ssd_mobilenet_v2_fpnlite_320.zip
Step 1
Step 1: Export TFLite inference graph
This step generates an intermediate SavedModel that can be used with the TFLite Converter via commandline or Python API.
To use the script:

# From the tensorflow/models/research/ directory
python object_detection/export_tflite_graph_tf2.py \
    --pipeline_config_path path/to/ssd_model/pipeline.config \
    --trained_checkpoint_dir path/to/ssd_model/checkpoint \
    --output_directory path/to/exported_model_directory

Hi! Try this solution (#9394 (comment)) It solve problem for me (and converting to INT too)

Hi @mpa74, thank you so much for your help. I have tried the solution you offered and I was able to get .tflite file with seemingly legit size. However, I was not able to quantize it into uint8 format.

In addition, when I perform step 1 and step 2, i am getting lots of error messages although it is giving out the converted file. So, I am not sure if they are converted properly. Could you let me know if you have experienced the same error message? Thank you so much.

##Error Message during step 1 (No Concrete functions found for untraced function)

W1022 17:38:15.026157 140288236451648 save.py:228] No concrete functions found for untraced function `projection_1_layer_call_and_return_conditional_losses` while saving. This function will not be callable after loading.
W1022 17:38:15.026237 140288236451648 save.py:228] No concrete functions found for untraced function `nearest_neighbor_upsampling_layer_call_fn` while saving. This function will not be callable after loading.
W1022 17:38:15.026307 140288236451648 save.py:228] No concrete functions found for untraced function `nearest_neighbor_upsampling_layer_call_and_return_conditional_losses` while saving. This function will not be callable after loading.
W1022 17:38:15.026376 140288236451648 save.py:228] No concrete functions found for untraced function `nearest_neighbor_upsampling_layer_call_and_return_conditional_losses` while saving. This function will not be callable after loading.
W1022 17:38:15.026456 140288236451648 save.py:228] No concrete functions found for untraced function `nearest_neighbor_upsampling_layer_call_fn` while saving. This function will not be callable after loading.
W1022 17:38:15.026524 140288236451648 save.py:228] No concrete functions found for untraced function `nearest_neighbor_upsampling_layer_call_and_return_conditional_losses` while saving. This function will not be callable after loading.
W1022 17:38:15.026593 140288236451648 save.py:228] No concrete functions found for untraced function `nearest_neighbor_upsampling_layer_call_and_return_conditional_losses` while saving. This function will not be callable after loading.
W1022 17:38:15.032178 140288236451648 save.py:228] No concrete functions found for untraced function `smoothing_2_layer_call_fn` while saving. This function will not be callable after loading.
W1022 17:38:15.032264 140288236451648 save.py:228] No concrete functions found for untraced function `smoothing_2_layer_call_and_return_conditional_losses` while saving. This function will not be callable after loading.
W1022 17:38:15.032336 140288236451648 save.py:228] No concrete functions found for untraced function `smoothing_2_layer_call_and_return_conditional_losses` while saving. This function will not be callable after loading.
W1022 17:38:15.037694 140288236451648 save.py:228] No concrete functions found for untraced function `smoothing_1_layer_call_fn` while saving. This function will not be callable after loading.
W1022 17:38:15.037776 140288236451648 save.py:228] No concrete functions found for untraced function `smoothing_1_layer_call_and_return_conditional_losses` while saving. This function will not be callable after loading.
W1022 17:38:15.037863 140288236451648 save.py:228] No concrete functions found for untraced function `smoothing_1_layer_call_and_return_conditional_losses` while saving. This function will not be callable after loading.
INFO:tensorflow:Unsupported signature for serialization: (([(<tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f9300ec47b8>, TensorSpec(shape=(None, 40, 40, 32), dtype=tf.float32, name='image_features/0/1')), (<tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f9300ec48d0>, TensorSpec(shape=(None, 20, 20, 96), dtype=tf.float32, name='image_features/1/1')), (<tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f9300ec45c0>, TensorSpec(shape=(None, 10, 10, 1280), dtype=tf.float32, name='image_features/2/1'))], False), {}).
I1022 17:38:18.807447 140288236451648 def_function.py:1170] Unsupported signature for serialization: (([(<tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f9300ec47b8>, TensorSpec(shape=(None, 40, 40, 32), dtype=tf.float32, name='image_features/0/1')), (<tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f9300ec48d0>, TensorSpec(shape=(None, 20, 20, 96), dtype=tf.float32, name='image_features/1/1')), (<tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f9300ec45c0>, TensorSpec(shape=(None, 10, 10, 1280), dtype=tf.float32, name='image_features/2/1'))], False), {}).
INFO:tensorflow:Unsupported signature for serialization: (([(<tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f9300f32cf8>, TensorSpec(shape=(None, 40, 40, 32), dtype=tf.float32, name='image_features/0/1')), (<tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f9300f32d30>, TensorSpec(shape=(None, 20, 20, 96), dtype=tf.float32, name='image_features/1/1')), (<tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f9300f32eb8>, TensorSpec(shape=(None, 10, 10, 1280), dtype=tf.float32, name='image_features/2/1'))], True), {}).

##Errors during the step 2

WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
2020-10-22 17:43:38.769464: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:316] Ignored output_format.
2020-10-22 17:43:38.769521: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:319] Ignored drop_control_dependency.
2020-10-22 17:43:38.769530: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:325] Ignored change_concat_input_ranges.
2020-10-22 17:43:38.770421: I tensorflow/cc/saved_model/reader.cc:32] Reading SavedModel from: ./saved_model
2020-10-22 17:43:38.838857: I tensorflow/cc/saved_model/reader.cc:55] Reading meta graph with tags { serve }
2020-10-22 17:43:38.838917: I tensorflow/cc/saved_model/reader.cc:93] Reading SavedModel debug info (if present) from: ./saved_model
2020-10-22 17:43:38.838986: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2020-10-22 17:43:38.839010: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-10-22 17:43:38.839019: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]
2020-10-22 17:43:39.042039: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2020-10-22 17:43:39.095908: I tensorflow/cc/saved_model/loader.cc:206] Restoring SavedModel bundle.
2020-10-22 17:43:39.154998: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 3591830000 Hz
2020-10-22 17:43:39.651706: I tensorflow/cc/saved_model/loader.cc:190] Running initialization op on SavedModel bundle at path: ./saved_model
2020-10-22 17:43:39.884801: I tensorflow/cc/saved_model/loader.cc:277] SavedModel load for tags { serve }; Status: success: OK. Took 1114382 microseconds.
2020-10-22 17:43:40.914707: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:194] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.
2020-10-22 17:43:41.582975: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2020-10-22 17:43:41.585994: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:0a:00.0 name: TITAN X (Pascal) computeCapability: 6.1
coreClock: 1.531GHz coreCount: 28 deviceMemorySize: 11.91GiB deviceMemoryBandwidth: 447.48GiB/s
2020-10-22 17:43:41.586061: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2020-10-22 17:43:41.586133: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2020-10-22 17:43:41.586188: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2020-10-22 17:43:41.586241: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2020-10-22 17:43:41.586310: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2020-10-22 17:43:41.586499: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2020-10-22 17:43:41.586580: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2020-10-22 17:43:41.586641: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2020-10-22 17:43:41.591077: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2020-10-22 17:43:41.591123: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-10-22 17:43:41.591139: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0
2020-10-22 17:43:41.591154: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N
2020-10-22 17:43:41.595432: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 11217 MB memory) -> physical GPU (device: 0, name: TITAN X (Pascal), pci bus id: 0000:0a:00.0, compute capability: 6.1)
Exception ignored in: <bound method Buckets.__del__ of <tensorflow.python.eager.monitoring.ExponentialBuckets object at 0x7f6e7542dc48>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/monitoring.py", line 407, in __del__
AttributeError: 'NoneType' object has no attribute 'TFE_MonitoringDeleteBuckets'
root@9e19458b0d02:/workspace#

Hi @SukyoungCho ! Looks like I have similar warnings on step 1, but error with ‘NoneType’ object I have never received. Sorry, I don’t know what to do with this error

[srjoglekar246]

@mpa74 If a SavedModel is getting generated by step 1, the logs just seem to be warnings based on APIs etc. You can ignore them

For step 2, you also need this for conversion:

converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8, tf.lite.OpsSet.TFLITE_BUILTINS]

The final post-processing op needs to run in float, there is no quantized version for it.

[SukyoungCho]

@srjoglekar246
Hi, I do appreciate for your help! It's been a while, but I'm still struggling from exporting a model into uint8 quantized .tflite model. In the meantime, I have tried to convert not only SSD models but also EfficientDet and CenterNet, but all of them failed.

If you don't mind, could you elaborate more on what do you mean by "The final post-processing op needs to run in float, there is no quantized version for it." this? It sounds confusing to me, since the models from TF1 OD API are eligible for uint8 quantization, tflite conversion, and EdgeTPU compile while they have post processing ops on them. Is it due to a difference in tflite conversion processes for TF1 and TF2?

To be honest, the reason why I was looking for EfficientDet uint8 was to deploy it on a board with coral accelerator. I have been using ssd mobiledet_dsp_coco from TF1 OD API, but its accuracy was not satisfactory (while i was only detecting 'person' the AP was about 38 and had way too many false positive). However, I found out that on the same image set, CenterNet and EfficientDet showed extremely low false positive rate (I ran an inference on the server for this not on the board).

So it would be very appreciated if you could manage to release uint8 quantization for EfficientDet or CenterNet so that I can EdgeTPU compile it and deploy it on a board! Or, if you could tell me if there are any models better than SSD Mobiledet that can be quantized into uint8, or EfficientDet for TF1. Thank you so much!

[srjoglekar246]

It looks like the Coral team has uncovered some issues with quantization of detection models. Will ping back once I have updates.

[srjoglekar246]

Can you mention how you used export_tflite_graph_tf2.py? And sorry about the wrong Colab link, this is correct one.

[CaptainPineapple]

Can you mention how you used export_tflite_graph_tf2.py? And sorry about the wrong Colab link, this is correct one.

oh my... thank you so much.... I just realized that my conversion batch script did not use the correct exporting script.
I was using exporter_main_v2.py instead of export_tflite_graph_tf2.py .... such a stupid mistake...

So far things look more promising. model conversion now successfully terminates with the default settings and the interpreter of the resulting tflite model actually finishes the inference step.
All out values from this

out_boxes = interpreter.get_tensor(output_details[0]['index'])
out_classes = interpreter.get_tensor(output_details[1]['index'])
out_scores = interpreter.get_tensor(output_details[2]['index'])

are just 0.0 though. Might that be a matter of finetuning the representative dataset? Seems a bit weird that the boxes output is a single value.

[srjoglekar246]

The TF version you are using for conversion should be >=2.4. If an earlier version is used, the converter doesn't work with that the exporting script does.

[CaptainPineapple]

i am running this on tf version 2.5.0

[srjoglekar246]

Can you check the model size? If the conversion is happening incorrectly, thew model should be very small (few KB).
Also, maybe try float conversion first?

[CaptainPineapple]

so far integer quantization is not active. All still on default settings with float32 as datatype. tflite model file is currently on 440 bytes. That seems a bit small i guess?

[srjoglekar246]

Yeah the converter doesn't seem to be working, and this was mainly observed when the converter was using an older version of TFLite. Can you try in a virtualenv with the latest TF?

[CaptainPineapple]

okay, thanks will try that. Will need some time to set that up though as i currently do not have anaconda installed.
Will report back with the results. Either way thank you so much for your support so far!

[CaptainPineapple]

So i set up anaconda with python 3.9 and added tensorflow with all dependencies. I retrained my model over night on this fresh install and then did the checkpoint to saved_model conversion as well as the saved_model to tflite conversion from the py39 virtual env. The result remains the same with 440 bytes of model size.

EDIT:
to make sure the object detection api is correctly i wiped the existing installation and reran all setup steps.
model_build_tf2_tests runs without issues.

EDIT2:
my guess at this point is that the conversion to saved_model format already fails.
After readding the evaluation step to my script where i simply try to process a testimage with the loaded model the result is unusable as the resultobj looks like this:
outputs: [<tf.Tensor 'Identity:0' shape=() dtype=float32>, <tf.Tensor 'Identity_1:0' shape=() dtype=float32>, <tf.Tensor 'Identity_2:0' shape=() dtype=float32>, <tf.Tensor 'Identity_3:0' shape=() dtype=float32>]

while the model converted with exporter_main_v2.py shows this output for the same code:
outputs: [<tf.Tensor 'Identity:0' shape=(1, 100) dtype=float32>, <tf.Tensor 'Identity_1:0' shape=(1, 100, 4) dtype=float32>, <tf.Tensor 'Identity_2:0' shape=(1, 100) dtype=float32>, <tf.Tensor 'Identity_3:0' shape=(1, 100, 4) dtype=float32>, <tf.Tensor 'Identity_4:0' shape=(1, 100) dtype=float32>, <tf.Tensor 'Identity_5:0' shape=(1,) dtype=float32>, <tf.Tensor 'Identity_6:0' shape=(1, 6669, 4) dtype=float32>, <tf.Tensor 'Identity_7:0' shape=(1, 6669, 4) dtype=float32>]

i'll investigate this further to try and figure out why this happens. I assume that the latter version is the correct desired output.

EDIT3:
i guess the prior described difference in outputs is not an error and rather a misunderstanding on my side: Am i correct that the result from export_tflite_graph_tf2 is only to be used to further transform it into a tf-lite model?
This assumption is taken from the following comment:
#9033 (comment)

[srjoglekar246]

i guess the prior described difference in outputs is not an error and rather a misunderstanding on my side: Am i correct that the result from export_tflite_graph_tf2 is only to be used to further transform it into a tf-lite model?

Yup :-)

I wonder why your TFLite model isn't coming out right though. Can you try running this workflow in your environment? If the code in that Colab runs, then the problem is with what we are doing for your use-case. If the Colab also fails, the issue is probably with the system setup.

[CaptainPineapple]

i guess the prior described difference in outputs is not an error and rather a misunderstanding on my side: Am i correct that the result from export_tflite_graph_tf2 is only to be used to further transform it into a tf-lite model?

Yup :-)

I wonder why your TFLite model isn't coming out right though. Can you try running this workflow in your environment? If the code in that Colab runs, then the problem is with what we are doing for your use-case. If the Colab also fails, the issue is probably with the system setup.

i'll try but even the start is already a bit messy as it apparently is not made to run on python 3.9.
some of the module require time.clock which has been removed starting at python 3.8. This already excludes the use of the following modules:
object_detection.utils.colab_utils
IPython.display.Image
IPython.display.display
Are there any parts that you'd deem skippable while still providing information if it is a setup or model issue?

EDIT: i guess skipping parts is not really possible as it all just builds on top of each other.
I'll try refeactoring it as far as possible.

[srjoglekar246]

Its mainly the TFLite section towards the end, instead of fine-tuning a model you can just use the SSD MobileNet downloaded from model zoo

[CaptainPineapple]

okay well that is intresting and confusing (at least to me)
the workflow you provided works as intended (as far as i can tell) and i got to a .tflite file with ~10KB.
Btw the link in your workflow description to the zoo models is broken. The url misses the "md" at the end.
I then ran the tflite_convert command for the model i created (the one that i failed to properly convert up to now) and i got a tflite model ~18KB that is loadable in python, was successfully used to create an interpreter and run inference for a testimage.

So from my understanding the code in python that i was trying to create to convert my saved_model to a tflite model was basically the way that the tflite_convert command uses just that it enables to option to add extra optimization steps as well as quantization. Is that correct?

[srjoglekar246]

No, that model size doesn't sound right. AFAIK, it only happens when an older version of the converter doesn't know how to handle the TFLite-friendly SavedModel - as a result, it outputs a small model with just some zeros as outputs. Any converter after version 2.4 should be able to handle it, but in your setup for some reason it doesn't :-/

[CaptainPineapple]

No, that model size doesn't sound right. AFAIK, it only happens when an older version of the converter doesn't know how to handle the TFLite-friendly SavedModel - as a result, it outputs a small model with just some zeros as outputs. Any converter after version 2.4 should be able to handle it, but in your setup for some reason it doesn't :-/

ah hold on i was just too stupid: its 10'000KB -> 10MB
In the end as i wrote above: the tflite mode works. I can load it. I can run inference on it.
At lease for me that sounds like the conversion was successful :)

[srjoglekar246]

Oh I just saw your code carefully again, and looks like you are using from_concrete_functions. We need to use from_saved_model for the converter to read the annotations that export_tflite_graph_t2.py puts into the model :-)

[CaptainPineapple]

Oh I just saw your code carefully again, and looks like you are using from_concrete_functions. We need to use from_saved_model for the converter to read the annotations that export_tflite_graph_t2.py puts into the model :-)

ah yes that makes sense as i added this as a workaround that was needed for the wrongly created saved_model file.
And that apparently was the issue all along. model converts and inference returns reasonable data.
Thank you so much for your support @srjoglekar246 . You really are my lifesaver here.
As the basic setup now works correctly i'll try to figure out the quantization myself.

TL;DR for others up to here:

• Use the correct script for model conversion: do use exporter_main_v2.py if you want to use the saved_model for direct inference. Use export_tflite_graph_tf2.py for further conversion into tflite.
• DIRECTLY import the model like this: converter = tf.lite.TFLiteConverter.from_saved_model(MODEL_PATH)
• run the steps Load mobile-friendly model and Model Conversion from this workflow to check your setup for possible issues.

[srjoglekar246]

Nice! and happy to help :-)

[yeshbourne]

#!/usr/bin/python

import numpy as np
import tensorflow as tf

#path to images (100 sample images used for training)
image_data = tf.data.Dataset.list_files('./quant-images/*.jpg')
(HEIGHT, WIDTH) = (640, 640)

def representative_dataset_gen():
for image_path in image_data:
img = tf.io.read_file(image_path)
img = tf.io.decode_image(img, channels=3)
img = tf.image.convert_image_dtype(img, tf.float32)
resized_img = tf.image.resize(img, (HEIGHT, WIDTH))
resized_img = resized_img[tf.newaxis, :]
yield [resized_img]

model_path = './exported-models/ptag-detector-model/ssd_mobilenet_v2_fpnlite_640x640/saved_model'

#import trained model from mobilenet 640 v2 fpn

converter = tf.lite.TFLiteConverter.from_saved_model(model_path) # using tensorflow

converter.representative_dataset = representative_dataset_gen
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8
converter.quantized_input_stats = {'normalized_input_image_tensor': (128, 128)}
converter.optimizations = [tf.lite.Optimize.DEFAULT]

tflite_model_quant = converter.convert()

interpreter = tf.lite.Interpreter(model_content=tflite_model_quant)
input_type = interpreter.get_input_details()[0]['dtype']
print ('input: ', input_type)
output_type = interpreter.get_output_details()[0]['dtype']
print ('output: ', output_type)

with open('detect_quant.tflite', 'wb') as f:
f.write(tflite_model_quant)

[canseloguzz]

For those who are still struggling to get the model converted here's my code for full quantization for edge-tpu compatibility

import numpy as np
import tensorflow as tf

#path to images
image_data = tf.data.Dataset.list_files("./quant-images/*.jpg")
HEIGHT, WIDTH = 640, 640

def representative_dataset_gen():
for image_path in image_data:
img = tf.io.read_file(image_path)
img = tf.io.decode_image(img, channels=3)
img = tf.image.convert_image_dtype(img, tf.float32)
resized_img = tf.image.resize(img, (HEIGHT, WIDTH))
resized_img = resized_img[tf.newaxis, :]
yield [resized_img]

model_path = './exported-models/ptag-detector-model/ssd_mobilenet_v2_fpnlite_640x640/saved_model'

#import trained model from mobilenet 640 v2 fpn
converter = tf.lite.TFLiteConverter.from_saved_model(model_path) #using tensorflow

converter.representative_dataset = representative_dataset_gen
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8
converter.quantized_input_stats = {"normalized_input_image_tensor": (128, 128)}
converter.optimizations = [tf.lite.Optimize.DEFAULT]

tflite_model_quant = converter.convert()

interpreter = tf.lite.Interpreter(model_content=tflite_model_quant)
input_type = interpreter.get_input_details()[0]['dtype']
print('input: ', input_type)
output_type = interpreter.get_output_details()[0]['dtype']
print('output: ', output_type)

with open('detect_quant.tflite', 'wb') as f:
f.write(tflite_model_quant)

Hello, thanks for sharing code. I want to try but can you explain what is the "./quant-images/*.jpg" . I'm working with coco dataset. What ı will give there? Thanks.

[yeshbourne]

For those who are still struggling to get the model converted here's my code for full quantization for edge-tpu compatibility
import numpy as np
import tensorflow as tf
#path to images
image_data = tf.data.Dataset.list_files("./quant-images/*.jpg")
HEIGHT, WIDTH = 640, 640
def representative_dataset_gen():
for image_path in image_data:
img = tf.io.read_file(image_path)
img = tf.io.decode_image(img, channels=3)
img = tf.image.convert_image_dtype(img, tf.float32)
resized_img = tf.image.resize(img, (HEIGHT, WIDTH))
resized_img = resized_img[tf.newaxis, :]
yield [resized_img]
model_path = './exported-models/ptag-detector-model/ssd_mobilenet_v2_fpnlite_640x640/saved_model'
#import trained model from mobilenet 640 v2 fpn
converter = tf.lite.TFLiteConverter.from_saved_model(model_path) #using tensorflow
converter.representative_dataset = representative_dataset_gen
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8
converter.quantized_input_stats = {"normalized_input_image_tensor": (128, 128)}
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_model_quant = converter.convert()
interpreter = tf.lite.Interpreter(model_content=tflite_model_quant)
input_type = interpreter.get_input_details()[0]['dtype']
print('input: ', input_type)
output_type = interpreter.get_output_details()[0]['dtype']
print('output: ', output_type)
with open('detect_quant.tflite', 'wb') as f:
f.write(tflite_model_quant)

Hello, thanks for sharing code. I want to try but can you explain what is the "./quant-images/*.jpg" . I'm working with coco dataset. What ı will give there? Thanks.

Since, I'm using a custom image set to train the folder point to the location of a my sample image set directory. In your case you can take a sample of 100 images from coco dataset and create a directory to convert. Just make sure you're setting the right dimension of the image you used for training since I'm using mobilenet v2 640x640 I'm resizing my images to that dimension.

[canseloguzz]

For those who are still struggling to get the model converted here's my code for full quantization for edge-tpu compatibility
import numpy as np
import tensorflow as tf
#path to images
image_data = tf.data.Dataset.list_files("./quant-images/*.jpg")
HEIGHT, WIDTH = 640, 640
def representative_dataset_gen():
for image_path in image_data:
img = tf.io.read_file(image_path)
img = tf.io.decode_image(img, channels=3)
img = tf.image.convert_image_dtype(img, tf.float32)
resized_img = tf.image.resize(img, (HEIGHT, WIDTH))
resized_img = resized_img[tf.newaxis, :]
yield [resized_img]
model_path = './exported-models/ptag-detector-model/ssd_mobilenet_v2_fpnlite_640x640/saved_model'
#import trained model from mobilenet 640 v2 fpn
converter = tf.lite.TFLiteConverter.from_saved_model(model_path) #using tensorflow
converter.representative_dataset = representative_dataset_gen
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8
converter.quantized_input_stats = {"normalized_input_image_tensor": (128, 128)}
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_model_quant = converter.convert()
interpreter = tf.lite.Interpreter(model_content=tflite_model_quant)
input_type = interpreter.get_input_details()[0]['dtype']
print('input: ', input_type)
output_type = interpreter.get_output_details()[0]['dtype']
print('output: ', output_type)
with open('detect_quant.tflite', 'wb') as f:
f.write(tflite_model_quant)

Hello, thanks for sharing code. I want to try but can you explain what is the "./quant-images/*.jpg" . I'm working with coco dataset. What ı will give there? Thanks.

Since, I'm using a custom image set to train the folder point to the location of a my sample image set directory. In your case you can take a sample of 100 images from coco dataset and create a directory to convert. Just make sure you're setting the right dimension of the image you used for training since I'm using mobilenet v2 640x640 I'm resizing my images to that dimension.

Thanks. When i try as you said, i'm getting this errror.
Have you got any idea for solution?

2021-09-07 10:42:39.125057: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-09-07 10:42:39.229012: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-09-07 10:42:39.230465: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-09-07 10:42:39.233353: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2021-09-07 10:42:39.235223: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-09-07 10:42:39.236955: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-09-07 10:42:39.238333: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-09-07 10:42:40.885935: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-09-07 10:42:40.887348: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-09-07 10:42:40.888663: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-09-07 10:42:40.889893: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 6522 MB memory: -> device: 0, name: NVIDIA GeForce RTX 2060 SUPER, pci bus id: 0000:01:00.0, compute capability: 7.5 buraya kadar ok! 2021-09-07 10:42:48.441192: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:351] Ignored output_format. 2021-09-07 10:42:48.441211: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:354] Ignored drop_control_dependency. 2021-09-07 10:42:48.441215: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:360] Ignored change_concat_input_ranges. 2021-09-07 10:42:48.441809: I tensorflow/cc/saved_model/reader.cc:38] Reading SavedModel from: /home/trio/Tensorflow/models/research/object_detection/my_train/exported/saved_model 2021-09-07 10:42:48.503227: I tensorflow/cc/saved_model/reader.cc:90] Reading meta graph with tags { serve } 2021-09-07 10:42:48.503259: I tensorflow/cc/saved_model/reader.cc:132] Reading SavedModel debug info (if present) from: /home/trio/Tensorflow/models/research/object_detection/my_train/exported/saved_model 2021-09-07 10:42:48.744661: I tensorflow/cc/saved_model/loader.cc:211] Restoring SavedModel bundle. 2021-09-07 10:42:49.202077: I tensorflow/cc/saved_model/loader.cc:195] Running initialization op on SavedModel bundle at path: /home/trio/Tensorflow/models/research/object_detection/my_train/exported/saved_model 2021-09-07 10:42:49.413967: I tensorflow/cc/saved_model/loader.cc:283] SavedModel load for tags { serve }; Status: success: OK. Took 972158 microseconds. 2021-09-07 10:42:50.216415: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:210] disabling MLIR crash reproducer, set env var MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.
loc(callsite(callsite("Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/ChangeCoordinateFrame/Scale/concat@__inference_call_func_10466" at "StatefulPartitionedCall@__inference_signature_wrapper_12450") at "StatefulPartitionedCall")): error: 'tf.ConcatV2' op is neither a custom op nor a flex op
error: failed while converting: 'main':
Some ops are not supported by the native TFLite runtime, you can enable TF kernels fallback using TF Select. See instructions: https://www.tensorflow.org/lite/guide/ops_select
TF Select ops: ConcatV2
Details:
tf.ConcatV2(tensor, tensor, tensor, tensor, tensor) -> (tensor<4xf32>) : {device = ""}

Traceback (most recent call last):
File "quantize.py", line 30, in
tflite_model_quant = converter.convert()
File "/usr/local/lib/python3.8/dist-packages/tensorflow/lite/python/lite.py", line 729, in wrapper
return self._convert_and_export_metrics(convert_func, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/tensorflow/lite/python/lite.py", line 715, in _convert_and_export_metrics
result = convert_func(self, *args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/tensorflow/lite/python/lite.py", line 994, in convert
result = _convert_saved_model(**converter_kwargs)
File "/usr/local/lib/python3.8/dist-packages/tensorflow/lite/python/convert_phase.py", line 215, in wrapper
raise converter_error from None # Re-throws the exception.
File "/usr/local/lib/python3.8/dist-packages/tensorflow/lite/python/convert_phase.py", line 208, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/tensorflow/lite/python/convert.py", line 821, in convert_saved_model
data = toco_convert_protos(
File "/usr/local/lib/python3.8/dist-packages/tensorflow/lite/python/convert.py", line 313, in toco_convert_protos
raise converter_error
tensorflow.lite.python.convert_phase.ConverterError: :0: error: loc(callsite(callsite("Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/ChangeCoordinateFrame/Scale/concat@__inference_call_func_10466" at "StatefulPartitionedCall@__inference_signature_wrapper_12450") at "StatefulPartitionedCall")): 'tf.ConcatV2' op is neither a custom op nor a flex op
:0: note: loc("StatefulPartitionedCall"): called from
:0: note: loc(callsite(callsite("Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/ChangeCoordinateFrame/Scale/concat@__inference_call_func_10466" at "StatefulPartitionedCall@__inference_signature_wrapper_12450") at "StatefulPartitionedCall")): Error code: ERROR_NEEDS_FLEX_OPS
:0: error: failed while converting: 'main':
Some ops are not supported by the native TFLite runtime, you can enable TF kernels fallback using TF Select. See instructions: https://www.tensorflow.org/lite/guide/ops_select
TF Select ops: ConcatV2
Details:
tf.ConcatV2(tensor, tensor, tensor, tensor, tensor) -> (tensor<4xf32>) : {device = ""}

`

[srjoglekar246]

@canseloguzz Can you try the instructions here to convert & run your model? Please try and see if the floating point conversion works fine first, then quantized.

[sayannath]

@canseloguzz Can you try the instructions here to convert & run your model? Please try and see if the floating point conversion works fine first, then quantized.

Hello @srjoglekar246 I have used this tutorial and tried to do the inference in android device it failed. Any workaround for this. I used SSD MobileNet 320x320.

I trained the model on my custom dataset. Exported the model using exporter_main_v2.py

The command I used:

python exporter_main_v2.py --input_type image_tensor --pipeline_config_path=models/ssd_mobilenet_v2_320x320_coco17_tpu-8/pipeline.config --trained_checkpoint_dir=models/ssd_mobilenet_v2_320x320_coco17_tpu-8 --output_directory exported_models/my_model

After exporting the model, I exported the TensorFlow Inference Graph

python export_tflite_graph_tf2.py --pipeline_config_path=models/ssd_mobilenet_v2_320x320_coco17_tpu-8/pipeline.config --trained_checkpoint_dir=models/ssd_mobilenet_v2_320x320_coco17_tpu-8 --output_directory=exported_models/tflite_export/

After this I converted the model into tflite

TFLITE_MODEL_PATH = "detector.tflite"

converter = tf.lite.TFLiteConverter.from_saved_model(exported_models/tflite_export//saved_model')
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_model = converter.convert()

with open(TFLITE_MODEL_PATH, 'wb') as f:
  f.write(tflite_model)

This is the floating point TF-Lite File. It's not working even after adding the metadata. I followed the steps mentioned here

Even I used this Colab Notebook, used this model but model was crashing on my android device. I downloaded a model from TF-Hub to check my app it was working fine.

I even used Netron to visualise the tflite file. There was a difference in both of them. The major difference is in the input arrays and output arrays.

The TF-Hub model accepts input as normalized_input_image_tensor and output arrays are TFLite_Detection_PostProcess','TFLite_Detection_PostProcess:1','TFLite_Detection_PostProcess:2','TFLite_Detection_PostProcess:3

But the tflite file which is converted doest not have this I also used this command to convert my model into tflite.

tflite_convert \
--saved_model_dir=exported_models/tflite_export/saved_model \
--output_file=exported_models/tflite_export/saved_model/detect.tflite \
--input_shapes=1,320,320,3 \
--input_arrays=normalized_input_image_tensor \
--output_arrays='TFLite_Detection_PostProcess','TFLite_Detection_PostProcess:1','TFLite_Detection_PostProcess:2','TFLite_Detection_PostProcess:3' \
--inference_type=FLOAT

The tflite I got after conversion is 2KB and I have used neutron to view it does not have any layers.

[srjoglekar246]

@sayannath You don't have you use exporter_main_v2.py is you are using export_tflite_graph_tf2.py.
Also, avoid providing any output_arrays & input_arrays params to tflite_convert. The actual model may have different different names for the tensors.

[judahkshitij]

@srjoglekar246 & Others, I have been trying to run inference after converting few models from TF2 obj detection zoo to TFLite using the process described in this guide, but getting wrong results from the tflite model (I am trying basic tflite model without doing any quantization or any other optimization as a first step).

The models I have tried are:

ssd_mobilenet_v2_320x320_coco17_tpu-8
ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8
ssd_mobilenet_v1_fpn_640x640_coco17_tpu-8
ssd_resnet101_v1_fpn_640x640_coco17_tpu-8

The inference code I am using is similar to that posted by OP in this thread: #9287.

Do we still need to export checkpoint using export_tflite_graph_tf2.py first and then convert resulting saved model to tflite(to be able to leverage various levels of quantization), or can we now (in April 2022) directly convert to tflite the saved model that gets downloaded when we download a model from TF2 detection zoo? Any help is appreciated.

[judahkshitij]

@SukyoungCho Were you able to successfully convert (by successfully convert, I mean not only able to convert to tflite models but also got decent/expected results) models from TF2 object detection zoo to tflite format while taking advantage of various types quantizations? Any help is appreciated. Thanks.

[Petros626]

@srjoglekar246 finally found a way to do it? seems that only @CaptainPineapple was successful

[sayannath]

Can you share the process @Petros626 ?

[Petros626]

@sayannath I have no process to share, only asking before I do the wrong thing. I converted in TF1 my model to a TFLite model successfully, but I know TensorFlow is sometimes difficult, so I wanted to ask first, if someone converted FPNLite 320x320 or 640x640 to a TFLite Model