ssd_mobilenet_v3_large_coco(Model Zoo) does not detect the object

[NobuoTsukamoto]

System information

• What is the top-level directory of the model you are using: models/research/object_detection
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04.3 LTS (google colab)
• TensorFlow installed from (source or binary): binary
• TensorFlow version (use command below): 1.15.0 (or 1.14.0)
• Bazel version (if compiling from source): -
• CUDA/cuDNN version: 10.1 / 7.6.3
• GPU model and memory:Tesla K80 / 11441MiB
• Exact command to reproduce:

Describe the problem

I exported a freeze graph from the ssd_mobilenet_v3_large_coco checkpoint in the Tensorflow detection model zoo, but no objects were detected.
There is no problem with ssd_mobilenet_v3_small_coco in the same procedure.

The following model zoo checkpoints were used.

• ssd_mobilenet_v3_large_coco
• ssd_mobilenet_v3_small_coco

Perhaps the frozen_inference_graph.pb file in the Model Zoo was exported for TensorFlow Lite. For this reason, I exported the frozen_inference_graph.pb file using the procedure "Exporting a trained model for inference".

Perform inference using the exported rozen_inference_graph.pb file. However, ssd_mobilenet_v3_large_coco does not detect objects. ssd_mobilenet_v3_small_coco runs well and detects objects. Also, retraining ssd_mobilenet_v3_large is detect the object.

Also, I created a TFLite Model according to MediaPipe's TensorFlow / TFLite Object Detection Model and tried Android Object Detection (CPU) demo, but ssd_mobilenet_v3_large_coco does not detect objects. For ssd_mobilenet_v3_small_coco is detect the object.

Source code / logs

I created a notebook to run on Google Colab.
https://gist.github.com/NobuoTsukamoto/ac670e1103d58ef77d5f5db284bf43b7
The original code is used from the Object Detection API Demo notebook which is not up to date.

[jkjung-avt]

I concur that I see a similar problem. I'm testing with the latest 'tensorflow/models' code and TensorFlow 1.12.2. I also export frozen inference graphs of ssd_mobilenet_v3_large_coco and ssd_mobilenet_v3_small_coco by myself. Test results are:

• ssd_mobilenet_v3_large_coco: cannot detect anything
• ssd_mobilenet_v3_small_coco: can detect dogs in the test images but with significantly more false positives comparing to ssd_mobilenet_v1_coco and ssd_mobilenet_v2_coco

[yanxiangyi]

Same. ssd_mobilenet_v3_large_coco cannot detect anything.

[srjoglekar246]

I am able to reproduce the (bad) results from that model :-(. I have reached out to the research team internally, they are investigating...

[srjoglekar246]

Looks like there was a problem with the model that was published, it should be updated in a couple of days or so...

[srjoglekar246]

This PR should fix it, feel free to use the new links for download.

[NobuoTsukamoto]

I checked it with a new model.
ssd_mobilenet_v3_large_coco can detect objects.
Thanks!

Notebook Link

[skulhare]

@srjoglekar246 Are there any architectural changes in the new model files (SSD MobileNet V3 files updated in the fix)? I converted the trained SSD MobileNet V3 Large from checkpoint to .pb and from checkpoints to .tflite too. I am getting detection boxes with the .pb format but .tflite format is detecting the random objects with very low confidence (<.15).
It looks like there is some issue with the tflite conversion process (checkpoints to tflite) for SSD MobileNet V3 architecture. Thanks in advance.

[srjoglekar246]

Problems with architecture should ideally be caught during conversion. Did you use ssd_mobilenet_v3_large_coco or the small version? Can you paste your conversion command?

[skulhare]

Yes, I am using SSD MobileNet V3 Large. I am using this config: https://github.com/tensorflow/models/blob/master/research/object_detection/samples/configs/ssdlite_mobilenet_v3_large_320x320_coco.config

These are the conversion commands and the logs :

python3 object_detection/export_tflite_ssd_graph.py --pipeline_config_path=/con_data/skulhare/tf_lite_work/ssdlite_mobilenetv3/checkpoints/pipeline.config --trained_checkpoint_prefix=/con_data/skulhare/tf_lite_work/ssdlite_mobilenetv3/checkpoints/model.ckpt-53802 --output_directory=/con_data/skulhare/tf_lite_work/ssdlite_mobilenetv3/frozen_model/

Logs :
2020-02-11 12:13:57.251079: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
WARNING:tensorflow:From /workspace/models/research/slim/nets/inception_resnet_v2.py:374: The name tf.GraphKeys is deprecated. Please use tf.compat.v1.GraphKeys instead.

WARNING:tensorflow:From /workspace/models/research/slim/nets/mobilenet/mobilenet.py:397: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

WARNING:tensorflow:From object_detection/export_tflite_ssd_graph.py:145: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.

WARNING:tensorflow:From object_detection/export_tflite_ssd_graph.py:135: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.

W0211 12:14:05.292659 140575287248704 deprecation_wrapper.py:119] From object_detection/export_tflite_ssd_graph.py:135: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.

WARNING:tensorflow:From /workspace/models/research/object_detection/export_tflite_ssd_graph_lib.py:193: The name tf.gfile.MakeDirs is deprecated. Please use tf.io.gfile.makedirs instead.

W0211 12:14:05.313710 140575287248704 deprecation_wrapper.py:119] From /workspace/models/research/object_detection/export_tflite_ssd_graph_lib.py:193: The name tf.gfile.MakeDirs is deprecated. Please use tf.io.gfile.makedirs instead.

WARNING:tensorflow:From /workspace/models/research/object_detection/export_tflite_ssd_graph_lib.py:237: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

W0211 12:14:05.314558 140575287248704 deprecation_wrapper.py:119] From /workspace/models/research/object_detection/export_tflite_ssd_graph_lib.py:237: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From /workspace/models/research/object_detection/meta_architectures/ssd_meta_arch.py:597: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0211 12:14:05.318428 140575287248704 deprecation_wrapper.py:119] From /workspace/models/research/object_detection/meta_architectures/ssd_meta_arch.py:597: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

WARNING:tensorflow:From /workspace/models/research/object_detection/core/anchor_generator.py:171: The name tf.assert_equal is deprecated. Please use tf.compat.v1.assert_equal instead.

W0211 12:14:16.637240 140575287248704 deprecation_wrapper.py:119] From /workspace/models/research/object_detection/core/anchor_generator.py:171: The name tf.assert_equal is deprecated. Please use tf.compat.v1.assert_equal instead.

WARNING:tensorflow:From /workspace/models/research/object_detection/predictors/convolutional_box_predictor.py:150: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.

W0211 12:14:16.660281 140575287248704 deprecation_wrapper.py:119] From /workspace/models/research/object_detection/predictors/convolutional_box_predictor.py:150: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.

INFO:tensorflow:depth of additional conv before box predictor: 0
I0211 12:14:16.660532 140575287248704 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0211 12:14:17.031816 140575287248704 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0211 12:14:17.520579 140575287248704 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0211 12:14:17.949950 140575287248704 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0211 12:14:18.142766 140575287248704 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
I0211 12:14:18.548604 140575287248704 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0
WARNING:tensorflow:From /workspace/models/research/object_detection/export_tflite_ssd_graph_lib.py:52: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

W0211 12:14:19.119874 140575287248704 deprecation_wrapper.py:119] From /workspace/models/research/object_detection/export_tflite_ssd_graph_lib.py:52: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2020-02-11 12:14:19.121490: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2020-02-11 12:14:19.213245: E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2020-02-11 12:14:19.213356: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: 3780a7911d90
2020-02-11 12:14:19.213369: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: 3780a7911d90
2020-02-11 12:14:19.213570: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 418.87.0
2020-02-11 12:14:19.213627: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 384.111.0
2020-02-11 12:14:19.213641: E tensorflow/stream_executor/cuda/cuda_diagnostics.cc:313] kernel version 384.111.0 does not match DSO version 418.87.0 -- cannot find working devices in this configuration
2020-02-11 12:14:19.268446: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2198685000 Hz
2020-02-11 12:14:19.269632: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x61e9c10 executing computations on platform Host. Devices:
2020-02-11 12:14:19.269664: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): ,
2020-02-11 12:14:19.707214: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set. If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU. To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
WARNING:tensorflow:From /workspace/models/research/object_detection/export_tflite_ssd_graph_lib.py:267: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

W0211 12:14:19.729221 140575287248704 deprecation_wrapper.py:119] From /workspace/models/research/object_detection/export_tflite_ssd_graph_lib.py:267: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.

WARNING:tensorflow:From /workspace/models/research/object_detection/export_tflite_ssd_graph_lib.py:292: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

W0211 12:14:19.734752 140575287248704 deprecation_wrapper.py:119] From /workspace/models/research/object_detection/export_tflite_ssd_graph_lib.py:292: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/tools/freeze_graph.py:127: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
W0211 12:14:20.823790 140575287248704 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/tools/freeze_graph.py:127: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
INFO:tensorflow:Restoring parameters from /con_data/skulhare/tf_lite_work/ssdlite_mobilenetv3/checkpoints/model.ckpt-53802
I0211 12:14:23.312593 140575287248704 saver.py:1280] Restoring parameters from /con_data/skulhare/tf_lite_work/ssdlite_mobilenetv3/checkpoints/model.ckpt-53802
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/tools/freeze_graph.py:233: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.convert_variables_to_constants
W0211 12:14:30.244218 140575287248704 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/tools/freeze_graph.py:233: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.convert_variables_to_constants
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/graph_util_impl.py:270: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.extract_sub_graph
W0211 12:14:30.244596 140575287248704 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/graph_util_impl.py:270: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.extract_sub_graph
INFO:tensorflow:Froze 406 variables.
I0211 12:14:31.269247 140575287248704 graph_util_impl.py:311] Froze 406 variables.
INFO:tensorflow:Converted 406 variables to const ops.
I0211 12:14:31.372385 140575287248704 graph_util_impl.py:364] Converted 406 variables to const ops.
2020-02-11 12:14:31.699083: I tensorflow/tools/graph_transforms/transform_graph.cc:317] Applying strip_unused_nodes

This gives me two files - tflite_graph.pb and tflite_graph.pbxt
Further, I am using this Python code to convert tflite_graph.pb to ssd_mobilenet_v3_large.tflite

# works with TF 1.14.0 container without GPU version of TF...  
import os 
import ipdb
import tensorflow as tf 

os.environ["CUDA_VISIBLE_DEVICES"]="-1"

frozen_model_path = "/con_data/skulhare/tf_lite_work/ssdlite_mobilenetv3/frozen_model/tflite_graph.pb" 
input_arrays = ["normalized_input_image_tensor"]
output_node_names = ['TFLite_Detection_PostProcess','TFLite_Detection_PostProcess:1','TFLite_Detection_PostProcess:2','TFLite_Detection_PostProcess:3']
input_shapes = {"normalized_input_image_tensor":[1,640,640,3]}


converter = tf.lite.TFLiteConverter.from_frozen_graph(frozen_model_path,input_arrays,output_node_names,input_shapes=input_shapes) 

converter.inference_type = tf.float32 

converter.allow_custom_ops = True


tflite_model = converter.convert()
open("/con_data/skulhare/tf_lite_work/ssdlite_mobilenetv3/frozen_model/ssdlite_mobilenet_v3_large.tflite", "wb").write(tflite_model)

Any help appreciated.
Thanks in advance.

[NobuoTsukamoto]

@skulhare
I convert to TF-Lite model by the following steps, and can detect objects. I use tflite_convert command. ssdlite mobilenet v3 large / small can detect objects.

## Frozen graph
$ INPUT_TYPE=image_tensor
$ PIPELINE_CONFIG_PATH= ...
$ TRAINED_CKPT_PREFIX= ...
$ EXPORT_DIR=...
$ python object_detection/export_inference_graph.py \
    --input_type=${INPUT_TYPE} \
    --pipeline_config_path=${PIPELINE_CONFIG_PATH} \
    --trained_checkpoint_prefix=${TRAINED_CKPT_PREFIX} \
    --output_directory=${EXPORT_DIR}

# Frozen graph(TF-Lite)
$ CONFIG_FILE= ...
$ CHECKPOINT_PATH= ...
$ OUTPUT_DIR= ...
$ python object_detection/export_tflite_ssd_graph.py \
    --pipeline_config_path=$CONFIG_FILE \
    --trained_checkpoint_prefix=$CHECKPOINT_PATH \
    --output_directory=$OUTPUT_DIR \
    --add_postprocessing_op=true

# TF-Lite convert
$ tflite_convert \
   --graph_def_file=$OUTPUT_DIR/tflite_graph.pb \
   --output_file=$OUTPUT_DIR/detect.tflite \
   --input_shapes=1,320,320,3 \
   --input_arrays=normalized_input_image_tensor \
   --output_arrays='TFLite_Detection_PostProcess','TFLite_Detection_PostProcess:1','TFLite_Detection_PostProcess:2','TFLite_Detection_PostProcess:3' \
   --inference_type=FLOAT \
   --inference_input_type=QUANTIZED_UINT8 \
   --allow_custom_ops \
   --mean_values=128 \
   --std_dev_values=128 \
   --change_concat_input_ranges=false

[yghstill]

@srjoglekar246 In your PR#8057, is the model mAP of ssd_mobilenet_v3 quantized or before the quantization?

[srjoglekar246]

I believe the model is floating-point, if I am not wrong.

[yghstill]

@srjoglekar246 When using the model you provided, I execute an evaluation of object_detection/model_main.py，Why the mAP result i get is 26.4%(large) and 16.9%(small)? The result you provided is 22.6%|15.4%？
Looking forward to your reply.

[gmt710]

The tensorflow-v1.12.0 is okay.

[srjoglekar246]

@still-wait Sorry this slipped through the cracks. Are you using the 2014 minival dataset to evaluate the images? That is probably the dataset that was used to evaluate the model, so if you used something different the values will change.