Efficientnet B7 classification conversion from tf to tflite fails tflite imagenet evaluation test

[suyash-narain]

System information

• Have I written custom code (as opposed to using a stock example script
  provided in TensorFlow): No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 20.04
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue
  happens on a mobile device: No
• TensorFlow installed from (source or binary): Binary
• TensorFlow version (use command below): 2.11
• Python version: 3.9

Describe the problem

I am trying to convert efficientnet_b7_classification model available in tfhub: https://storage.googleapis.com/tfhub-modules/tensorflow/efficientnet/b7/classification/1.tar.gz to tflite

I use the below code snippet to convert the saved model to tflite:

import tensorflow as tf
converter = tf.lite.TFLiteConverter.from_saved_model('saved_model')
tflite_model = converter.convert()
with open('model.tflite', 'wb') as f:
f.write(tflite_model)

on visualizing the model on netron, i saw that the input to the model is of shape (1,1,1,3). whereas, the input to efficientnet_b7 is 600x600. So the converted model should have the shape (1,600,600,3), but i don't see this.
The output of the model is of the shape (1,1000).

I try to change the input shape using the below snippet:
model = tf.saved_model.load('saved_model')
concrete_func = model.signatures["serving_default"]
concrete_func.inputs[0].set_shape([1,600,600,3])
converter = tf.lite.TFLiteConverter.from_concrete_functions([concrete_func],model)
tflite_model = converter.conver

But on visualizing the model, i still see the input shape as (1,1,1,3), and output shape is (1,1000). The model has been trained on imagenet so output shape should be fine with the label file consisting of 1000 labels starting from 1-1000 instead of 0-1000 which includes the dummy as well.

so from my labelfile, i removed the dummy, and it then used the same labelfile to evaluate the tflite model using the imagenet_image_classification run_eval binary. On running it against the converted tflite model, i get the below error log, which seemingly says that the model output shape is wrong. What is the correct way to go about converting and testing efficientnet_b7 model?
The output shape of mobilenets is (1,1001), whereas that of efficientnets is (1,1000) even though both are trained on imagenet dataset. why is that so?

thanks!

Source code / logs

$ bazel run -c opt -- //tensorflow/lite/tools/evaluation/tasks/imagenet_image_classification:run_eval --model_file=/home/mtk/Downloads/efficientnet_b7_float_model.tflite --ground_truth_images_path=/home/mtk/Downloads/ILSVRC2012_img_val --ground_truth_labels=/home/mtk/Documents/val.txt --model_output_labels=/home/mtk/Documents/tflite_models/imagenet_classes.txt --output_file_path=tmp/accuracy_output.txt --num_images=0
INFO: Options provided by the client:
Inherited 'common' options: --isatty=1 --terminal_columns=79
INFO: Reading rc options for 'run' from /home/mtk/Documents/tensorflow/.bazelrc:
Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'run' from /home/mtk/Documents/tensorflow/.bazelrc:
Inherited 'build' options: --define framework_shared_object=true --define tsl_protobuf_header_only=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true --experimental_cc_shared_library --experimental_link_static_libraries_once=false --incompatible_enforce_config_setting_visibility --deleted_packages=tensorflow/compiler/mlir/tfrt,tensorflow/compiler/mlir/tfrt/benchmarks,tensorflow/compiler/mlir/tfrt/jit/python_binding,tensorflow/compiler/mlir/tfrt/jit/transforms,tensorflow/compiler/mlir/tfrt/python_tests,tensorflow/compiler/mlir/tfrt/tests,tensorflow/compiler/mlir/tfrt/tests/ir,tensorflow/compiler/mlir/tfrt/tests/analysis,tensorflow/compiler/mlir/tfrt/tests/jit,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_tfrt,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_jitrt,tensorflow/compiler/mlir/tfrt/tests/tf_to_corert,tensorflow/compiler/mlir/tfrt/tests/tf_to_tfrt_data,tensorflow/compiler/mlir/tfrt/tests/saved_model,tensorflow/compiler/mlir/tfrt/transforms/lhlo_gpu_to_tfrt_gpu,tensorflow/core/runtime_fallback,tensorflow/core/runtime_fallback/conversion,tensorflow/core/runtime_fallback/kernel,tensorflow/core/runtime_fallback/opdefs,tensorflow/core/runtime_fallback/runtime,tensorflow/core/runtime_fallback/util,tensorflow/core/tfrt/eager,tensorflow/core/tfrt/eager/backends/cpu,tensorflow/core/tfrt/eager/backends/gpu,tensorflow/core/tfrt/eager/core_runtime,tensorflow/core/tfrt/eager/cpp_tests/core_runtime,tensorflow/core/tfrt/gpu,tensorflow/core/tfrt/run_handler_thread_pool,tensorflow/core/tfrt/runtime,tensorflow/core/tfrt/saved_model,tensorflow/core/tfrt/graph_executor,tensorflow/core/tfrt/saved_model/tests,tensorflow/core/tfrt/tpu,tensorflow/core/tfrt/utils
INFO: Found applicable config definition build:short_logs in file /home/mtk/Documents/tensorflow/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:v2 in file /home/mtk/Documents/tensorflow/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
INFO: Found applicable config definition build:linux in file /home/mtk/Documents/tensorflow/.bazelrc: --host_copt=-w --copt=-Wno-all --copt=-Wno-extra --copt=-Wno-deprecated --copt=-Wno-deprecated-declarations --copt=-Wno-ignored-attributes --copt=-Wno-array-bounds --copt=-Wunused-result --copt=-Werror=unused-result --copt=-Wswitch --copt=-Werror=switch --copt=-Wno-error=unused-but-set-variable --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++17 --host_cxxopt=-std=c++17 --config=dynamic_kernels --experimental_guard_against_concurrent_changes
INFO: Found applicable config definition build:dynamic_kernels in file /home/mtk/Documents/tensorflow/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS
INFO: Analyzed target //tensorflow/lite/tools/evaluation/tasks/imagenet_image_classification:run_eval (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //tensorflow/lite/tools/evaluation/tasks/imagenet_image_classification:run_eval up-to-date:
bazel-bin/tensorflow/lite/tools/evaluation/tasks/imagenet_image_classification/run_eval
INFO: Elapsed time: 0.091s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/tensorflow/lite/tools/evaluation/tasks/imagenet_image_classification/run_eval '--model_file=/home/mtk/Downloads/efficientnet_b7_float_model.tflite' '--ground_truth_images_path=/home/mtk/Downloads/ILSVRC2012_img_val' '--ground_truth_labels=/home/mtk/Documents/val.txt' '--model_output_labels=/home/mtk/Documents/tflite_models/imagenet_classes.txt' '--output_INFO: Build completed successfully, 1 total action
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
INFO: Evaluated: 0%
2023-03-20 18:49:35.911460: E tensorflow/lite/tools/evaluation/stages/topk_accuracy_eval_stage.cc:80] model_output_ not set correctly
ERROR: Could not run the task evaluation!

[tiruk007]

@suyash-narain
Could you please try to convert the model by using hub.load() or hub.KerasLayer() as per Doc1 & Doc2 then try to evaluate tf-lite model. Please find the gist here for reference and let us know if still issue persists.

Thank you !

[suyash-narain]

@suyash-narain Could you please try to convert the model by using hub.load() or hub.KerasLayer() as per Doc1 & Doc2 then try to evaluate tf-lite model. Please find the gist here for reference and let us know if still issue persists.

Thank you !

Hi, i tried to convert the model using hub.load() or hub.KerasLayer() as specified in the colab gist sent by you. I still get the same error.

if i use the below method:

import tensorflow as tf
import tensorflow_hub as hub

m = tf.keras.Sequential([hub.KerasLayer("https://tfhub.dev/tensorflow/efficientnet/b7/classification/1")])
m.build([None, 600, 600, 3])# Batch input shape.

m.save('model')

converter = tf.lite.TFLiteConverter.from_saved_model("/content/model")
converter.target_spec.supported_ops = [
tf.lite.OpsSet.TFLITE_BUILTINS, # enable TensorFlow Lite ops.
tf.lite.OpsSet.SELECT_TF_OPS, # enable TensorFlow ops.
]

tflite_file = "effnetB7_model.tflite"
with open(tflite_file, 'wb') as f:
f.write(converter.convert())

the input shape becomes (1,600,600,3) as desired, and has the output shape (1,1000). The official ImageNetLabels.txt available at https://storage.googleapis.com/download.tensorflow.org/data/ImageNetLabels.txt has total of 1001 labels ranging from (0-1000). So in that way, the output shape for this model should be (1,1001). When I run it against tflite imagenet evaluation tool, i get the same error 'model_output_ not set correctly'. If i remove 0-background label from label file to bring total labels to 1000, i still get the same error.

I get similar error results if i use hub.load()

[tiruk007]

@suyash-narain
Could you please provide detailed steps of tflite imagenet evaluation test to replicate the issue reported here ?

Thank you!

[suyash-narain]

@tiruk007 Please find the detailed steps I exercised towards building the provided tflite imagenet evaluation tool:

1. clone tensorflow from github:

$ git clone "https://github.com/tensorflow/tensorflow.git"
$ cd tensorflow

2. Download ILSVRC validation dataset consisting of 50k images from http://image-net.org/request: ILSVRC2012_img_val https://image-net.org/data/ILSVRC/2012/ILSVRC2012_img_val.tar

3. Download ILSVRC 2012 dev kit: ILSVRC2012_devkit_t12 (tasks1&2) https://image-net.org/data/ILSVRC/2012/ILSVRC2012_devkit_t12.tar.gz

4. Generate Ground Truth Labels:

python /tensorflow/lite/tools/evaluation/tasks/imagenet_image_classification/generate_validation_labels.py \
--ilsvrc_devkit_dir=path/to/ILSVRC2012_devkit_t12 \
--validation_labels_output=output_labels.txt

5. Download imagenet labels from https://storage.googleapis.com/download.tensorflow.org/data/ImageNetLabels.txt

6. Build and run on Ubuntu desktop:

bazel run -c opt
--
//tensorflow/lite/tools/evaluation/tasks/imagenet_image_classification:run_eval
--model_file=mbnv2_model_test1.tflite
--ground_truth_images_path=path/to/ILSVRC2012_img_val
--ground_truth_labels=path/to/output_labels.txt
--model_output_labels=path/to/ImageNetLabels.txt
--output_file_path=accuracy_output.txt
--num_images=0 # Run on all images.

[pjpratik]

Hi @suyash-narain

Sorry for the delayed response.

As the tool for evaluation is based on ILSVRC 2012 task, the classes with shape [1,1001] are only accepted for the evaluation. The tensorflow hub provides modules trained on ImageNet (ILSVRC-2012-CLS) which return the output shapes as [1,1001] and can be evaluated using the tool.

Please find the list of models here.

Thanks.

[suyash-narain]

Hi @pjpratik
Your reply confuses me. If i use a mobilenetv2 tflite model created using keras application layers as below:

model = tf.keras.applications.mobilenet_v2.MobileNetV2(
input_shape=None,
alpha=1.0,
include_top=True,
weights='imagenet',
input_tensor=None,
pooling='avg',
classes=1000,
classifier_activation='softmax'
)
converter = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_file = "mobilenet_v2_keras_model.tflite"
with open(tflite_file, 'wb') as f:
f.write(converter.convert())

i get output model shape as [1.1000]. I run this model against imagenet labels having 1000 classes from 1-1000 (removing 0-background), and it works. Same way, if i run efficientnetlite4 downloaded from https://tfhub.dev/tensorflow/efficientnet/lite4/classification/2, and run it against imagenet labels having 1000 classes from 1-1000 after removing 0-background class, it still works.
Its just the efficientnet B7 downloaded from tfhub and converted to tflite that fails. even the classification accuracy on individual images is off the charts.
So what changes can i make to execute efficientnet B7 perfectly against imagenet evaluation tool, or increase accuracy of tflite model. I think this model was also trained against imagenet dataset.

thanks

Hi @suyash-narain

Sorry for the delayed response.

As the tool for evaluation is based on ILSVRC 2012 task, the classes with shape [1,1001] are only accepted for the evaluation. The tensorflow hub provides modules trained on ImageNet (ILSVRC-2012-CLS) which return the output shapes as [1,1001] and can be evaluated using the tool.

Please find the list of models here.

Thanks.

[pjpratik]

Hi @suyash-narain Thanks for the clarification.

Sorry for the delayed response. I was able to reproduce this issue. Please find the screenshot here.

@sachinprasadhs Could you please look into this? Thanks.