Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DeepLab with TensorFlow Mobile or TensorFlow Lite #4278

Open
dailystudio opened this issue May 16, 2018 · 19 comments

Comments

@dailystudio
Copy link

commented May 16, 2018

Hello @aquariusjay,

We just want to run this modal on Android. We have tried two approach TensorFlow Mobile and TensorFlow Lite.

With TensorFlow Mobile, we download the pre-trained modals with MobileNetV2:
mobilenetv2_coco_voc_trainaug
mobilenetv2_coco_voc_trainval
mobilenetv2_coco_cityscapes_trainfine

We can successfully load the modal, but when run the inference, we get the following error:

05-16 16:06:34.122 611-635/? E/AndroidRuntime: FATAL EXCEPTION: AsyncTask #1
    Process: com.orange.labs.colorme.dev, PID: 611
    java.lang.RuntimeException: An error occurred while executing doInBackground()
        at android.os.AsyncTask$3.done(AsyncTask.java:325)
        at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:354)
        at java.util.concurrent.FutureTask.setException(FutureTask.java:223)
        at java.util.concurrent.FutureTask.run(FutureTask.java:242)
        at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:243)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
        at java.lang.Thread.run(Thread.java:761)
     Caused by: java.lang.IllegalArgumentException: No OpKernel was registered to support Op 'Slice' with these attrs.  Registered devices: [CPU], Registered kernels:
      device='CPU'; T in [DT_BOOL]
      device='CPU'; T in [DT_FLOAT]
      device='CPU'; T in [DT_INT32]
    
    	 [[Node: SemanticPredictions = Slice[Index=DT_INT32, T=DT_INT64](ArgMax, SemanticPredictions/begin, SemanticPredictions/size)]]
        at org.tensorflow.Session.run(Native Method)
        at org.tensorflow.Session.access$100(Session.java:48)
        at org.tensorflow.Session$Runner.runHelper(Session.java:298)
        at org.tensorflow.Session$Runner.runAndFetchMetadata(Session.java:260)
        at org.tensorflow.contrib.android.TensorFlowInferenceInterface.run(TensorFlowInferenceInterface.java:220)
        at org.tensorflow.contrib.android.TensorFlowInferenceInterface.run(TensorFlowInferenceInterface.java:197)
        at com.dailystudio.deeplab.DeeplabV3.segment(DeeplabV3.java:104)
        at com.dailystudio.deeplab.DeeplabApplication$1.doInBackground(DeeplabApplication.java:46)
        at com.dailystudio.deeplab.DeeplabApplication$1.doInBackground(DeeplabApplication.java:22)
        at android.os.AsyncTask$2.call(AsyncTask.java:305)
        at java.util.concurrent.FutureTask.run(FutureTask.java:237)

I think this is caused by the output node "SemanticPredictions" call the operation Slice with INT64 data. This is not supported by TensorFlow Mobile yet.

With TensorFlow Lite, we use the following command to convert is to tflite format:

toco \
	--input_file=$(pwd)/model/frozen_inference_graph.pb \
	--input_format=TENSORFLOW_GRAPHDEF \
	--output_format=TFLITE \
	--output_file=$(pwd)/model/deeplabv3_mnv2_pascal_trainval.tflite \
	--inference_type=FLOAT \
	--input_type=QUANTIZED_UINT8 \
	--input_arrays=ImageTensor \
	--output_arrays=SemanticPredictions \
	--input_shapes=1,513,513,3 \
	--default_ranges_min=0 --default_ranges_max=255

We get the following warnings:

2018-05-16 16:19:29.429205: W tensorflow/contrib/lite/toco/toco_cmdline_flags.cc:245] --input_type is deprecated. It was an ambiguous flag that set both --input_data_types and --inference_input_type. If you are trying to complement the input file with information about the type of input arrays, use --input_data_type. If you are trying to control the quantization/dequantization of real-numbers input arrays in the output file, use --inference_input_type.
2018-05-16 16:19:29.492371: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1270] Converting unsupported operation: Equal
2018-05-16 16:19:29.492813: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1270] Converting unsupported operation: LogicalAnd
2018-05-16 16:19:29.521856: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] Before Removing unused ops: 812 operators, 1241 arrays (0 quantized)
2018-05-16 16:19:29.550535: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] After Removing unused ops pass 1: 802 operators, 1222 arrays (0 quantized)
2018-05-16 16:19:29.582482: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] Before general graph transformations: 802 operators, 1222 arrays (0 quantized)
2018-05-16 16:19:29.610492: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] After general graph transformations pass 1: 148 operators, 358 arrays (0 quantized)
2018-05-16 16:19:29.613697: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] After general graph transformations pass 2: 148 operators, 358 arrays (0 quantized)
2018-05-16 16:19:29.616876: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] After general graph transformations pass 3: 143 operators, 348 arrays (0 quantized)
2018-05-16 16:19:29.619981: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] After general graph transformations pass 4: 142 operators, 346 arrays (0 quantized)
2018-05-16 16:19:29.622935: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] After general graph transformations pass 5: 141 operators, 344 arrays (0 quantized)
2018-05-16 16:19:29.626348: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] Before dequantization graph transformations: 141 operators, 344 arrays (0 quantized)
2018-05-16 16:19:29.629622: I tensorflow/contrib/lite/toco/allocate_transient_arrays.cc:329] Total transient array allocated size: 3158144 bytes, theoretical optimal value: 3158144 bytes.
2018-05-16 16:19:29.631849: F tensorflow/contrib/lite/toco/tflite/export.cc:315] Some of the operators in the model are not supported by the standard TensorFlow Lite runtime. If you have a custom implementation for them you can disable this error with --allow_custom_ops. Here is a list of operators for which you will need custom implementations: ExpandDims, Slice, Stack, TensorFlowShape.
./convert-lite.sh: line 11: 70835 Abort trap: 6           /Volumes/Workspace/tensorflow/workspace/tensorflow/bazel-bin/tensorflow/contrib/lite/toco/toco --input_file=$(pwd)/model/frozen_inference_graph.pb --input_format=TENSORFLOW_GRAPHDEF --output_format=TFLITE --output_file=$(pwd)/model/deeplabv3_mnv2_pascal_trainval.tflite --inference_type=FLOAT --input_type=QUANTIZED_UINT8 --input_arrays=ImageTensor --output_arrays=SemanticPredictions --input_shapes=1,513,513,3 --default_ranges_min=0 --default_ranges_max=255

The model could not be loaded successfully. I think it is caused the warning:
Here is a list of operators for which you will need custom implementations: ExpandDims, Slice, Stack, TensorFlowShape.

Is it possible to update node SemanticPredictions to use INT32 data type on Slice operation? Or do you have any suggestion on how to run it with TensorFlow lite?

@aquariusjay

This comment has been minimized.

Copy link
Contributor

commented May 16, 2018

Hi dailystudio,

Thanks for bringing up this issue.
Regarding your questions, I would have a few suggestions at this moment:

  1. You could try to change the export_model.py to use int32 SemanticPredictions, and see what happens (e.g., there may be other issues).
  2. You could contact TensorFlow Mobile and TensorFlow Lite to support those operations.

We will look into this issue more carefully in the near future.

Thanks,

@dailystudio

This comment has been minimized.

Copy link
Author

commented May 16, 2018

Hello aquariusjay,
Thanks for your valuable advices!

I modify the line in export_model.py (around line 131):

    semantic_predictions = tf.slice(
        tf.cast(predictions[common.OUTPUT_TYPE], tf.int32),
        [0, 0, 0],
        [1, resized_image_size[0], resized_image_size[1]])

add tf.cast to cast predictions to int32. And then export model with the following command:

python export_model.py --checkpoint_path model/model.ckpt-30000 --export_path model/frozen_inference_graph.pb

Here I only pointed the checkpoint path and export path, rest parameters kept with default value.

Now running the model with Tensorflow Mobile is successful!

The new issue is that the output array (SemanticPredictions) is a zero array, all the elements are 0.
There is no any error or warning are printed during the inference.

Do you have any suggestion? Is it caused by some unsupported op in the model? Or should I add more parameters during exporting the model?

@aquariusjay

This comment has been minimized.

Copy link
Contributor

commented May 16, 2018

You need to make sure you have provided the right flag values for the model variant (e.g., MobileNet-v2 or Xception_65) that you are using.
Check local_test.sh or local_test_mobilenetv2.sh for reference.

@dailystudio

This comment has been minimized.

Copy link
Author

commented May 17, 2018

Hello aquariusjay,

Now it works on my mobile devices. I add model variant parameter during export model and also fix a dimension issue of passing the width and heigh in wrong order during the inference.

Thanks for your help. I will accomplish it as complete demo.

@liangxiao05

This comment has been minimized.

Copy link
Contributor

commented May 17, 2018

@dailystudio,hi,could you share which mobile do you run it on ,and how is the latency?

@dailystudio

This comment has been minimized.

Copy link
Author

commented May 17, 2018

@liangxiao05, at Oneplus 3T and Oneplus 5, 900ms ~ 2700ms per inference.

@Shenghsin

This comment has been minimized.

Copy link

commented May 17, 2018

@dailystudio could you share your demo?

@JoseRNFaria

This comment has been minimized.

Copy link

commented May 18, 2018

@dailystudio
I tried your solution and it worked. Thanks.
In my pixel 2 the run time is ~1300 - 1500 ms if you reduce the input images sizes to 256x256 the time goes to ~400 - 500 ms.

I'm trying to pass the model to tflite but a error is making it impossible.

@dailystudio

This comment has been minimized.

Copy link
Author

commented May 18, 2018

@JoseRNFaria Glad to here that my works can help you!

@Shenghsin

This comment has been minimized.

Copy link

commented May 21, 2018

@dailystudio I have the same problem "The new issue is that the output array (SemanticPredictions) is a zero array, all the elements are 0." Could you tell me how to fix it?? Thanks!

@dailystudio

This comment has been minimized.

Copy link
Author

commented May 25, 2018

@Shenghsin @aquariusjay @JoseRNFaria , I have written a demo app of this model. I am still updating the document. Here is the repository link:
https://github.com/dailystudio/ml/tree/master/deeplab

@llhe

This comment has been minimized.

Copy link

commented Jul 17, 2018

@dailystudio To run deeplabv3+ on mobile phones faster, you can checkout MACE which has deeplabv3+ in the MACE Model Zoo. Here are some benchmark results which includes deeplab-v3-plus-mobilenet-v2.

@austingg

This comment has been minimized.

Copy link

commented Jul 26, 2018

hope we can use tensorflowlite to run the deeplab v3+ model, since segmentation task ususally cost much more than classification task. quantization may help a lot.

@weehe91325

This comment has been minimized.

Copy link

commented Aug 21, 2018

@dailystudio, did you managed to use in your demo the model converted to TF Lite?
Does anyone know if this is possible?

@sumsuddin

This comment has been minimized.

Copy link
Contributor

commented Sep 5, 2018

Hi, @dailystudio. Just change this line to predictions[output] = tf.argmax(logits, 3, output_type=dtypes.int32).

@SanthoshRajendiran

This comment has been minimized.

Copy link

commented Nov 15, 2018

@dailystudio @aquariusjay @sumsuddin.. Is there support for TFLite conversion of DeeplabV3 with MobilenetV2? Could not find relevant documentation.

@melody-rain

This comment has been minimized.

Copy link

commented Nov 29, 2018

watching this

@essalahsouad

This comment has been minimized.

Copy link

commented Aug 2, 2019

@dailystudio

Hello aquariusjay,
Now it works on my mobile devices. I add model variant parameter during export model and also fix a dimension issue of passing the width and heigh in wrong order during the inference.
Thanks for your help. I will accomplish it as complete demo.

how do you solve the issue, because have tried to convert the pb file to tflite by using tf converter
tflite_convert ----output_format=TFLITE --inference_type=FLOAT --inference_input_type=FlOAT --input_arrays=sub_2 --input_shapes=1,257,257,3 --output_arrays=ResizeBilinear_2 --output_file=mobilenet.tflite --graph_def=mobilenet.pb --mean_values=128 --std_dev_values=127 --allow_custom_ops --post_training_quantize
and i got a slow inference on mobile(using iphone 6)
can check this for me ??

@helloItzMe

This comment has been minimized.

Copy link

commented Sep 24, 2019

can anyone create a py script file for converting xception model frozen_graph.pb file to tflite??

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.