Skip to content
This repository has been archived by the owner on May 12, 2024. It is now read-only.

Converted mediapipe palm detection coreml model cannot be deployed on IOS app #27

Closed
TaoZappar opened this issue Feb 16, 2022 · 7 comments
Closed

Converted mediapipe palm detection coreml model cannot be deployed on IOS app #27

TaoZappar opened this issue Feb 16, 2022 · 7 comments

Comments

@TaoZappar
Copy link

TaoZappar commented Feb 16, 2022 •
edited by PINTO0309

Issue Type

Bug

OS

Mac OS

OS architecture

x86_64

Programming Language

Python

Framework

CoreML

Download URL for tflite file

https://github.com/google/mediapipe/blob/master/mediapipe/modules/palm_detection/palm_detection_full.tflite

Convert Script

docker run -it --rm \
  -v `pwd`:/home/user/workdir \
  ghcr.io/pinto0309/tflite2tensorflow:latest

tflite2tensorflow \
  --model_path palm_detection_full.tflite \
  --flatc_path ../flatc \
  --schema_path ../schema.fbs \
  --output_pb

tflite2tensorflow \
  --model_path palm_detection_full.tflite \
  --flatc_path ../flatc \
  --schema_path ../schema.fbs \
  --output_no_quant_float32_tflite \
  --output_dynamic_range_quant_tflite \
  --output_weight_quant_tflite \
  --output_float16_quant_tflite \
  --output_integer_quant_tflite \
  --string_formulas_for_normalization 'data / 255.0' \
  --output_tfjs \
  --output_coreml \
  --output_tftrt_float32 \
  --output_tftrt_float16 \
  --output_onnx \
  --onnx_opset 13 \
  --output_openvino_and_myriad

Description

when I deployed the converted mp palm detection coreml model on IOS app, I got error like:

2022-02-15 16:31:50.577312+0000 MLModelCamera[12602:4637828] [espresso] [Espresso::handle_ex_plan] exception=Espresso exception: "Generic error": [Exception from layer 240: model_1/model/add_24/add]Espresso exception: "Invalid blob shape": elementwise_kernel_cpu: Cannot broadcast [12, 2, 1, 256, 1] and [12, 1, 1, 256, 1] status=-1

seems like the resizeBilinear op causes this problem. Is there any way to fix this issue? Thanks in advance.

The related code in tflite2tensorlfow.py is:

        elif op_type == 'RESIZE_BILINEAR':
            input_tensor = tensors[op['inputs'][0]]
            size_detail = interpreter._get_tensor_details(op['inputs'][1])
            size = interpreter.get_tensor(size_detail['index'])
            size_height = size[0]
            size_width  = size[1]

            options = op['builtin_options']
            align_corners = options['align_corners']
            half_pixel_centers = options['half_pixel_centers']

            def upsampling2d_bilinear(x, size_height, size_width, align_corners, half_pixel_centers):
                if optimizing_for_edgetpu_flg:
                    return tf.image.resize_bilinear(x, (size_height, size_width))
                else:
                    if optimizing_for_openvino_and_myriad:
                        if half_pixel_centers:
                            return tf.image.resize_bilinear(
                                x,
                                (size_height, size_width),
                                align_corners=False,
                                half_pixel_centers=half_pixel_centers
                            )
                        else:
                            return tf.image.resize_bilinear(
                                x,
                                (size_height, size_width),
                                align_corners=True,
                                half_pixel_centers=half_pixel_centers
                            )
                    else:
                        return tfv2.image.resize(
                            x,
                            [size_height, size_width],
                            method='bilinear'
                        )

            output_detail = interpreter._get_tensor_details(op['outputs'][0])
            lambda_name = get_op_name(output_detail['name']) + '_lambda'
            output_tensor_lambda = tf.keras.layers.Lambda(
                upsampling2d_bilinear,
                arguments={'size_height': size_height,
                            'size_width': size_width,
                            'align_corners': align_corners,
                            'half_pixel_centers': half_pixel_centers},
                name=lambda_name
            )(input_tensor)
            output_tensor = tf.identity(
                output_tensor_lambda,
                name=get_op_name(output_detail['name'])
            )
            tensors[output_detail['index']] = output_tensor

Relevant Log Output

2022-02-15 16:31:50.577312+0000 MLModelCamera[12602:4637828] [espresso] [Espresso::handle_ex_plan] exception=Espresso exception: "Generic error": [Exception from layer 240: model_1/model/add_24/add]Espresso exception: "Invalid blob shape": elementwise_kernel_cpu: Cannot broadcast [12, 2, 1, 256, 1] and [12, 1, 1, 256, 1] status=-1

Source code for simple inference testing code

No response
@TaoZappar TaoZappar changed the title converted mediapipe palm detection coreml model cannot deploy on IOS app converted mediapipe palm detection coreml model cannot be deployed on IOS app Feb 16, 2022
@TaoZappar TaoZappar changed the title converted mediapipe palm detection coreml model cannot be deployed on IOS app Converted mediapipe palm detection coreml model cannot be deployed on IOS app Feb 16, 2022
@PINTO0309
Copy link
Owner

Sad to say, I don't own an IOS environment, so I can't test it. I'm not entirely sure if it will work, but could you please check if the CoreML model you downloaded from the following URL works?
model_coreml_float32.mlmodel.zip

docker run -it --rm \
-v `pwd`:/home/user/workdir \
ghcr.io/pinto0309/tflite2tensorflow:latest

wget https://github.com/google/mediapipe/raw/master/mediapipe/modules/palm_detection/palm_detection_full.tflite

tflite2tensorflow \
--model_path palm_detection_full.tflite \
--flatc_path ../flatc \
--schema_path ../schema.fbs \
--output_pb

tflite2tensorflow \
--model_path palm_detection_full.tflite \
--flatc_path ../flatc \
--schema_path ../schema.fbs \
--output_coreml

@TaoZappar
Copy link
Author

TaoZappar commented Feb 16, 2022 •
edited

Thanks for the reply. Just test the model on the IOS, still get the same error.

2022-02-16 12:04:35.315685+0000 MLModelCamera[12740:4737111] [espresso] [Espresso::handle_ex_plan] exception=Espresso exception: "Generic error": [Exception from layer 240: model_1/model/add_24/add]Espresso exception: "Invalid blob shape": elementwise_kernel_cpu: Cannot broadcast [12, 2, 1, 256, 1] and [12, 1, 1, 256, 1] status=-1
2022-02-16 12:04:35.315728+0000 MLModelCamera[12740:4737111] [coreml] Error computing NN outputs -1
2022-02-16 12:04:35.315755+0000 MLModelCamera[12740:4737111] [coreml] Failure in -executePlan:error:.
failed to perform

Screenshot from 2022-02-16 12-08-20

I checked the layer 240 of the graph, pretty sure the ResizeBilinear caused this problem, as it has also been mentioned in many onnx->coreml conversion, e.g. :

(onnx/onnx-coreml#474)
(apple/coremltools#1105)

I also noticed that the new height and new width in resizebilinear layer are all 0. Is that something related?

Thanks

@PINTO0309
Copy link
Owner

I don't think the number 240 in the error message is directly related to the error. The error occurs in the operation named model_1/model/add_24/add.

[Exception from layer 240: model_1/model/add_24/add]Espresso exception: "Invalid blob shape": elementwise_kernel_cpu: Cannot broadcast [12, 2, 1, 256, 1] and [12, 1, 1, 256, 1] status=-1

image

The shape of the tensor shown in the error message is obviously wrong.

Cannot broadcast [12, 2, 1, 256, 1] and [12, 1, 1, 256, 1]

This is because there are no operations in this model that process in 5 dimensions.

I also noticed that the new height and new width in resizebilinear layer are all 0. Is that something related?

No, not at all. It is correct that new_height and new_width are set to zero, as per the specification. Since the second input is set to [24, 24], ResizeBilinear (upsample) will produce an output of size 24 in height and width.
image

The way I see it, it's a bug in CoreML.

@TaoZappar
Copy link
Author

TaoZappar commented Feb 16, 2022 via email

@TaoZappar
Copy link
Author

TaoZappar commented Feb 16, 2022 •
edited

Hi, Katsuya

I am glad to tell you I find a way to work around. I was inspired by this article https://machinethink.net/blog/coreml-upsampling/ (go to the Cheatsheet section)
Instead of using tf.image.resize, I use tf.compat.v1.image.resize_bilinear. The hacked code I used for coreml specifically is as follows:

        elif op_type == 'RESIZE_BILINEAR':
            input_tensor = tensors[op['inputs'][0]]
            size_detail = interpreter._get_tensor_details(op['inputs'][1])
            size = interpreter.get_tensor(size_detail['index'])
            size_height = size[0]
            size_width = size[1]

            options = op['builtin_options']
            align_corners = options['align_corners']
            half_pixel_centers = options['half_pixel_centers']

            def upsampling2d_bilinear(x, size_height, size_width, align_corners, half_pixel_centers):
                if optimizing_for_edgetpu_flg:
                    return tf.image.resize_bilinear(x, (size_height, size_width))
                else:
                    if optimizing_for_openvino_and_myriad:
                        if half_pixel_centers:
                            return tf.image.resize_bilinear(
                                x,
                                (size_height, size_width),
                                align_corners=False,
                                half_pixel_centers=half_pixel_centers
                            )
                        else:
                            return tf.image.resize_bilinear(
                                x,
                                (size_height, size_width),
                                align_corners=True,
                                half_pixel_centers=half_pixel_centers
                            )
                    else:
                        y = tf.compat.v1.image.resize_bilinear(x, [size_height, size_width], align_corners=True) # slightly better way, which works on IOS
                        # y = tfv2.keras.layers.UpSampling2D()(x) #   no matching solution in COREML
                        # y = tfv2.image.resize(                  #   no matching solution in COREML
                        #     x,
                        #     [size_height, size_width],
                        #     method='bilinear'
                        # )
                        return y

@PINTO0309
Copy link
Owner

I see. I will add an option to enable CoreML optimization. Thank you for doing some meaningful research! 😸

@PINTO0309
Copy link
Owner

PINTO0309 commented Feb 17, 2022 •
edited

Commit: ab7d962
Release: https://github.com/PINTO0309/tflite2tensorflow/releases/tag/v1.19.0

image

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@PINTO0309 @TaoZappar