[FRONTEND][TFLITE] Add support for TFLite_Detection_PostProcess #4543

mbaret · 2019-12-18T17:02:50Z

This adds support for the custom operator TFLite_Detection_PostProcess which is commonly used in object detection networks such as SSD Mobilenet. It only adds support for when use_regular_nms = False.

This implementation makes use of the existing multibox_transform_loc and non_max_suppression operators. Their design is closely coupled to the mxnet version of this custom operator, so a number of transformations have to be added before and after to get them to accept the tflite format for this operator.

This PR depends on the fix in #4541.

kevinthesun · 2019-12-18T18:32:00Z

@yongwww

python/tvm/relay/frontend/tflite.py

FrozenGene

Only minor comment on code style

python/tvm/relay/frontend/tflite.py

tests/python/frontend/tflite/test_forward.py

yongwww · 2019-12-20T17:49:00Z

tests/python/frontend/tflite/test_forward.py

+def test_detection_postprocess():
+    tf_model_file = tf_testing.get_workload_official(
+        "http://download.tensorflow.org/models/object_detection/"
+        "ssd_mobilenet_v2_quantized_300x300_coco_2019_01_03.tar.gz",


would you mind adding more models like ssd_mobilenetv1?

I can do, but where would you like me to pull it from? I see that ssd mobilenet v1 without the post process op is hosted under "https://raw.githubusercontent.com/dmlc/web-data/master/tensorflow/models/", would it be possible to host the version with the post process op here as well?

If possible, we'd like to pull the model from the related official website, for example https://www.tensorflow.org/lite/models/object_detection/overview for ssd mobilenet v1.

OK - I did see that model but weirdly it was as a .zip, not a tar as with most other hosted models. I'll see if I can open another PR to extend get_workload_official to zips and then will add the test.

The test looks non-trivial to add because quite a small difference in the convolutional part of the network can result in significant changes to the ordering of the output tensor (eg. we might see at different detection at the cut off threshold). I'm not sure what the best way is to proceed, do you have any thoughts?

Alright, we could remove ssd mobilenet model because of this limitation, but we should still keep the unit testing of detection postprocess. After we resolve the limitation, we could add ssd mobilenet testing back. Morever, we could remove the atol=1 of test_qconv2d and so on. Because we could get the same result completely compared with the tflite. Does it make sense to you?

This test is a bit misleading because it doesn't actually run ssd mobilenet, it just test the postprocess op. I couldn't find a way to create the op using the tflite python API, so what I did instead was take a model that has it and then run it through the tflite converter but with the converter inputs set to the inputs of the postprocess op rather than the input to the network.

This has the net effect of producing a single postprocess op, so this should already be a unit test (and it passes). I can add the end-to-end tests if/when we resolve the QNN accuracy issue. I'll open an RFC shortly to describe why rounding is a particularly significant in the case of this operator.

I think if we could view the TOCO source code, maybe we could find how to construct detection_postprocess. Please refer our _test_prelu comment. I ever write what the pattern tflite could produce prelu. However, current way is acceptable too in my opinion.

I've written a discuss post here: 5528.

@mbaret How did you set converter input as inputs of postprocess op, when I do that it gives me error :
tensorflow/lite/toco/model_cmdline_flags.cc:263] Check failed: mean_values.size() == model_flags->input_arrays_size()

The inputs to postprocess op >1 ('raw_outputs/box_encodings','raw_outputs/class_predictions') also anchors constant

python/tvm/relay/frontend/tflite.py

yongwww

overall lgtm, pls fix comments

tqchen · 2020-01-10T22:59:03Z

@mbarrett97 please update as per review comment, @FrozenGene please https://docs.tvm.ai/contribute/code_review.html#approve-and-request-changes-explicitly

kevinthesun

LGTM

python/tvm/relay/frontend/tflite.py

mbaret · 2020-01-29T22:17:34Z

@FrozenGene are there any other changes you want?

FrozenGene

Final comment.

python/tvm/relay/frontend/tflite.py

tqchen · 2020-02-06T03:40:16Z

@mbarrett97 please rebase, @FrozenGene please followup :)

This adds support for the custom operator TFLite_Detection_PostProcess which is commonly used in object detection networks such as SSD Mobilenet. It only adds support for when use_regular_nms = False. Change-Id: I819b253c0eb6f0fa55da65d2634e09359b888828

Change-Id: Ie5baa092deae9a8bcffd2ebd9f6d346b90e58afd

Change-Id: Ib08f02b5f1a59a883048bfb36e4321152cd2e7f2

Change-Id: If1171fc03d211a809cedeb800804394972af4060

Change-Id: I3ce7e69b8d2c73aec57369c1c64ea1eec07f087b

Change-Id: I49eaafc3369070f8f3e85fbb965ad20972096c68

Change-Id: I542a787d11422ea83c52147b2cb1144fcef0dd77

Change-Id: I2971b8ecebe08c882b2481a99f67cfbe515e0b1f

Change-Id: I393f3b3b62be73e427498d98456fb1d5a214e0af

The linter was updated, so I needed to fix a small style issue as a result. Change-Id: Ia3c954565a00de92e7fb1912eae9ed9875d60c7c

u99127 · 2020-02-12T13:14:51Z

All changes have been done ? Anything left to merge this in ?

tqchen · 2020-02-12T16:22:13Z

@FrozenGene please https://docs.tvm.ai/contribute/code_review.html#approve-and-request-changes-explicitly

FrozenGene · 2020-02-13T02:07:51Z

Thanks everyone , merged now

…he#4543) * [FRONTEND][TFLITE] Add support for TFLite_Detection_PostProcess This adds support for the custom operator TFLite_Detection_PostProcess which is commonly used in object detection networks such as SSD Mobilenet. It only adds support for when use_regular_nms = False. Change-Id: I819b253c0eb6f0fa55da65d2634e09359b888828 * Added a test for the tflite custom op Change-Id: Ie5baa092deae9a8bcffd2ebd9f6d346b90e58afd * Removed trailing comma Change-Id: Ib08f02b5f1a59a883048bfb36e4321152cd2e7f2 * Added spaces between divide Change-Id: If1171fc03d211a809cedeb800804394972af4060 * Formatted comment Change-Id: I3ce7e69b8d2c73aec57369c1c64ea1eec07f087b * Reduced line length in test Change-Id: I49eaafc3369070f8f3e85fbb965ad20972096c68 * Set random seed for test Change-Id: I542a787d11422ea83c52147b2cb1144fcef0dd77 * Fixes to style Change-Id: I2971b8ecebe08c882b2481a99f67cfbe515e0b1f * Assert for incorrect number of inputs Change-Id: I393f3b3b62be73e427498d98456fb1d5a214e0af * Change comparison to pass linting The linter was updated, so I needed to fix a small style issue as a result. Change-Id: Ia3c954565a00de92e7fb1912eae9ed9875d60c7c

kevinthesun reviewed Dec 18, 2019

View reviewed changes

python/tvm/relay/frontend/tflite.py Outdated Show resolved Hide resolved

FrozenGene reviewed Dec 19, 2019

View reviewed changes

python/tvm/relay/frontend/tflite.py Outdated Show resolved Hide resolved

python/tvm/relay/frontend/tflite.py Outdated Show resolved Hide resolved

tqchen added the status: need update need update based on feedbacks label Dec 19, 2019

yongwww requested changes Dec 20, 2019

View reviewed changes

tqchen assigned kevinthesun Jan 6, 2020

yongwww approved these changes Jan 6, 2020

View reviewed changes

kevinthesun approved these changes Jan 10, 2020

View reviewed changes

FrozenGene requested changes Jan 11, 2020

View reviewed changes

python/tvm/relay/frontend/tflite.py Outdated Show resolved Hide resolved

FrozenGene requested changes Jan 11, 2020

View reviewed changes

python/tvm/relay/frontend/tflite.py Outdated Show resolved Hide resolved

python/tvm/relay/frontend/tflite.py Outdated Show resolved Hide resolved

mbaret force-pushed the ssd branch from 3dcba97 to 497f82e Compare January 27, 2020 16:10

FrozenGene reviewed Jan 30, 2020

View reviewed changes

python/tvm/relay/frontend/tflite.py Show resolved Hide resolved

mbaret force-pushed the ssd branch from 497f82e to c211f6a Compare February 3, 2020 16:34

mbaret force-pushed the ssd branch from c211f6a to bc54a20 Compare February 10, 2020 10:38

mbaret added 10 commits February 10, 2020 11:00

Added a test for the tflite custom op

6378ef5

Change-Id: Ie5baa092deae9a8bcffd2ebd9f6d346b90e58afd

Removed trailing comma

eeb611e

Change-Id: Ib08f02b5f1a59a883048bfb36e4321152cd2e7f2

Added spaces between divide

aa678cf

Change-Id: If1171fc03d211a809cedeb800804394972af4060

Formatted comment

505e7f0

Change-Id: I3ce7e69b8d2c73aec57369c1c64ea1eec07f087b

Reduced line length in test

a7fc2a3

Change-Id: I49eaafc3369070f8f3e85fbb965ad20972096c68

Set random seed for test

5f9cba6

Change-Id: I542a787d11422ea83c52147b2cb1144fcef0dd77

Fixes to style

7fca345

Change-Id: I2971b8ecebe08c882b2481a99f67cfbe515e0b1f

Assert for incorrect number of inputs

7e69839

Change-Id: I393f3b3b62be73e427498d98456fb1d5a214e0af

Change comparison to pass linting

5b7f549

The linter was updated, so I needed to fix a small style issue as a result. Change-Id: Ia3c954565a00de92e7fb1912eae9ed9875d60c7c

mbaret force-pushed the ssd branch from bc54a20 to 5b7f549 Compare February 10, 2020 12:12

tqchen added status: need review and removed status: need update need update based on feedbacks labels Feb 12, 2020

FrozenGene approved these changes Feb 13, 2020

View reviewed changes

FrozenGene merged commit 70c6382 into apache:master Feb 13, 2020

anijain2305 mentioned this pull request Apr 24, 2020

TFlite e2e FP32 Object detection model #5439

Closed

ZihengJiang mentioned this pull request Sep 17, 2020

TVM v0.7 Release Note Candidate #6486

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FRONTEND][TFLITE] Add support for TFLite_Detection_PostProcess #4543

[FRONTEND][TFLITE] Add support for TFLite_Detection_PostProcess #4543

mbaret commented Dec 18, 2019 •

edited

kevinthesun commented Dec 18, 2019

FrozenGene left a comment

yongwww Dec 20, 2019

mbaret Jan 6, 2020

yongwww Jan 6, 2020

mbaret Jan 8, 2020

mbaret Jan 14, 2020

FrozenGene Jan 27, 2020

mbaret Jan 27, 2020

FrozenGene Jan 27, 2020 •

edited

mbaret Jan 28, 2020

sjoshi30 Feb 20, 2020 •

edited

yongwww left a comment

tqchen commented Jan 10, 2020

kevinthesun left a comment

mbaret commented Jan 29, 2020

FrozenGene left a comment

tqchen commented Feb 6, 2020

u99127 commented Feb 12, 2020

tqchen commented Feb 12, 2020

FrozenGene commented Feb 13, 2020

[FRONTEND][TFLITE] Add support for TFLite_Detection_PostProcess #4543

[FRONTEND][TFLITE] Add support for TFLite_Detection_PostProcess #4543

Conversation

mbaret commented Dec 18, 2019 • edited

kevinthesun commented Dec 18, 2019

FrozenGene left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FrozenGene Jan 27, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sjoshi30 Feb 20, 2020 • edited

Choose a reason for hiding this comment

yongwww left a comment

Choose a reason for hiding this comment

tqchen commented Jan 10, 2020

kevinthesun left a comment

Choose a reason for hiding this comment

mbaret commented Jan 29, 2020

FrozenGene left a comment

Choose a reason for hiding this comment

tqchen commented Feb 6, 2020

u99127 commented Feb 12, 2020

tqchen commented Feb 12, 2020

FrozenGene commented Feb 13, 2020

mbaret commented Dec 18, 2019 •

edited

FrozenGene Jan 27, 2020 •

edited

sjoshi30 Feb 20, 2020 •

edited