Support per-channel quantized INT8 weights unpacking in XNNPACK delegate #50875

dev0x13 · 2021-07-21T13:58:00Z

This MR expands INT8 weights unpacking for FP32 inference in XNNPACK delegate support added in the previous MR for per-channel dynamic range quantized model. Previous changes have only supported per-tensor quantization mode, which is obsolete in recent TensorFlow releases.
I have not added proper testing yet, because I'd like to get some suggestions from a maintainer, i.e. the best way to organize such testing while keeping the codebase clean (specifically, how to change testers). Thank you in advance!

dev0x13 · 2021-08-02T08:33:36Z

@multiverse-tf Could you review this please?

…EADME

gbaned · 2021-08-10T14:44:57Z

@dev0x13 Can you please resolve conflicts? Thanks!

dev0x13 · 2021-08-10T14:52:32Z

@gbaned Done

dev0x13 · 2021-08-17T13:45:06Z

@multiverse-tf Could you review this please?

dev0x13 · 2021-08-25T07:51:24Z

It's been more than a month since I've created this MR. Should I close it now due to the lack of review?

multiverse-tf · 2021-08-25T08:54:36Z

It's been more than a month since I've created this MR. Should I close it now due to the lack of review?

Sorry for not catching this review request earlier. In the past month, I think we've also added native per-channel quantized INT8 op support in XNNPACK. @Maratyszcza, could you shed more light on the support? Thx

dev0x13 · 2021-08-27T08:33:39Z

@multiverse-tf XNNPACK does not support per-channel dynamic range quantization. This MR adds the support.

dev0x13 · 2021-09-13T09:12:50Z

Closing due to the lack of response from maintainers.

…ate (MR review fixes) 1. Minor cosmetic fixes suggested by MR reviewer are applied to XNNPACK delegate test set. 2. Tests for per-channel quantized INT8 weights unpacking in XNNPACK delegate are refactored to perform proper per-channel quantization parameters computation at runtime instead of using random-initialized quantized tensors. 3. Added per-tensor and per-channel quantized weights unpacking tests for TRANSPOSE_CONV op in XNNPACK delegate.

dev0x13 · 2021-11-16T10:44:58Z

@Maratyszcza Thank you for the review! All the changes are made.

Maratyszcza · 2021-11-17T02:35:02Z

tensorflow/lite/delegates/xnnpack/binary_elementwise_tester.cc

@@ -66,7 +66,7 @@ void BinaryElementwiseTester::Test(tflite::BuiltinOperator binary_op,
  if (Input1Static()) {
    ASSERT_FALSE(Input2Static());
  }
-  if (FP16Weights() || INT8Weights()) {
+  if (FP16Weights() || INT8Weights() || INT8ChannelWiseWeights()) {
    ASSERT_TRUE(Input1Static() || Input2Static());
  }



Check that if channelwise weights are used, the static input has at least one dimension (i.e. isn't a scalar)

Should be if (INT8ChannelWiseWeights() && Input1Static()) ... + the same for input2

Sorry, my bad. Fixed

Maratyszcza · 2021-11-17T02:59:11Z

tensorflow/lite/delegates/xnnpack/binary_elementwise_tester.cc

-            std::bind(QuantizeInt8, std::placeholders::_1, 0, input1_scale));
+        input1_scales.resize(1);
+        input1_zero_points.resize(1, 0);
+        input1_scales[0] = GetInt8QuantizationScale(input1_data);


Combine with resize

Maratyszcza · 2021-11-17T03:00:11Z

tensorflow/lite/delegates/xnnpack/test_util.cc

+  std::vector<int> current_dim(num_dims, 0);
+
+  do {
+    size_t offset =


const size_t

Maratyszcza · 2021-11-17T03:01:39Z

tensorflow/lite/delegates/xnnpack/test_util.cc

+                            current_dim.data(), 0, nullptr);
+    const int channel_idx = current_dim[quantized_dimension];
+    const float val = data[offset];
+    if (has_min_max_value[channel_idx]) {


Initialize min to -std::numeric_limits<float>::infinity() and max to +std::numeric_limits<float>::infinity() and remove these checks

Maratyszcza

LG overall. Please revert the change in README and simplify the code per comments, and we're good to go.

…ate (MR review fixes) 1. Reverted changes in XNNPACK delegate README. 2. Made some adjustments for XNNPACK tests code suggested by the MR reviewer.

dev0x13 · 2021-11-17T09:18:56Z

@Maratyszcza Thank you! All the suggested changes are made.

Maratyszcza · 2021-11-19T16:04:21Z

tensorflow/lite/delegates/xnnpack/test_util.cc

+                            current_dim.data(), 0, nullptr);
+    const int channel_idx = current_dim[quantized_dimension];
+    const float val = data[offset];
+    if (min[channel_idx] > val) {


These lines can be further simplified using std::min, std::max

…ate (MR review fixes)

Maratyszcza · 2021-11-19T19:12:47Z

LGTM. @gbaned could you merge?

PiperOrigin-RevId: 411934765 Change-Id: Ida9cd3723742a3b92139345fccc29b60d5383be0

dev0x13 · 2021-11-26T13:32:12Z

I noticed that the PR was rolled back right after merge. Is there something wrong with it?

Maratyszcza · 2021-11-26T18:57:41Z

transpose_conv_test failed under sanitizers with UndefinedBehaviorSanitizer: float-cast-overflow tensorflow/lite/delegates/xnnpack/test_util.cc:32:28

dev0x13 · 2021-11-27T15:34:28Z

@Maratyszcza Fixed this in my fork, but it seems that this PR cannot be reopened. What do I need to do in order to submit this fix to make my changes merged again? Thank you in advance!

dev0x13 · 2021-12-03T10:10:57Z

@Maratyszcza I am really sorry for bothering you, but could you clarify what I need to do in order to submit this fix to make my changes merged again?

Maratyszcza · 2021-12-04T23:16:14Z

prelu_test fails under asan too, with the same error

dev0x13 · 2021-12-06T09:56:25Z

@Maratyszcza Fixed

Maratyszcza · 2021-12-09T04:27:25Z

Re-landed in a6d352f

dev0x13 · 2021-12-09T05:08:32Z

Thank you!

Support per-channel quantized INT8 weights unpacking in XNNPACK delegate

8d336e9

google-ml-butler bot added the size:M CL Change Size: Medium label Jul 21, 2021

google-cla bot added the cla: yes label Jul 21, 2021

gbaned self-assigned this Jul 21, 2021

gbaned added the comp:lite TF Lite related issues label Jul 21, 2021

gbaned added this to Assigned Reviewer in PR Queue via automation Jul 21, 2021

gbaned requested a review from wangtz July 21, 2021 15:11

gbaned added the awaiting review Pull request awaiting review label Jul 23, 2021

wangtz requested a review from multiverse-tf July 26, 2021 09:48

dev0x13 added 2 commits August 9, 2021 11:37

Merge branch 'master' into xnnpack-per-channel-int8-weights-unpack

c906f53

Mention quantized INT8 weights unpacking option in XNNPACK delegate R…

0020076

…EADME

gbaned requested review from multiverse-tf and removed request for multiverse-tf August 9, 2021 15:31

gbaned added stat:awaiting response Status - Awaiting response from author and removed awaiting review Pull request awaiting review labels Aug 10, 2021

Merge branch 'master' into xnnpack-per-channel-int8-weights-unpack

a4a9bbe

tensorflowbutler removed the stat:awaiting response Status - Awaiting response from author label Aug 12, 2021

multiverse-tf removed the request for review from wangtz August 25, 2021 08:55

dev0x13 closed this Sep 13, 2021

PR Queue automation moved this from Assigned Reviewer to Closed/Rejected Sep 13, 2021

dev0x13 reopened this Oct 17, 2021

PR Queue automation moved this from Closed/Rejected to Assigned Reviewer Oct 17, 2021

PR Queue automation moved this from Assigned Reviewer to Reviewer Requested Changes Nov 15, 2021

dev0x13 added 2 commits November 16, 2021 11:24

Merge branch 'master' into xnnpack-per-channel-int8-weights-unpack

56fbd17

Maratyszcza reviewed Nov 17, 2021

View reviewed changes

Maratyszcza suggested changes Nov 17, 2021

View reviewed changes

Support per-channel quantized INT8 weights unpacking in XNNPACK deleg…

c1086be

…ate (MR review fixes) 1. Reverted changes in XNNPACK delegate README. 2. Made some adjustments for XNNPACK tests code suggested by the MR reviewer.

tensorflowbutler removed the awaiting review Pull request awaiting review label Nov 19, 2021

Maratyszcza reviewed Nov 19, 2021

View reviewed changes

Support per-channel quantized INT8 weights unpacking in XNNPACK deleg…

4953170

…ate (MR review fixes)

Maratyszcza approved these changes Nov 19, 2021

View reviewed changes

google-ml-butler bot added kokoro:force-run Tests on submitted change ready to pull PR ready for merge process labels Nov 19, 2021

kokoro-team removed the kokoro:force-run Tests on submitted change label Nov 19, 2021

copybara-service bot merged commit 15d21b8 into tensorflow:master Nov 24, 2021

google-ml-butler bot removed the ready to pull PR ready for merge process label Nov 24, 2021

copybara-service bot pushed a commit that referenced this pull request Nov 24, 2021

Rollback of PR #50875

784f602

PiperOrigin-RevId: 411934765 Change-Id: Ida9cd3723742a3b92139345fccc29b60d5383be0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support per-channel quantized INT8 weights unpacking in XNNPACK delegate #50875

Support per-channel quantized INT8 weights unpacking in XNNPACK delegate #50875

dev0x13 commented Jul 21, 2021

dev0x13 commented Aug 2, 2021

gbaned commented Aug 10, 2021

dev0x13 commented Aug 10, 2021

dev0x13 commented Aug 17, 2021

dev0x13 commented Aug 25, 2021

multiverse-tf commented Aug 25, 2021

dev0x13 commented Aug 27, 2021

dev0x13 commented Sep 13, 2021

dev0x13 commented Nov 16, 2021

Maratyszcza Nov 17, 2021

dev0x13 Nov 17, 2021

Maratyszcza Nov 19, 2021

dev0x13 Nov 19, 2021

Maratyszcza Nov 17, 2021

dev0x13 Nov 17, 2021

Maratyszcza Nov 17, 2021

dev0x13 Nov 17, 2021

Maratyszcza Nov 17, 2021

dev0x13 Nov 17, 2021

Maratyszcza left a comment

dev0x13 commented Nov 17, 2021

Maratyszcza Nov 19, 2021

dev0x13 Nov 19, 2021

Maratyszcza commented Nov 19, 2021

dev0x13 commented Nov 26, 2021

Maratyszcza commented Nov 26, 2021

dev0x13 commented Nov 27, 2021

dev0x13 commented Dec 3, 2021

Maratyszcza commented Dec 4, 2021

dev0x13 commented Dec 6, 2021

Maratyszcza commented Dec 9, 2021

dev0x13 commented Dec 9, 2021

Support per-channel quantized INT8 weights unpacking in XNNPACK delegate #50875

Support per-channel quantized INT8 weights unpacking in XNNPACK delegate #50875

Conversation

dev0x13 commented Jul 21, 2021

dev0x13 commented Aug 2, 2021

gbaned commented Aug 10, 2021

dev0x13 commented Aug 10, 2021

dev0x13 commented Aug 17, 2021

dev0x13 commented Aug 25, 2021

multiverse-tf commented Aug 25, 2021

dev0x13 commented Aug 27, 2021

dev0x13 commented Sep 13, 2021

dev0x13 commented Nov 16, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Maratyszcza left a comment

Choose a reason for hiding this comment

dev0x13 commented Nov 17, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Maratyszcza commented Nov 19, 2021

dev0x13 commented Nov 26, 2021

Maratyszcza commented Nov 26, 2021

dev0x13 commented Nov 27, 2021

dev0x13 commented Dec 3, 2021

Maratyszcza commented Dec 4, 2021

dev0x13 commented Dec 6, 2021

Maratyszcza commented Dec 9, 2021

dev0x13 commented Dec 9, 2021