Enable Exynos backend Quatization

[Jiseong-oh]

Summary

• Implemented quantized strategies for enn-backend.
• Added support for ENN's quantization strategies.
• Successfully verified multiple quantized models.

Test plan

python -m executorch.examples.samsung.scripts.${MODEL_NAME} -c e9955 -p A8W8

cc @SS-JIA @digantdesai @kimishpatel

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/14464

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 5 New Failures, 27 Pending, 3 Unrelated Failures

As of commit 3d58d14 with merge base d39992f ():

NEW FAILURES - The following jobs have failed:

• pull / test-qnn-wheel-packages-linux (3.10) / linux-job (gh)
  RuntimeError: Command docker exec -t 1f472b1b2dd1a78a05031f242f889d34fb5591c5ff1d273105c4eaf2af1a2d0c /exec failed with exit code 1
• pull / test-qnn-wheel-packages-linux (3.11) / linux-job (gh)
  RuntimeError: Command docker exec -t ccad68d912f84746c4fcd46584a13f88ad2f5b626c5f7e5fc5b4cbef41d96d66 /exec failed with exit code 1
• pull / test-qnn-wheel-packages-linux (3.12) / linux-job (gh)
  RuntimeError: Command docker exec -t e378c0441db5330f95ce1637e841d57957946b8a05ebc2e5bafb7b2afaafd797 /exec failed with exit code 1
• pull / test-samsung-models-linux / linux-job (gh)
  RuntimeError: Command docker exec -t a6d9315bd243ea01a57f8e5a09b8af4a9945db28fe7309532d36945ade15adbd /exec failed with exit code 1
• Test CUDA Builds / test-voxtral-cuda-e2e / linux-job (gh)
  RuntimeError: Command docker exec -t f103b3d92ad74df1ad2e5d2da41d3d3083135994f949311be01aff472ad13c90 /exec failed with exit code 2

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• pull / unittest / linux / linux-job (gh) (detected as infra flaky with no log or failing log classifier)
• pull / unittest-arm-backend-with-no-fvp (test_pytest_ops) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)
• trunk / test-qnn-model (fp32, mb) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[SS-JIA]

Why not apply these passes in enn_preprocess.py instead? With the current pattern, users will have to remember to call get_enn_pass_list() and pass the result to to_edge_transform_and_lower() which creates a point of failure if they forget to call it.

It seems to me that at the very least AnnotateQparamsPass should be moved to enn_preprocess.py, since it is required to preserve the quantization parameters. If it is not applied, then the FoldQDQPass() applied at the start of enn_preprocess.py will erase all quantization parameter information.

[Chen03ZhaoSamsung]

We want some passes to be done before to_backend, and we define to_edge_transform_and_lower_enn to make it easy to use. AnnotateQparamsPass could be moved to enn_preprocess.py, so we have moved it now. Thanks for your recommendation.

[Sangsooko]

Thanks for pointing this out. I'll address this item in the next update.

[SS-JIA]

I'm curious in what situations this is needed. Typically, after the pt2e quantization process, weight tensors should already be quantized and have int8 data type.

[Chen03ZhaoSamsung]

It seems constant tensors are not quantized after convert_pt2e, but just has quant/dequant node after the constant tensor nodes. So we quantize all the constant tensors here.

[SS-JIA]

Would you happen to have an example of this behaviour? My understanding is that the activation tensor will have quant/dequant, but the weight tensor will have only a quant node. If you inspect the tensor data backing the weight tensor, it should have int8 type.

For instance, here are some logs I collected while debugging MobileNet V2 quantized with XNNPACK Quantizer that I got a while back:

# dq input node:
torch.float32: torch.Size([1, 384, 14, 14]) = quantized_decomposed::dequantize_per_tensor(torch.int8: torch.Size([1, 384, 14, 14]), 0.06534181535243988, 16, -128, 127, torch.int8,  ...)
# dq weight node:
torch.float32: torch.Size([96, 384, 1, 1]) = quantized_decomposed::dequantize_per_channel(torch.int8: torch.Size([96, 384, 1, 1]), torch.float32: torch.Size([96]), torch.int64: torch.Size([96]), 0, -127, 127, torch.int8,  ...)
# weight tensor:
tensor: torch.Size([96, 384, 1, 1]), torch.int8, -127, 127
# scales tensor:
tensor: torch.Size([96]), torch.float32, 0.004114292096346617, 0.009169017896056175
# zeros tensor:
tensor: torch.Size([96]), torch.int64, 0, 0
# conv node:
torch.float32: torch.Size([1, 96, 14, 14]) = aten::conv2d(torch.float32: torch.Size([1, 384, 14, 14]), torch.float32: torch.Size([96, 384, 1, 1]), torch.float32: torch.Size([96]),  ...)

[Sangsooko]

As you can see in the following code which is a part of mobilebert, the div has quantized input and a constant input.
The above code changes the constant number of 5.656854249492381 into a quantized one.
Because this changing is only for the internal operation with real HWs, the code can be moved to our backend codes.

    %div : [num_users=1] = call_function[target=torch.ops.aten.div.Tensor](args = (%dequantize_per_tensor_default_424, 5.656854249492381), kwargs = {})
    %quantize_per_tensor_default_425 :   [num_users=1] = call_function[target=torch.ops.quantized_decomposed.quantize_per_tensor.default](args = (%div, 0.15715381503105164, -78, -128, 127, torch.int8), kwargs = {})
    %dequantize_per_tensor_default_425 : [num_users=1] = call_function[target=torch.ops.quantized_decomposed.dequantize_per_tensor.default](args = (%quantize_per_tensor_default_425, 0.15715381503105164, -78, -128, 127, torch.int8), kwargs = {out_dtype: torch.float16})

If you think the additional quantization process after convert_pt2e is not appropriate according to the ExecuTorch code policy, we will move this code to our backend in the next update.

[SS-JIA]

Understood. I figured it might have been for binary ops that operate with constant args. In that case LGTM!

[SS-JIA]

Also, please run lintrunner to fix the lint errors.

# install lintrunner dependencies
pip install lintrunner
pip install lintrunner_adapters

cd executorch
lintrunner -a --verbose

[SS-JIA]

Overall LGTM, but before I stamp:

1. Could you guys rebase to latest main and re-submit the PR? I want to see if the failures get resolved.
2. There's a remaining comment regarding manually quantizing constant tensors when serializing the model. To me, I don't think this should be required since the quantization workflow should be quantizing the constant tensor for you already.

[SS-JIA]

LGTM! Thanks for addressing the comments.

Would you mind rebasing one more time before merging? Just to ensure that the merge base is fairly up to date. Thanks!

[SS-JIA]

Sorry, one more thing. In addition to rebasing, can you include the fix for getting the AI_LITECORE_API_KEY that's implemented in #14866?

The fix was to move up when secrets-env is declared.

[SS-JIA]

I decided to just merge #14866, so please just rebase past it and this PR LGTM!

[SS-JIA]

Validated that samsung test is passing via #14977. I believe that this PR cannot access the repository API secret because the merge branch is from a fork.