Quantize compatible node + activation patterns as one block

[Tessil]

Annotate conv1d/conv2d/linear followed by relu/relu6 patterns as one block and fuse the activation into its parent.

The activation will then be implicitly done in the tosa.rescale node that will have a -128 zero-point.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/7555

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit b60972b with merge base 08770b7 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[swolchok]

This PR caused a type-checking failure:

executorch/backends/arm/_passes/fuse_quantized_activation_pass.py:27:48 Uninitialized local [61]: Local variable `zp` is undefined, or not always defined.
executorch/backends/arm/_passes/fuse_quantized_activation_pass.py:27:54 Uninitialized local [61]: Local variable `qmin` is undefined, or not always defined.

[freddan80]

@Tessil will you have a look?

[Tessil]

@swolchok Thanks for the info. What would be the best way to fix it as the PR has already gone through? Raise a new one with a fix?

Note that the zp and qmin will theoretically never be evaluated if they are uninitialized with the short-circuit and as they are initialized in any case when is_quantized is true and evaluated only if is_quantized is true. Though still best to fix the type-checking issue.

[swolchok]

Raise a new one with a fix?

yep!

[Tessil]

@swolchok Fix: #7671
Thanks

[3l1]

hey @Tessil, a bit late to this PR, but ... can you please clarify why we are specializing on the check for zp == qmin only here? there should be allowance for several variations of quantization observers that may not satisfy this equality check?

cc @freddan80 @digantdesai @limintang @gggekov

[digantdesai]

Also FYI @Ninja91

[Ninja91]

@Tessil Following up on @3l1 question about this check's need?

[Tessil]

@3l1 @Ninja91 Sorry for the late reply. The FuseQuantizedActivationPass requires zp == qmin (which should always be the case for asymmetric quantization of activations with a [0, inf) range) as the activation lower bound cut is managed by the tosa.rescale.

[3l1]

@3l1 @Ninja91 Sorry for the late reply. The FuseQuantizedActivationPass requires zp == qmin (which should always be the case for asymmetric quantization of activations with a [0, inf) range) as the activation lower bound cut is managed by the tosa.rescale.

thanks. yes this makes sense, i just saw that you were checking for a relu, so this makes sense for that activation -- although i think expecting qmin may be a bit strict, it could easily be, depending on the quantization config, qmin + epsilon

[digantdesai]

as the activation lower bound cut is managed by the tosa.rescale.

Can you say more, the rescale OP should be able handle qmin != zp. Just trying to better understand, if the quantization flow is ensuring the zp is "valid" for [0, inf) range, why do we have this constraint here.

[Tessil]

The tosa.rescale op is able to handle qmin != zp but with a tosa.rescale with qmin == zp we are sure that any rescaled output will be => 0 (when dequantized) and thus the rescale does an implicit ReLU (hence why it is optimised out).

It might be possible to relax the constraint but that would be best to check with the other members of the team as I'm working on a different project.

[digantdesai]

Thanks @Tessil

the rescale does an implicit ReLU (hence why it is optimised out)

Makes sense to remove relu through rescale qmin. That said, @Ninja91 might also found a bug where quantizer does generate a Relu q/dq with qmin which is != zp when used in some pattern.

It might be possible to relax the constraint but that would be best to check with the other members of the team as I'm working on a different project.

Who would be the right PoC?

[Ninja91]

@Tessil Please loop in the right POCs on the linked bug.

[Tessil]

That said, @Ninja91 might also found a bug where quantizer does generate a Relu q/dq with qmin which is != zp when used in some pattern.

Yeah, that's quite strange as the output of a ReLU should only observe >= 0 values hence a zp equals to qmin for asymmetric quantization.

Who would be the right PoC?

@digantdesai @Ninja91 Best check with @freddan80

[digantdesai]

FYI - #12959