[Quant] Respect non_leaf_module_list for activation modules #88498

andrewor14 · 2022-11-04T15:59:06Z

Stack from ghstack (oldest at bottom):

-> [Quant] Respect non_leaf_module_list for activation modules #88498

Summary: This commit fixes the bug where non_leaf_module_list
was not respected for activation modules like torch.nn.Sigmoid
and torch.nn.Tanh. Today, these modules default to
default_fixed_qparams_range_0to1_fake_quant, and there is no
way to configure them to use any other activation_post_process
(e.g. FixedQParamsObserver) (see this mapping).
non_leaf_module_list is a "list of non-leaf modules we want
to add observer" (see prepare docstring). If the user explicitly
specified to insert observers for these modules, we should respect
that instead of continuing to use the default.

Test Plan:
python test/test_quantization.py TestQuantizeEagerPTQStatic.test_activations_in_non_leaf_module_list

Reviewers: vkuzo, jerryzh168

Subscribers: vkuzo, jerryzh168

cc @jerryzh168 @jianyuh @raghuramank100 @jamesr66a @vkuzo @jgong5 @Xia-Weiwen @leslie-fang-intel

Summary: This commit fixes the bug where `non_leaf_module_list` was not respected for activation modules like `torch.nn.Sigmoid` and `torch.nn.Tanh`. Today, these modules default to `default_fixed_qparams_range_0to1_fake_quant`, and there is no way to configure them to use any other activation_post_process (e.g. FixedQParamsObserver). (source)[https://github.com/pytorch/pytorch/blob/dc00bb51b8d370bf3891f0edb2c6e0c2914e329a/torch/ao/quantization/quantization_mappings.py#L188-L193] `non_leaf_module_list` is a "list of non-leaf modules we want to add observer" (see prepare docstring). If the user explicitly specified to insert observers for these modules, we should respect that instead of continuing to use the default. Test Plan: python test/test_quantization.py TestQuantizeEagerPTQStatic.test_activations_in_non_leaf_module_list Reviewers: vkuzo, jerryzh168 Subscribers: vkuzo, jerryzh168 [ghstack-poisoned]

pytorch-bot · 2022-11-04T15:59:09Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/88498

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit e366ed8:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

Summary: This commit fixes the bug where `non_leaf_module_list` was not respected for activation modules like `torch.nn.Sigmoid` and `torch.nn.Tanh`. Today, these modules default to `default_fixed_qparams_range_0to1_fake_quant`, and there is no way to configure them to use any other activation_post_process (e.g. FixedQParamsObserver). (source)[https://github.com/pytorch/pytorch/blob/dc00bb51b8d370bf3891f0edb2c6e0c2914e329a/torch/ao/quantization/quantization_mappings.py#L188-L193] `non_leaf_module_list` is a "list of non-leaf modules we want to add observer" (see prepare docstring). If the user explicitly specified to insert observers for these modules, we should respect that instead of continuing to use the default. Test Plan: python test/test_quantization.py TestQuantizeEagerPTQStatic.test_activations_in_non_leaf_module_list Reviewers: vkuzo, jerryzh168 Subscribers: vkuzo, jerryzh168 ghstack-source-id: 8466b870c5f643d53b54fbac35fafa5c2841e919 Pull Request resolved: #88498

Summary: This commit fixes the bug where `non_leaf_module_list` was not respected for activation modules like `torch.nn.Sigmoid` and `torch.nn.Tanh`. Today, these modules default to `default_fixed_qparams_range_0to1_fake_quant`, and there is no way to configure them to use any other activation_post_process (e.g. FixedQParamsObserver). [source](https://github.com/pytorch/pytorch/blob/dc00bb51b8d370bf3891f0edb2c6e0c2914e329a/torch/ao/quantization/quantization_mappings.py#L188-L193) `non_leaf_module_list` is a "list of non-leaf modules we want to add observer" (see prepare docstring). If the user explicitly specified to insert observers for these modules, we should respect that instead of continuing to use the default. Test Plan: python test/test_quantization.py TestQuantizeEagerPTQStatic.test_activations_in_non_leaf_module_list Reviewers: vkuzo, jerryzh168 Subscribers: vkuzo, jerryzh168 [ghstack-poisoned]

Summary: This commit fixes the bug where `non_leaf_module_list` was not respected for activation modules like `torch.nn.Sigmoid` and `torch.nn.Tanh`. Today, these modules default to `default_fixed_qparams_range_0to1_fake_quant`, and there is no way to configure them to use any other activation_post_process (e.g. FixedQParamsObserver). [source](https://github.com/pytorch/pytorch/blob/dc00bb51b8d370bf3891f0edb2c6e0c2914e329a/torch/ao/quantization/quantization_mappings.py#L188-L193) `non_leaf_module_list` is a "list of non-leaf modules we want to add observer" (see prepare docstring). If the user explicitly specified to insert observers for these modules, we should respect that instead of continuing to use the default. Test Plan: python test/test_quantization.py TestQuantizeEagerPTQStatic.test_activations_in_non_leaf_module_list Reviewers: vkuzo, jerryzh168 Subscribers: vkuzo, jerryzh168 ghstack-source-id: 8466b870c5f643d53b54fbac35fafa5c2841e919 Pull Request resolved: #88498

Summary: This commit fixes the bug where `non_leaf_module_list` was not respected for activation modules like `torch.nn.Sigmoid` and `torch.nn.Tanh`. Today, these modules default to `default_fixed_qparams_range_0to1_fake_quant`, and there is no way to configure them to use any other activation_post_process (e.g. FixedQParamsObserver) (see this [mapping](https://github.com/pytorch/pytorch/blob/dc00bb51b8d370bf3891f0edb2c6e0c2914e329a/torch/ao/quantization/quantization_mappings.py#L188-L193)). `non_leaf_module_list` is a "list of non-leaf modules we want to add observer" (see prepare docstring). If the user explicitly specified to insert observers for these modules, we should respect that instead of continuing to use the default. Test Plan: python test/test_quantization.py TestQuantizeEagerPTQStatic.test_activations_in_non_leaf_module_list Reviewers: vkuzo, jerryzh168 Subscribers: vkuzo, jerryzh168 [ghstack-poisoned]

Summary: This commit fixes the bug where `non_leaf_module_list` was not respected for activation modules like `torch.nn.Sigmoid` and `torch.nn.Tanh`. Today, these modules default to `default_fixed_qparams_range_0to1_fake_quant`, and there is no way to configure them to use any other activation_post_process (e.g. FixedQParamsObserver) (see this [mapping](https://github.com/pytorch/pytorch/blob/dc00bb51b8d370bf3891f0edb2c6e0c2914e329a/torch/ao/quantization/quantization_mappings.py#L188-L193)). `non_leaf_module_list` is a "list of non-leaf modules we want to add observer" (see prepare docstring). If the user explicitly specified to insert observers for these modules, we should respect that instead of continuing to use the default. Test Plan: python test/test_quantization.py TestQuantizeEagerPTQStatic.test_activations_in_non_leaf_module_list Reviewers: vkuzo, jerryzh168 Subscribers: vkuzo, jerryzh168 ghstack-source-id: 8466b870c5f643d53b54fbac35fafa5c2841e919 Pull Request resolved: #88498

jerryzh168 · 2022-11-04T17:46:55Z

torch/ao/quantization/quantize.py

        elif non_leaf_module_list is not None and type_before_parametrizations(child) in non_leaf_module_list:
            if needs_observation(child):
                insert_activation_post_process(child)
+        elif _has_special_act_post_process(child):


is this a hack so that eager mode api is bc? should we also considering removing this hack by asking users to explicitly set fixed qparam configs for these ops so that people don't need to use non_leaf_module api to workaround this limitation?

I think so. There's no validation logic for eager mode right now so we just override it here. Yes I think simply removing it will make things cleaner, though it'll be a broader scope change since it'll break a lot of existing use cases. I'll leave that for a future PR to unblock #88456.

Filed #88579

jerryzh168

LGTM, I think we should also consider removing the hack and break the BC for these ops (I think we already did that in fx graph mode quant)

andrewor14 · 2022-11-04T20:03:31Z

Thanks, I'm merging this first to unblock #88456. We can discuss further if we want to do the validation for eager mode separately.

andrewor14 · 2022-11-04T20:03:48Z

@pytorchbot merge

pytorchmergebot · 2022-11-04T20:05:19Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

…88498) Summary: This commit fixes the bug where `non_leaf_module_list` was not respected for activation modules like `torch.nn.Sigmoid` and `torch.nn.Tanh`. Today, these modules default to `default_fixed_qparams_range_0to1_fake_quant`, and there is no way to configure them to use any other activation_post_process (e.g. FixedQParamsObserver) (see this [mapping](https://github.com/pytorch/pytorch/blob/dc00bb51b8d370bf3891f0edb2c6e0c2914e329a/torch/ao/quantization/quantization_mappings.py#L188-L193)). `non_leaf_module_list` is a "list of non-leaf modules we want to add observer" (see prepare docstring). If the user explicitly specified to insert observers for these modules, we should respect that instead of continuing to use the default. Test Plan: python test/test_quantization.py TestQuantizeEagerPTQStatic.test_activations_in_non_leaf_module_list Reviewers: vkuzo, jerryzh168 Subscribers: vkuzo, jerryzh168 Pull Request resolved: pytorch#88498 Approved by: https://github.com/jerryzh168

andrewor14 requested review from jerryzh168 and z-a-f as code owners November 4, 2022 15:59

pytorch-bot bot added the release notes: quantization release notes category label Nov 4, 2022

github-actions bot added the oncall: quantization Quantization support in PyTorch label Nov 4, 2022

andrewor14 added the topic: bug fixes topic category label Nov 4, 2022

andrewor14 mentioned this pull request Nov 4, 2022

[Quant] Allow setting fixed qparams for inner LSTM ops #88456

Closed

andrewor14 requested a review from vkuzo November 4, 2022 16:02

jerryzh168 reviewed Nov 4, 2022

View reviewed changes

jerryzh168 approved these changes Nov 4, 2022

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Nov 4, 2022

pytorchmergebot added the Merged label Nov 4, 2022

pytorchmergebot closed this in 7560a7b Nov 4, 2022

andrewor14 mentioned this pull request Nov 7, 2022

[Quant] Validate FixedQParams observers in eager mode #88579

Open

facebook-github-bot deleted the gh/andrewor14/32/head branch June 8, 2023 15:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Quant] Respect non_leaf_module_list for activation modules #88498

[Quant] Respect non_leaf_module_list for activation modules #88498

andrewor14 commented Nov 4, 2022 •

edited by pytorch-bot bot

pytorch-bot bot commented Nov 4, 2022 •

edited

jerryzh168 Nov 4, 2022 •

edited

andrewor14 Nov 4, 2022

andrewor14 Nov 8, 2022

jerryzh168 left a comment •

edited

andrewor14 commented Nov 4, 2022

andrewor14 commented Nov 4, 2022

pytorchmergebot commented Nov 4, 2022

[Quant] Respect non_leaf_module_list for activation modules #88498

[Quant] Respect non_leaf_module_list for activation modules #88498

Conversation

andrewor14 commented Nov 4, 2022 • edited by pytorch-bot bot

pytorch-bot bot commented Nov 4, 2022 • edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/88498

✅ No Failures

jerryzh168 Nov 4, 2022 • edited

Choose a reason for hiding this comment

andrewor14 Nov 4, 2022

Choose a reason for hiding this comment

andrewor14 Nov 8, 2022

Choose a reason for hiding this comment

jerryzh168 left a comment • edited

Choose a reason for hiding this comment

andrewor14 commented Nov 4, 2022

andrewor14 commented Nov 4, 2022

pytorchmergebot commented Nov 4, 2022

Merge started

andrewor14 commented Nov 4, 2022 •

edited by pytorch-bot bot

pytorch-bot bot commented Nov 4, 2022 •

edited

jerryzh168 Nov 4, 2022 •

edited

jerryzh168 left a comment •

edited