ns for fx: support comparing fp32 vs fp32_prepared/int8, except shadowed #61129
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Summary: Adds support the comparing fp32 model (without quantization) to a fp32 model prepared with quantization. The main missing feature was handling conv-bn fusion, since this fusion for PTQ happens outside of quantization patterns. Adds testing for this case for comparing weights and comparing activations Adds a TODO for also handling this for shadow activations, we need to first stop removing observers in graph passes before we can add this support, will be in a future PR. Test Plan: ``` python test/test_quantization.py TestFXGraphMatcherModels.test_mobilenet_v2 python test/test_quantization.py TestFXGraphMatcherModels.test_mobilenet_v2_qat python test/test_quantization.py TestFXNumericSuiteCoreAPIsModels.test_compare_activations_conv ``` Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]
💊 CI failures summary and remediationsAs of commit b68e341 (more details on the Dr. CI page and at hud.pytorch.org/pr/61129): 💚 💚 Looks good so far! There are no failures yet. 💚 💚 Preview docs built from this PR This comment was automatically generated by Dr. CI (expand for details).Follow this link to opt-out of these comments for your Pull Requests.Please report bugs/suggestions to the (internal) Dr. CI Users group. |
vkuzo
added a commit
that referenced
this pull request
Jul 1, 2021
Summary: Adds support the comparing fp32 model (without quantization) to a fp32 model prepared with quantization. The main missing feature was handling conv-bn fusion, since this fusion for PTQ happens outside of quantization patterns. Adds testing for this case for comparing weights and comparing activations Adds a TODO for also handling this for shadow activations, we need to first stop removing observers in graph passes before we can add this support, will be in a future PR. Test Plan: ``` python test/test_quantization.py TestFXGraphMatcherModels.test_mobilenet_v2 python test/test_quantization.py TestFXGraphMatcherModels.test_mobilenet_v2_qat python test/test_quantization.py TestFXNumericSuiteCoreAPIsModels.test_compare_activations_conv ``` Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: 792bbd5 Pull Request resolved: #61129
|
@vkuzo has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
… shadowed" Summary: Adds support the comparing fp32 model (without quantization) to a fp32 model prepared with quantization. The main missing feature was handling conv-bn fusion, since this fusion for PTQ happens outside of quantization patterns. Adds testing for this case for comparing weights and comparing activations Adds a TODO for also handling this for shadow activations, we need to first stop removing observers in graph passes before we can add this support, will be in a future PR. Test Plan: ``` python test/test_quantization.py TestFXGraphMatcherModels.test_mobilenet_v2 python test/test_quantization.py TestFXGraphMatcherModels.test_mobilenet_v2_qat python test/test_quantization.py TestFXNumericSuiteCoreAPIsModels.test_compare_activations_conv ``` Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D29520009](https://our.internmc.facebook.com/intern/diff/D29520009) [ghstack-poisoned]
This was referenced Jul 7, 2021
|
@vkuzo has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
| self.assert_ns_compare_dict_valid(results) | ||
| # test both m vs mp and mp vs mq | ||
| for m1, m2 in ((m, mp), (mp, mq)): | ||
| results = extract_weights_fun('a', mp, 'b', mq) |
There was a problem hiding this comment.
Should this be m1 and m2?
There was a problem hiding this comment.
yes, great catch, will fix that
There was a problem hiding this comment.
yes, great catch, will fix that
raghuramank100
approved these changes
Jul 8, 2021
hx89
approved these changes
Jul 9, 2021
there was a use case discussed during design review to compare fp32 to quantized. This PR implements this functionality (once you can compare A to B and B to C, you can compare A to C). Let me update the summary to say this. |
vkuzo
changed the title
…xcept shadowed" Summary: Adds support the comparing fp32 model (without quantization) to a fp32 model prepared with quantization or a quantized model. The main missing feature was handling conv-bn fusion, since this fusion for PTQ happens outside of quantization patterns. Adds testing for this case for comparing weights and comparing activations Adds a TODO for also handling this for shadow activations, we need to first stop removing observers in graph passes before we can add this support, will be in a future PR. Test Plan: ``` python test/test_quantization.py TestFXGraphMatcherModels.test_mobilenet_v2 python test/test_quantization.py TestFXGraphMatcherModels.test_mobilenet_v2_qat python test/test_quantization.py TestFXNumericSuiteCoreAPIsModels.test_compare_activations_conv ``` Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D29520009](https://our.internmc.facebook.com/intern/diff/D29520009) [ghstack-poisoned]
|
@vkuzo has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
…xcept shadowed" Summary: Adds support the comparing fp32 model (without quantization) to a fp32 model prepared with quantization or a quantized model. The main missing feature was handling conv-bn fusion, since this fusion for PTQ happens outside of quantization patterns. Adds testing for this case for comparing weights and comparing activations Adds a TODO for also handling this for shadow activations, we need to first stop removing observers in graph passes before we can add this support, will be in a future PR. Test Plan: ``` python test/test_quantization.py TestFXGraphMatcherModels.test_mobilenet_v2 python test/test_quantization.py TestFXGraphMatcherModels.test_mobilenet_v2_qat python test/test_quantization.py TestFXNumericSuiteCoreAPIsModels.test_compare_activations_conv ``` Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D29520009](https://our.internmc.facebook.com/intern/diff/D29520009) [ghstack-poisoned]
|
@vkuzo has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
|
This pull request has been merged in a70505c. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Stack from ghstack:
Summary:
Adds support the comparing fp32 model (without quantization) to a
fp32 model prepared with quantization or a quantized model. The main missing feature was
handling conv-bn fusion, since this fusion for PTQ happens outside
of quantization patterns.
Adds testing for this case for comparing weights and comparing
activations
Adds a TODO for also handling this for shadow activations, we need to
first stop removing observers in graph passes before we can add
this support, will be in a future PR.
Test Plan:
Reviewers:
Subscribers:
Tasks:
Tags:
Differential Revision: D29520009