-
Notifications
You must be signed in to change notification settings - Fork 751
Creating Paramaterized Test For Quantizers For Easier Testing #16098
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/16098
Note: Links to docs will display an error until the docs builds have been completed. ❌ 2 New Failures, 1 Unrelated FailureAs of commit b6c6203 with merge base 56e131b ( NEW FAILURES - The following jobs have failed:
UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
This PR needs a
|
…h#16098) Summary: We consolidate the two tests we created into a single testing function using parameterization. This will make testing future Quantizers much easier, and there will be a lot less code duplication. Reviewed By: hsharma35 Differential Revision: D88054917
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR consolidates quantizer annotation tests into a single parameterized test function, reducing code duplication and making it easier to add new quantizer tests. The refactoring uses the parameterized library to test both matmul and linear operations with their respective 16-bit activation quantizers.
Key changes:
- Introduced parameterized testing approach using
@parameterized.expanddecorator - Created reusable graph builder helper methods (
_build_matmul_graph,_build_linear_graph) - Added new imports for GraphBuilder, parameterized library, and quantization annotation types
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| backends/cadence/aot/tests/test_quantizer_ops.py | Added parameterized test infrastructure with QuantizerAnnotationTest class containing graph builder helpers and consolidated test method for verifying quantizer annotations on matmul and linear operations |
| backends/cadence/aot/TARGETS | Added dependencies for parameterized testing library, graph_builder module, pass_base, and torchao to support the new test infrastructure |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| name: str, | ||
| graph_builder_fn: GraphBuilderFn, | ||
| quantizer: CadenceQuantizer, | ||
| target: OpOverload, |
Copilot
AI
Dec 5, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The target parameter is unused in the test function. Consider removing it from the parameter list (lines 94 and 103) if it's not needed, or use it to validate that op_node.target matches the expected target operation for additional test robustness.
| actual_specs = [ | ||
| annotation.input_qspec_map[op_node.args[i]] | ||
| for i in range(len(expected_input_qspecs)) | ||
| ] |
Copilot
AI
Dec 5, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Potential KeyError: accessing annotation.input_qspec_map[op_node.args[i]] could fail if op_node.args[i] is not in the map. Consider adding a check or using .get() with a clear error message to make debugging easier if the key is missing.
| actual_specs = [ | |
| annotation.input_qspec_map[op_node.args[i]] | |
| for i in range(len(expected_input_qspecs)) | |
| ] | |
| actual_specs = [] | |
| for i in range(len(expected_input_qspecs)): | |
| key = op_node.args[i] | |
| if key not in annotation.input_qspec_map: | |
| raise KeyError( | |
| f"Key {key!r} not found in input_qspec_map. " | |
| f"Available keys: {list(annotation.input_qspec_map.keys())}" | |
| ) | |
| actual_specs.append(annotation.input_qspec_map[key]) |
…h#16098) Summary: We consolidate the two tests we created into a single testing function using parameterization. This will make testing future Quantizers much easier, and there will be a lot less code duplication. Reviewed By: hsharma35, zonglinpeng Differential Revision: D88054917
c924c92 to
904280a
Compare
…h#16098) Summary: We consolidate the two tests we created into a single testing function using parameterization. This will make testing future Quantizers much easier, and there will be a lot less code duplication. Reviewed By: hsharma35, zonglinpeng Differential Revision: D88054917
904280a to
381f9d0
Compare
…h#16098) Summary: We consolidate the two tests we created into a single testing function using parameterization. This will make testing future Quantizers much easier, and there will be a lot less code duplication. Reviewed By: hsharma35, zonglinpeng Differential Revision: D88054917
…h#16098) Summary: We consolidate the two tests we created into a single testing function using parameterization. This will make testing future Quantizers much easier, and there will be a lot less code duplication. Reviewed By: hsharma35, zonglinpeng Differential Revision: D88054917
…h#16098) Summary: Pull Request resolved: pytorch#16098 We consolidate the two tests we created into a single testing function using parameterization. This will make testing future Quantizers much easier, and there will be a lot less code duplication. Reviewed By: hsharma35, zonglinpeng Differential Revision: D88054917
381f9d0 to
c2bdbdd
Compare
…h#16098) Summary: We consolidate the two tests we created into a single testing function using parameterization. This will make testing future Quantizers much easier, and there will be a lot less code duplication. Reviewed By: hsharma35, zonglinpeng Differential Revision: D88054917
c2bdbdd to
2c83f7b
Compare
…h#16098) Summary: We consolidate the two tests we created into a single testing function using parameterization. This will make testing future Quantizers much easier, and there will be a lot less code duplication. Reviewed By: hsharma35, zonglinpeng Differential Revision: D88054917
…h#16098) Summary: We consolidate the two tests we created into a single testing function using parameterization. This will make testing future Quantizers much easier, and there will be a lot less code duplication. Reviewed By: hsharma35, zonglinpeng Differential Revision: D88054917
2c83f7b to
f395af3
Compare
…h#16098) Summary: We consolidate the two tests we created into a single testing function using parameterization. This will make testing future Quantizers much easier, and there will be a lot less code duplication. Reviewed By: hsharma35, zonglinpeng Differential Revision: D88054917
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| name: str, | ||
| graph_builder_fn: GraphBuilderFn, | ||
| quantizer: CadenceQuantizer, | ||
| target: OpOverload, |
Copilot
AI
Dec 8, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The target parameter is declared but never used in the test function. Consider removing it if it's not needed, or add an assertion to verify that the operation node matches the expected target.
…h#16098) Summary: We consolidate the two tests we created into a single testing function using parameterization. This will make testing future Quantizers much easier, and there will be a lot less code duplication. Reviewed By: hsharma35, zonglinpeng Differential Revision: D88054917
…h#16098) Summary: We consolidate the two tests we created into a single testing function using parameterization. This will make testing future Quantizers much easier, and there will be a lot less code duplication. Reviewed By: hsharma35, zonglinpeng Differential Revision: D88054917
f395af3 to
095eee9
Compare
…h#16098) Summary: We consolidate the two tests we created into a single testing function using parameterization. This will make testing future Quantizers much easier, and there will be a lot less code duplication. Reviewed By: hsharma35, zonglinpeng Differential Revision: D88054917
…6089) Summary: We test the quantizer we added in D87996796 correctly annotates the graph. We use the graph builder to build the graph with metadata(that's needed for quantizer.annotate to recognize the nodes), and we ensure that the quantization params are as expected. Reviewed By: zonglinpeng, hsharma35 Differential Revision: D88053808
…6097) Summary: We test the CadenceWith16BitLinearActivationQuantizer. We use the graph builder to build the graph with metadata(that's needed for quantizer.annotate to recognize the nodes), and we ensure that the quantization params are as expected. Reviewed By: zonglinpeng, hsharma35 Differential Revision: D88054651
…h#16098) Summary: We consolidate the two tests we created into a single testing function using parameterization. This will make testing future Quantizers much easier, and there will be a lot less code duplication. Reviewed By: hsharma35, zonglinpeng Differential Revision: D88054917
095eee9 to
b6c6203
Compare
…h#16098) Summary: We consolidate the two tests we created into a single testing function using parameterization. This will make testing future Quantizers much easier, and there will be a lot less code duplication. Reviewed By: hsharma35, zonglinpeng Differential Revision: D88054917
Summary:
We consolidate the two tests we created into a single testing function using parameterization.
This will make testing future Quantizers much easier, and there will be a lot less code duplication.
Reviewed By: hsharma35
Differential Revision: D88054917