[Quant] onednn backend switch to ideep new api without affacting performance #91056
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ormance [ghstack-poisoned]
Xia-Weiwen
requested review from
jerryzh168,
z-a-f,
salilsdesai,
kimishpatel,
digantdesai and
jianyuh
as code owners
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/91056
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit e166522: This comment was automatically generated by Dr. CI and updates every 15 minutes. |
Xia-Weiwen
added a commit
that referenced
this pull request
Dec 17, 2022
…ormance ghstack-source-id: 0df32fbbe7639bc0f5d4335715213502eb0ab1bd Pull Request resolved: #91056
github-actions
bot
added
the
module: cpu
…acting performance" > Reopen of #90354 **Summary** Onednn quantization backend switch to new API in `third_party/ideep`. - `struct forward_params` for conv/deconv are changed. Modify primitive cache accordingly. - Use new versions of `prepare` and `compute` API. Fp32 and int8 paths separated. The old ones will be deprecated. - Now `ideep::tensor::reorder_if_differ_in` supports block-to-block reorder. Use it instead of defining a util function `onednn_utils::try_reorder`. - For new API of transposed convolution, we can use a flag to keep weight desc align with oneDNN thus needless to transpose it explicitly in PyTorch. - Use `is_channels_last` flag to specify layout of src/dst when querying expected weight desc. It won't impact correctness. Performance should be unaffected or slightly better. FBGEMM and QNNPACK backends are not affected. Performance results are given below. 1. End-to-end performance of static quantized models (from torchvision) (throughput: fps, higher is better)  2. Op benchmark of dynamic quantized linear (Latency: ms, lower is better)  Test method & env: - Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz - Run multi-instances on a single node. Use one core for each instance. - Use Jemalloc and Intel OpenMP **Test plan** python test/test_quantization.py cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]
Xia-Weiwen
added a commit
that referenced
this pull request
Dec 18, 2022
…ormance ghstack-source-id: c583ee8ab681ee8a1bb95609fe66c1fac742bbb8 Pull Request resolved: #91056
Xia-Weiwen
added
intel
jgong5
approved these changes
Dec 19, 2022
Xia-Weiwen
added a commit
to Xia-Weiwen/pytorch
that referenced
this pull request
Jan 11, 2023
…ormance ghstack-source-id: c583ee8ab681ee8a1bb95609fe66c1fac742bbb8 Pull Request resolved: pytorch#91056
Xia-Weiwen
added a commit
to Xia-Weiwen/pytorch
that referenced
this pull request
Jan 16, 2023
…ormance ghstack-source-id: c583ee8ab681ee8a1bb95609fe66c1fac742bbb8 Pull Request resolved: pytorch#91056
…acting performance" > Reopen of #90354 **Summary** Onednn quantization backend switch to new API in `third_party/ideep`. - `struct forward_params` for conv/deconv are changed. Modify primitive cache accordingly. - Use new versions of `prepare` and `compute` API. Fp32 and int8 paths separated. The old ones will be deprecated. - Now `ideep::tensor::reorder_if_differ_in` supports block-to-block reorder. Use it instead of defining a util function `onednn_utils::try_reorder`. - For new API of transposed convolution, we can use a flag to keep weight desc align with oneDNN thus needless to transpose it explicitly in PyTorch. - Use `is_channels_last` flag to specify layout of src/dst when querying expected weight desc. It won't impact correctness. Performance should be unaffected or slightly better. FBGEMM and QNNPACK backends are not affected. Performance results are given below. 1. End-to-end performance of static quantized models (from torchvision) (throughput: fps, higher is better)  2. Op benchmark of dynamic quantized linear (Latency: ms, lower is better)  Test method & env: - Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz - Run multi-instances on a single node. Use one core for each instance. - Use Jemalloc and Intel OpenMP **Test plan** python test/test_quantization.py cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]
Xia-Weiwen
added a commit
that referenced
this pull request
Jan 16, 2023
…ormance ghstack-source-id: 5151171ed10b77c817a260c6d816a17d75f6e1f6 Pull Request resolved: #91056
…acting performance" > Reopen of #90354 **Summary** Onednn quantization backend switch to new API in `third_party/ideep`. - `struct forward_params` for conv/deconv are changed. Modify primitive cache accordingly. - Use new versions of `prepare` and `compute` API. Fp32 and int8 paths separated. The old ones will be deprecated. - Now `ideep::tensor::reorder_if_differ_in` supports block-to-block reorder. Use it instead of defining a util function `onednn_utils::try_reorder`. - For new API of transposed convolution, we can use a flag to keep weight desc align with oneDNN thus needless to transpose it explicitly in PyTorch. - Use `is_channels_last` flag to specify layout of src/dst when querying expected weight desc. It won't impact correctness. Performance should be unaffected or slightly better. FBGEMM and QNNPACK backends are not affected. Performance results are given below. 1. End-to-end performance of static quantized models (from torchvision) (throughput: fps, higher is better)  2. Op benchmark of dynamic quantized linear (Latency: ms, lower is better)  Test method & env: - Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz - Run multi-instances on a single node. Use one core for each instance. - Use Jemalloc and Intel OpenMP **Test plan** python test/test_quantization.py cc jgong5 mingfeima XiaobingSuper sanchitintel ashokei jingxu10 [ghstack-poisoned]
Xia-Weiwen
added a commit
that referenced
this pull request
Jan 16, 2023
…ormance ghstack-source-id: 70704c08e9fb76d91951aab332fba94a128c1b15 Pull Request resolved: #91056
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
|
Hmm, this seems to be using new iDeep APIs, why was it landed before internal update? |
Hi @malfet. Since the double checkout issue has been solved by #92239, I thought it would be OK to land this. If it is breaking something, please go ahead to revert it. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Stack from ghstack (oldest at bottom):
Summary
Onednn quantization backend switch to new API in
third_party/ideep.struct forward_paramsfor conv/deconv are changed. Modify primitive cache accordingly.prepareandcomputeAPI. Fp32 and int8 paths separated. The old ones will be deprecated.ideep::tensor::reorder_if_differ_insupports block-to-block reorder. Use it instead of defining a util functiononednn_utils::try_reorder.is_channels_lastflag to specify layout of src/dst when querying expected weight desc.It won't impact correctness. Performance should be unaffected or slightly better.
FBGEMM and QNNPACK backends are not affected.
Performance results are given below.
End-to-end performance of static quantized models (from torchvision)
(throughput: fps, higher is better)
Op benchmark of dynamic quantized linear
(Latency: ms, lower is better)
Test method & env:
Test plan
python test/test_quantization.py
cc @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10