[quant][pyper] Support aten::embedding_bag quantization in graph mode #43989

supriyar · 2020-09-01T21:25:39Z

Stack from ghstack:

[quant][pyper] Support quantization of ops in fork-wait subgraph #44048 [quant][pyper] Support quantization of ops in fork-wait subgraph
[quant][pyper] make embedding_bag quantization static #44008 [quant][pyper] make embedding_bag quantization static
[quant][pyper] Support aten::embedding_bag quantization in graph mode #43989 [quant][pyper] Support aten::embedding_bag quantization in graph mode

Summary:
When we trace the model it produces aten::embedding_bag node in the graph,
Add necessary passes in graph mode to help support quantizing it as well

Test Plan:
python test/test_quantization.py TestQuantizeDynamicJitOps.test_embedding_bag

Reviewers:

Subscribers:

Tasks:

Tags:

Differential Revision: D23460485

Summary: When we trace the model it produces aten::embedding_bag node in the graph, Add necessary passes in graph mode to help support quantizing it as well Test Plan: python test/test_quantization.py TestQuantizeDynamicJitOps.test_embedding_bag Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]

dr-ci · 2020-09-01T21:37:33Z

💊 CI failures summary and remediations

As of commit 119b273 (more details on the Dr. CI page):

2/3 failures possibly* introduced in this PR
- 2/2 non-CircleCI failure(s)
1/3 broken upstream at merge base 6474057 on Sep 04 from 3:37am to 11:28am PDT (13 commits; f8f35fd - 0c2bc4f)

🚧 1 fixed upstream failure:

These were probably caused by upstream breakages that were already fixed.

Please rebase on the viable/strict branch (expand for instructions)

If your commit is newer than viable/strict, you can try basing on an older, stable commit:

git fetch https://github.com/pytorch/pytorch viable/strict
git rebase --onto FETCH_HEAD $(git merge-base origin/master HEAD)

If your commit is older than viable/strict:

git fetch https://github.com/pytorch/pytorch viable/strict
git rebase FETCH_HEAD

Check out the recency history of this "viable master" tracking branch.

pytorch_xla_linux_bionic_py3_6_clang9_test on Sep 04 from 3:37am to 11:28am PDT (13 commits; f8f35fd - 0c2bc4f)
- 🔁 rerun

Extra GitHub checks: 1 failed

Failed: GitHub Actions - clang-tidy

ci.pytorch.org: 1 failed

Failed: pr/pytorch-linux-bionic-rocm3.7-py3.6

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 18 times.

… graph mode" Summary: When we trace the model it produces aten::embedding_bag node in the graph, Add necessary passes in graph mode to help support quantizing it as well Test Plan: python test/test_quantization.py TestQuantizeDynamicJitOps.test_embedding_bag Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D23460485](https://our.internmc.facebook.com/intern/diff/D23460485) [ghstack-poisoned]

Summary: When we trace the model it produces aten::embedding_bag node in the graph, Add necessary passes in graph mode to help support quantizing it as well Test Plan: python test/test_quantization.py TestQuantizeDynamicJitOps.test_embedding_bag Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: 1686d1ba8c2eac1cc750b771f4fd542a473c2bd3 Pull Request resolved: #43989

… graph mode" Summary: When we trace the model it produces aten::embedding_bag node in the graph, Add necessary passes in graph mode to help support quantizing it as well Test Plan: python test/test_quantization.py TestQuantizeDynamicJitOps.test_embedding_bag Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D23460485](https://our.internmc.facebook.com/intern/diff/D23460485) [ghstack-poisoned]

codecov · 2020-09-02T22:23:22Z

Codecov Report

❗ No coverage uploaded for pull request base (gh/supriyar/170/base@6474057). Click here to learn what that means.
The diff coverage is n/a.

@@                   Coverage Diff                   @@
##             gh/supriyar/170/base   #43989   +/-   ##
=======================================================
  Coverage                        ?   69.27%           
=======================================================
  Files                           ?      381           
  Lines                           ?    47239           
  Branches                        ?        0           
=======================================================
  Hits                            ?    32724           
  Misses                          ?    14515           
  Partials                        ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6474057...119b273. Read the comment docs.

vkuzo

lg, feel free to wait for @jerryzh168 if a deeper review is needed

vkuzo · 2020-09-03T18:19:20Z

torch/csrc/jit/passes/quantization/helper.cpp

@@ -259,10 +260,16 @@ bool matchArgPattern(
 bool isWeight(Value* v) {
  bool result = matchArgPattern(
      v,
-      AtenFuncArgs(
-          {{"conv1d", 1}, {"conv2d", 1}, {"conv3d", 1}, {"linear", 1}}),
+      // ate::embedding_bag(%weight, %input, %offsets, %scale_grad_by_freq,


nit: aten?

raghuramank100 · 2020-09-03T18:28:37Z

test/quantization/test_quantize_jit.py

+            from torch.quantization import QConfigDynamic, PlaceholderObserver
+            int4_dynamic_qconfig = QConfigDynamic(activation=PlaceholderObserver.with_args(dtype=torch.float,
+                                                                                           custom_op_name="embedding_bag_4bit"),
+                                                  weight=PlaceholderObserver.with_args(custom_op_name="embedding_bag_4bit"))


Why do we have a placeholder observer for weights?. My understanding is that we can use real observers for 8 bit but not for 4 bit currently. Is that correct?

We currently use real observers and torchbind classes for eager mode 8-bit embedding quant currently. For graph mode we implemented this initially using the custom prepack ops for PyPer for 8bit and 4bit, to be consistent with C2.
Going forward, in fx we can implement embeddingbag quantization using observers. I feel it is a bit of an overkill to update this code to use observers for 8-bit and placeholder observers for 4-bit. Let me know your thoughts.

… graph mode" Summary: When we trace the model it produces aten::embedding_bag node in the graph, Add necessary passes in graph mode to help support quantizing it as well Test Plan: python test/test_quantization.py TestQuantizeDynamicJitOps.test_embedding_bag Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D23460485](https://our.internmc.facebook.com/intern/diff/D23460485) [ghstack-poisoned]

facebook-github-bot · 2020-09-05T20:13:31Z

This pull request has been merged in a0ae416.

supriyar requested a review from apaszke as a code owner September 1, 2020 21:25

facebook-github-bot added the oncall: jit Add this issue/PR to JIT oncall triage queue label Sep 1, 2020

supriyar changed the title ~~[quant] Support aten::embedding_bag quantization in graph mode~~ [quant][pyper] Support aten::embedding_bag quantization in graph mode Sep 1, 2020

This was referenced Sep 2, 2020

[quant][pyper] make embedding_bag quantization static #44008

Closed

[quant][pyper] Support quantization of ops in fork-wait subgraph #44048

Closed

supriyar requested review from jerryzh168, vkuzo and raghuramank100 September 2, 2020 20:48

vkuzo approved these changes Sep 3, 2020

View reviewed changes

raghuramank100 reviewed Sep 3, 2020

View reviewed changes

facebook-github-bot closed this in a0ae416 Sep 5, 2020

facebook-github-bot added the merged label Sep 5, 2020

facebook-github-bot deleted the gh/supriyar/170/head branch September 9, 2020 14:18

mruberry added the Merged label Oct 28, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[quant][pyper] Support aten::embedding_bag quantization in graph mode #43989

[quant][pyper] Support aten::embedding_bag quantization in graph mode #43989

supriyar commented Sep 1, 2020 •

edited

dr-ci bot commented Sep 1, 2020 •

edited

codecov bot commented Sep 2, 2020 •

edited

vkuzo left a comment

vkuzo Sep 3, 2020

raghuramank100 Sep 3, 2020 •

edited

supriyar Sep 3, 2020

facebook-github-bot commented Sep 5, 2020

[quant][pyper] Support aten::embedding_bag quantization in graph mode #43989

[quant][pyper] Support aten::embedding_bag quantization in graph mode #43989

Conversation

supriyar commented Sep 1, 2020 • edited

dr-ci bot commented Sep 1, 2020 • edited

💊 CI failures summary and remediations

🚧 1 fixed upstream failure:

Extra GitHub checks: 1 failed

ci.pytorch.org: 1 failed

codecov bot commented Sep 2, 2020 • edited

Codecov Report

vkuzo left a comment

Choose a reason for hiding this comment

vkuzo Sep 3, 2020

Choose a reason for hiding this comment

raghuramank100 Sep 3, 2020 • edited

Choose a reason for hiding this comment

supriyar Sep 3, 2020

Choose a reason for hiding this comment

facebook-github-bot commented Sep 5, 2020

supriyar commented Sep 1, 2020 •

edited

dr-ci bot commented Sep 1, 2020 •

edited

codecov bot commented Sep 2, 2020 •

edited

raghuramank100 Sep 3, 2020 •

edited