[Gradient Compression] divide by world size before all_reduce to avoid overflow

[wayi1]

Stack from ghstack:

• [Gradient Compression] divide by world size before all_reduce to avoid overflow #57410 [Gradient Compression] divide by world size before all_reduce to avoid overflow

FP16 gradient compression may run into 'inf' issue. switching to division before allreduce can avoid this problem.

Differential Revision: D28128628

[facebook-github-bot]

💊 CI failures summary and remediations

As of commit eaa2897 (more details on the Dr. CI page):

---

• 2/3 failures possibly* introduced in this PR
  • 1/2 non-scanned failure(s)
• 1/3 tentatively recognized as flaky ❄️
  • Click here to rerun these jobs

---

🕵️ 1 new failure recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

pytorch_linux_xenial_cuda11_1_cudnn8_py3_gcc7_test1 (1/1)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

May 01 05:48:50 AssertionError: False is not tr...lowed difference with rtol=0 and atol=0 is only 0!

May 01 05:48:50 ----------------------------------------------------------------------
May 01 05:48:50 Traceback (most recent call last):
May 01 05:48:50   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 374, in wrapper
May 01 05:48:50     self._join_processes(fn)
May 01 05:48:50   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 566, in _join_processes
May 01 05:48:50     self._check_return_codes(elapsed_time)
May 01 05:48:50   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_distributed.py", line 619, in _check_return_codes
May 01 05:48:50     i, first_process.exitcode, p.exitcode
May 01 05:48:50   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_utils.py", line 1397, in assertEqual
May 01 05:48:50     super().assertTrue(result, msg=self._get_assert_msg(msg, debug_msg=debug_msg))
May 01 05:48:50 AssertionError: False is not true : Scalars failed to compare as equal! Comparing 1 and -11 gives a difference of 12, but the allowed difference with rtol=0 and atol=0 is only 0!
May 01 05:48:50 Expect process 1 exit code to match Process 0 exit code of -11, but got 1
May 01 05:48:50 
May 01 05:48:50 ----------------------------------------------------------------------
May 01 05:48:50 Ran 199 tests in 213.489s
May 01 05:48:50 
May 01 05:48:50 FAILED (failures=1, skipped=119)
May 01 05:48:50 
May 01 05:48:50 Generating XML reports...
May 01 05:48:50 Generated XML report: test-reports/dist-nccl/distributed.test_distributed_fork/TEST-TestDistBackendWithFork-20210501054517.xml
May 01 05:48:50 Traceback (most recent call last):

---

❄️ 1 failure tentatively classified as flaky

but reruns have not yet been triggered to confirm:

pytorch_linux_xenial_py3_clang5_asan_test2 (1/1)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun) ❄️

May 01 08:17:50 unknown file: Failure

May 01 08:17:50 [       OK ] Kernel.Softmax2D (26 ms)
May 01 08:17:50 [ RUN      ] Kernel.Softmax3D
May 01 08:17:50 [       OK ] Kernel.Softmax3D (142 ms)
May 01 08:17:50 [ RUN      ] Kernel.Softmax4D
May 01 08:17:50 [       OK ] Kernel.Softmax4D (192 ms)
May 01 08:17:50 [ RUN      ] Kernel.ConstantTensors
May 01 08:17:50 [       OK ] Kernel.ConstantTensors (19 ms)
May 01 08:17:50 [ RUN      ] Kernel.ConstantTensorsNonContiguous
May 01 08:17:50 [       OK ] Kernel.ConstantTensorsNonContiguous (18 ms)
May 01 08:17:50 [ RUN      ] Kernel.RunFast
May 01 08:17:50 unknown file: Failure
May 01 08:17:50 C++ exception with description "SimpleIREvaluator::call_raw is not implemented yet" thrown in the test body.
May 01 08:17:50 [  FAILED  ] Kernel.RunFast (5 ms)
May 01 08:17:50 [----------] 14 tests from Kernel (616 ms total)
May 01 08:17:50 
May 01 08:17:50 [----------] 140 tests from LoopNest
May 01 08:17:50 [ RUN      ] LoopNest.ExprSimple01
May 01 08:17:50 [       OK ] LoopNest.ExprSimple01 (1 ms)
May 01 08:17:50 [ RUN      ] LoopNest.ExprLower01
May 01 08:17:50 [       OK ] LoopNest.ExprLower01 (0 ms)
May 01 08:17:50 [ RUN      ] LoopNest.ExprSimple02

---

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

[rohan-varma]

LGTM!

[facebook-github-bot]

This pull request has been merged in c07babb.