[nnc] Test cases for uneven split + reorder

[bertmaher]

Stack from ghstack:

• [nnc] Test cases for uneven split + reorder #53091 [nnc] Test cases for uneven split + reorder

Split with tail followed by reorder causes a segfault in NNC
Split with mask followed by reorder generates invalid code that writes out of
bounds

Differential Revision: D26746254

[facebook-github-bot]

💊 CI failures summary and remediations

As of commit 55f5d95 (more details on the Dr. CI page):

---

• 1/2 failures introduced in this PR
• 1/2 broken upstream at merge base d382693 from Mar 01 until Mar 02

---

🕵️ 1 new failure recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

pytorch_linux_xenial_py3_6_gcc5_4_test (1/1)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Mar 02 20:52:47 [E request_callback_no_python.cpp:656] Received error while processing request type 258: RuntimeError: Can not pickle torch.futures.Future

Mar 02 20:52:47 At:
Mar 02 20:52:47   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(122): serialize
Mar 02 20:52:47   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(175): serialize
Mar 02 20:52:47 
Mar 02 20:52:47 [E request_callback_no_python.cpp:656] Received error while processing request type 258: RuntimeError: Can not pickle torch.futures.Future
Mar 02 20:52:47 
Mar 02 20:52:47 At:
Mar 02 20:52:47   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(122): serialize
Mar 02 20:52:47   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(175): serialize
Mar 02 20:52:47 
Mar 02 20:52:47 [E request_callback_no_python.cpp:656] Received error while processing request type 258: RuntimeError: Can not pickle torch.futures.Future
Mar 02 20:52:47 
Mar 02 20:52:47 At:
Mar 02 20:52:47   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(122): serialize
Mar 02 20:52:47   /opt/conda/lib/python3.6/site-packages/torch/distributed/rpc/internal.py(175): serialize
Mar 02 20:52:47 
Mar 02 20:52:47 ok (2.045s)
Mar 02 20:52:49   test_return_future_remote (__main__.ProcessGroupRpcTestWithSpawn) ... RPC was initialized with the PROCESS_GROUP backend which is deprecated and slated to be removed and superseded by the TENSORPIPE backend. It is recommended to migrate to the TENSORPIPE backend.
Mar 02 20:52:49 RPC was initialized with the PROCESS_GROUP backend which is deprecated and slated to be removed and superseded by the TENSORPIPE backend. It is recommended to migrate to the TENSORPIPE backend.
Mar 02 20:52:49 RPC was initialized with the PROCESS_GROUP backend which is deprecated and slated to be removed and superseded by the TENSORPIPE backend. It is recommended to migrate to the TENSORPIPE backend.
Mar 02 20:52:49 RPC was initialized with the PROCESS_GROUP backend which is deprecated and slated to be removed and superseded by the TENSORPIPE backend. It is recommended to migrate to the TENSORPIPE backend.

---

🚧 1 fixed upstream failure:

These were probably caused by upstream breakages that were already fixed.

Please rebase on the viable/strict branch (expand for instructions)

If your commit is older than viable/strict, run these commands:

git fetch https://github.com/pytorch/pytorch viable/strict
git rebase FETCH_HEAD

• pytorch_linux_xenial_cuda10_2_cudnn7_py3_gcc7_test1 from Mar 01 until Mar 02 (a586c02 - 931100f)
  • 🔁 rerun

---

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions to the (internal) Dr. CI Users group.

[bertmaher]

Obviously since these cause segfaults I won't land until those are cleared up ;-)

[navahgar]

Obviously since these cause segfaults I won't land until those are cleared up ;-)

How about disabling and landing them? So that it is easier to test and enable on fix.

[asuhan]

Up to you and the difficulty of the fix whether to land those disabled or wait, but obviously we'll need to re-review if we wait.

Out of curiosity, how did you find these, convolutions with shapes which need epilogue for vectorization?

[bertmaher]

Up to you and the difficulty of the fix whether to land those disabled or wait, but obviously we'll need to re-review if we wait.

Out of curiosity, how did you find these, convolutions with shapes which need epilogue for vectorization?

Yeah I’ll go ahead and disable and land so we can parallelization the fixes.

I found them when trying to vectorize a 56x56 convolution by a factor of 16; I was hoping either to get an 8-wide epilogue or a mask, and neither worked :-)

[facebook-github-bot]

This pull request has been merged in 565d823.