[NCCL][Test Only] no change

[mingzhe09088]

Stack from ghstack:

• [NCCL][Test Only] no change #45320 [NCCL][Test Only] no change
• [NCCL][Test Only] test send/recv on OSS #45140 [NCCL][Test Only] test send/recv on OSS
• [NCCL] create NCCL communicator for send/recv on demand #44922 [NCCL] create NCCL communicator for send/recv on demand
• [NCCL] Support NCCL Send/Recv #44921 [NCCL] Support NCCL Send/Recv

Differential Revision: D23922690

[dr-ci]

💊 CI failures summary and remediations

As of commit 74424f1 (more details on the Dr. CI page):

---

• 2/2 failures introduced in this PR

---

🕵️ 2 new failures recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

pytorch_linux_xenial_py3_clang7_onnx_build (1/2)

Step: "Build" (full log | diagnosis details | 🔁 rerun)

Sep 26 00:14:32 caused by: Connection refused (os error 111)

Sep 26 00:14:32 ++++ extract_trap_cmd 
Sep 26 00:14:32 ++++ printf '%s\n' '' 
Sep 26 00:14:32 +++ printf '%s\n' cleanup 
Sep 26 00:14:32 ++ trap -- ' 
Sep 26 00:14:32 cleanup' EXIT 
Sep 26 00:14:32 ++ [[ pytorch-linux-xenial-py3-clang7-onnx-build != *pytorch-win-* ]] 
Sep 26 00:14:32 ++ which sccache 
Sep 26 00:14:32 ++ sccache --stop-server 
Sep 26 00:14:32 Stopping sccache server... 
Sep 26 00:14:32 error: couldn't connect to server 
Sep 26 00:14:32 caused by: Connection refused (os error 111) 
Sep 26 00:14:32 ++ true 
Sep 26 00:14:32 ++ rm /var/lib/jenkins/sccache_error.log 
Sep 26 00:14:32 rm: cannot remove '/var/lib/jenkins/sccache_error.log': No such file or directory 
Sep 26 00:14:32 ++ true 
Sep 26 00:14:32 ++ [[ pytorch-linux-xenial-py3-clang7-onnx-build == *rocm* ]] 
Sep 26 00:14:32 ++ SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 
Sep 26 00:14:32 ++ SCCACHE_IDLE_TIMEOUT=1200 
Sep 26 00:14:32 ++ RUST_LOG=sccache::server=error 
Sep 26 00:14:32 ++ sccache --start-server 
Sep 26 00:14:32 Starting sccache server... 

pytorch_macos_10_13_py3_test (2/2)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

Sep 26 01:57:27 [E request_callback_no_python.cpp:618] Received error while processing request type 2: RuntimeError: Can not pickle torch.futures.Future

Sep 26 01:57:27 At: 
Sep 26 01:57:27   /Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py(94): serialize 
Sep 26 01:57:27   /Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py(146): serialize 
Sep 26 01:57:27  
Sep 26 01:57:27 [E request_callback_no_python.cpp:618] Received error while processing request type 2: RuntimeError: Can not pickle torch.futures.Future 
Sep 26 01:57:27  
Sep 26 01:57:27 At: 
Sep 26 01:57:27   /Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py(94): serialize 
Sep 26 01:57:27   /Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py(146): serialize 
Sep 26 01:57:27  
Sep 26 01:57:27 [E request_callback_no_python.cpp:618] Received error while processing request type 2: RuntimeError: Can not pickle torch.futures.Future 
Sep 26 01:57:27  
Sep 26 01:57:27 At: 
Sep 26 01:57:27   /Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py(94): serialize 
Sep 26 01:57:27   /Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py(146): serialize 
Sep 26 01:57:27  
Sep 26 01:57:28 ok (1.569s) 
Sep 26 01:57:29   test_return_future_remote (__main__.ProcessGroupRpcTestWithSpawn) ... RPC was initialized with the PROCESS_GROUP backend which is deprecated and slated to be removed and superseded by the TENSORPIPE backend. It is recommended to migrate to the TENSORPIPE backend. 
Sep 26 01:57:29 RPC was initialized with the PROCESS_GROUP backend which is deprecated and slated to be removed and superseded by the TENSORPIPE backend. It is recommended to migrate to the TENSORPIPE backend. 
Sep 26 01:57:29 RPC was initialized with the PROCESS_GROUP backend which is deprecated and slated to be removed and superseded by the TENSORPIPE backend. It is recommended to migrate to the TENSORPIPE backend. 
Sep 26 01:57:29 RPC was initialized with the PROCESS_GROUP backend which is deprecated and slated to be removed and superseded by the TENSORPIPE backend. It is recommended to migrate to the TENSORPIPE backend. 

---

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 5 times.

[github-actions]

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.