Fix possible padding length overflow in DistributedSampler #45329

agemor · 2020-09-25T05:58:09Z

This fix handles cases for len(dataset) * 2 < num_replica in DistributedSampler. (which previous code resulted in error.)

codecov · 2020-09-25T09:48:02Z

Codecov Report

Merging #45329 into master will decrease coverage by 0.00%.
The diff coverage is 75.00%.

@@            Coverage Diff             @@
##           master   #45329      +/-   ##
==========================================
- Coverage   68.08%   68.08%   -0.01%     
==========================================
  Files         393      393              
  Lines       50960    50963       +3     
==========================================
+ Hits        34698    34699       +1     
- Misses      16262    16264       +2

Impacted Files	Coverage Δ
torch/utils/data/distributed.py	`74.46% <75.00%> (-0.54%)`	⬇️
torch/testing/_internal/expecttest.py	`77.55% <0.00%> (-1.03%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 03dde4c...2d622d2. Read the comment docs.

rohan-varma

Thank you for the fix! Could we add a unitttest for this to ensure no future regression? The file distributed_test.py has a DistributedSampler test which you can use as an example.

dr-ci · 2020-09-26T05:17:34Z

💊 CI failures summary and remediations

As of commit 2d622d2 (more details on the Dr. CI page):

4/5 failures possibly* introduced in this PR
- 1/4 non-CircleCI failure(s)
1/5 tentatively recognized as flaky ❄️
- Click here to rerun these jobs

🕵️ 3 new failures recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

pytorch_macos_10_13_py3_test (1/3)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

Sep 26 07:06:51 [E request_callback_no_python.cpp:618] Received error while processing request type 2: RuntimeError: Can not pickle torch.futures.Future

Sep 26 07:06:51 At: 
Sep 26 07:06:51   /Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py(94): serialize 
Sep 26 07:06:51   /Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py(146): serialize 
Sep 26 07:06:51  
Sep 26 07:06:51 [E request_callback_no_python.cpp:618] Received error while processing request type 2: RuntimeError: Can not pickle torch.futures.Future 
Sep 26 07:06:51  
Sep 26 07:06:51 At: 
Sep 26 07:06:51   /Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py(94): serialize 
Sep 26 07:06:51   /Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py(146): serialize 
Sep 26 07:06:51  
Sep 26 07:06:51 [E request_callback_no_python.cpp:618] Received error while processing request type 2: RuntimeError: Can not pickle torch.futures.Future 
Sep 26 07:06:51  
Sep 26 07:06:51 At: 
Sep 26 07:06:51   /Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py(94): serialize 
Sep 26 07:06:51   /Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/torch/distributed/rpc/internal.py(146): serialize 
Sep 26 07:06:51  
Sep 26 07:06:51 ok (1.579s) 
Sep 26 07:06:52   test_return_future_remote (__main__.ProcessGroupRpcTestWithSpawn) ... RPC was initialized with the PROCESS_GROUP backend which is deprecated and slated to be removed and superseded by the TENSORPIPE backend. It is recommended to migrate to the TENSORPIPE backend. 
Sep 26 07:06:52 RPC was initialized with the PROCESS_GROUP backend which is deprecated and slated to be removed and superseded by the TENSORPIPE backend. It is recommended to migrate to the TENSORPIPE backend. 
Sep 26 07:06:52 RPC was initialized with the PROCESS_GROUP backend which is deprecated and slated to be removed and superseded by the TENSORPIPE backend. It is recommended to migrate to the TENSORPIPE backend. 
Sep 26 07:06:52 RPC was initialized with the PROCESS_GROUP backend which is deprecated and slated to be removed and superseded by the TENSORPIPE backend. It is recommended to migrate to the TENSORPIPE backend.

pytorch_python_doc_build (2/3)

Step: "Doc Build and Push" (full log | diagnosis details | 🔁 rerun)

Sep 26 06:03:55 Makefile:38: recipe for target 'html' failed

Sep 26 06:03:54 copying images... [100%] scripts/activation_images/ReLU6.png 
Sep 26 06:03:54  
Sep 26 06:03:54 copying static files... ... done 
Sep 26 06:03:54 copying extra files... done 
Sep 26 06:03:55 dumping search index in English (code: en)... done 
Sep 26 06:03:55 dumping object inventory... done 
Sep 26 06:03:55 build finished with problems, 4 warnings. 
Sep 26 06:03:55 /var/lib/jenkins/workspace/docs/src/pytorch-sphinx-theme/pytorch_sphinx_theme/search.html:21: RemovedInSphinx30Warning: To modify script_files in the theme is deprecated. Please insert a <script> tag directly in your theme instead. 
Sep 26 06:03:55   {% trans %}Please activate JavaScript to enable the search 
Sep 26 06:03:55 make: *** [html] Error 1 
Sep 26 06:03:55 Makefile:38: recipe for target 'html' failed

pytorch_doc_test (3/3)

Step: "Doc test" (full log | diagnosis details | 🔁 rerun)

Sep 26 05:52:07 caused by: Connection refused (os error 111)

Sep 26 05:52:07 +++++ extract_trap_cmd 
Sep 26 05:52:07 +++++ printf '%s\n' '' 
Sep 26 05:52:07 ++++ printf '%s\n' cleanup 
Sep 26 05:52:07 +++ trap -- ' 
Sep 26 05:52:07 cleanup' EXIT 
Sep 26 05:52:07 +++ [[ pytorch-linux-xenial-py3.6-gcc5.4-build != *pytorch-win-* ]] 
Sep 26 05:52:07 +++ which sccache 
Sep 26 05:52:07 +++ sccache --stop-server 
Sep 26 05:52:07 Stopping sccache server... 
Sep 26 05:52:07 error: couldn't connect to server 
Sep 26 05:52:07 caused by: Connection refused (os error 111) 
Sep 26 05:52:07 +++ true 
Sep 26 05:52:07 +++ rm /var/lib/jenkins/sccache_error.log 
Sep 26 05:52:07 +++ [[ pytorch-linux-xenial-py3.6-gcc5.4-build == *rocm* ]] 
Sep 26 05:52:07 +++ SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 
Sep 26 05:52:07 +++ SCCACHE_IDLE_TIMEOUT=1200 
Sep 26 05:52:07 +++ RUST_LOG=sccache::server=error 
Sep 26 05:52:07 +++ sccache --start-server 
Sep 26 05:52:07 Starting sccache server... 
Sep 26 05:52:08 +++ sccache --zero-stats 
Sep 26 05:52:08 Compile requests                 0

❄️ 1 failure tentatively classified as flaky

but reruns have not yet been triggered to confirm:

pytorch_xla_linux_bionic_py3_6_clang9_test (1/1)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun) ❄️

Sep 26 06:18:43 CondaHTTPError: HTTP 000 CONNECTION FAILED for url

Sep 26 06:17:22 ++ [[ pytorch-xla-linux-bionic-py3.6-clang9-test == *pytorch-linux-xenial-cuda10.1-cudnn7-py3* ]] 
Sep 26 06:17:22 ++ [[ pytorch-xla-linux-bionic-py3.6-clang9-test == *pytorch-linux-trusty-py3.6-gcc7* ]] 
Sep 26 06:17:22 ++ [[ pytorch-xla-linux-bionic-py3.6-clang9-test == *pytorch_macos* ]] 
Sep 26 06:17:22 ++ BUILD_TEST_LIBTORCH=0 
Sep 26 06:17:22 ++ [[ pytorch-xla-linux-bionic-py3.6-clang9-test == *pytorch-xla-linux-bionic* ]] 
Sep 26 06:17:22 ++ which conda 
Sep 26 06:17:22 /opt/conda/bin/conda 
Sep 26 06:17:22 ++ conda install -q -y cmake 
Sep 26 06:18:43 Collecting package metadata (current_repodata.json): ...working... failed 
Sep 26 06:18:43  
Sep 26 06:18:43 CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://repo.anaconda.com/pkgs/main/noarch/current_repodata.json> 
Sep 26 06:18:43 Elapsed: - 
Sep 26 06:18:43  
Sep 26 06:18:43 An HTTP error occurred when trying to retrieve this URL. 
Sep 26 06:18:43 HTTP errors are often intermittent, and a simple retry will get you on your way. 
Sep 26 06:18:43  
Sep 26 06:18:43 If your current network has https://www.anaconda.com blocked, please file 
Sep 26 06:18:43 a support request with your network engineering team. 
Sep 26 06:18:43  
Sep 26 06:18:43 'https://repo.anaconda.com/pkgs/main/noarch' 
Sep 26 06:18:43

ci.pytorch.org: 1 failed

Failed: pr/pytorch-linux-bionic-rocm3.7-py3.6

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 5 times.

agemor · 2020-09-26T09:32:34Z

Sure 👍 I added the unit test assuring that tiny datasets are adequately padded in DistributedSampler .

rohan-varma

Thanks, LGTM!

facebook-github-bot

@rohan-varma has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot · 2020-10-15T02:15:43Z

@rohan-varma merged this pull request in a699108.

facebook-github-bot · 2020-10-15T02:16:06Z

@rohan-varma merged this pull request in a699108.

Fix possible padding length overflow

2355127

agemor requested a review from apaszke as a code owner September 25, 2020 05:58

pytorchbot added the open source label Sep 25, 2020

zou3519 requested a review from VitalyFedyunin September 25, 2020 14:11

zou3519 added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Sep 25, 2020

rohan-varma self-requested a review September 25, 2020 16:53

rohan-varma reviewed Sep 25, 2020

View reviewed changes

Add unit test for padding tiny dataset in DistributedSampler

3e28073

agemor requested review from mrshenli, pritamdamania87 and zhaojuanmao as code owners September 26, 2020 04:58

agemor added 2 commits September 26, 2020 14:15

Add unit test for padding tiny datasets in DistributedSampler

aa7f3e3

Merge branch 'master' of github.com:agemor/pytorch

2d622d2

agemor requested a review from rohan-varma September 29, 2020 06:03

rohan-varma approved these changes Oct 8, 2020

View reviewed changes

facebook-github-bot reviewed Oct 8, 2020

View reviewed changes

facebook-github-bot closed this in a699108 Oct 15, 2020

facebook-github-bot added the Merged label Oct 15, 2020

ejguan mentioned this pull request Dec 15, 2020

Bad error message for DistributedSampler when num_replicas is too high #49172

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix possible padding length overflow in DistributedSampler #45329

Fix possible padding length overflow in DistributedSampler #45329

agemor commented Sep 25, 2020

codecov bot commented Sep 25, 2020 •

edited

rohan-varma left a comment

dr-ci bot commented Sep 26, 2020 •

edited

agemor commented Sep 26, 2020

rohan-varma left a comment

facebook-github-bot left a comment

facebook-github-bot commented Oct 15, 2020

facebook-github-bot commented Oct 15, 2020

Fix possible padding length overflow in DistributedSampler #45329

Fix possible padding length overflow in DistributedSampler #45329

Conversation

agemor commented Sep 25, 2020

codecov bot commented Sep 25, 2020 • edited

Codecov Report

rohan-varma left a comment

Choose a reason for hiding this comment

dr-ci bot commented Sep 26, 2020 • edited

💊 CI failures summary and remediations

🕵️ 3 new failures recognized by patterns

pytorch_macos_10_13_py3_test (1/3)

pytorch_python_doc_build (2/3)

pytorch_doc_test (3/3)

❄️ 1 failure tentatively classified as flaky

pytorch_xla_linux_bionic_py3_6_clang9_test (1/1)

ci.pytorch.org: 1 failed

agemor commented Sep 26, 2020

rohan-varma left a comment

Choose a reason for hiding this comment

facebook-github-bot left a comment

Choose a reason for hiding this comment

facebook-github-bot commented Oct 15, 2020

facebook-github-bot commented Oct 15, 2020

codecov bot commented Sep 25, 2020 •

edited

dr-ci bot commented Sep 26, 2020 •

edited