Add `generator=` kwarg for DataLoader & random samplers #39737

ShawnZhong · 2020-06-09T17:20:05Z

Fix #39572

Add generator= kwarg for DataLoader & random samplers

cc: @ssnl, @deeppatel4557, @albanD, @mitar

dr-ci · 2020-06-09T18:15:38Z

💊 CI failures summary and remediations

As of commit d95ff17 (more details on the Dr. CI page):

9/9 failures possibly* introduced in this PR
- 2/9 non-CircleCI failure(s)

🕵️ 7 new failures recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

pytorch_windows_vs2019_py36_cpu_test2 (1/7)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

RuntimeError: test_autograd failed!

  File "test_autograd.py", line 6451, in <module> 
    run_tests() 
  File "C:\Users\circleci\project\build\win_tmp\build\torch\testing\_internal\common_utils.py", line 278, in run_tests 
    import xmlrunner 
ModuleNotFoundError: No module named 'xmlrunner' 
Traceback (most recent call last): 
  File "run_test.py", line 711, in <module> 
    main() 
  File "run_test.py", line 704, in main 
    raise RuntimeError(message) 
RuntimeError: test_autograd failed! 
 
(base) circleci@PACKER-5ECD3249 C:\Users\circleci\project\test>if ERRORLEVEL 1 exit /b 1  
+ cleanup
+ retcode=1
+ set +x

pytorch_windows_vs2019_py36_cpu_test1 (2/7)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

CMake Error at CMakeLists.txt:3 (ENABLE_LANGUAGE):

    Tell CMake where to find the compiler by setting either the environment 
    variable "CC" or the CMake cache entry CMAKE_C_COMPILER to the full path to 
    the compiler, or to the compiler name if it is in the PATH. 
   
   
  -- Configuring incomplete, errors occurred! 
  See also "C:/Users/circleci/AppData/Local/Temp/pip-install-m450xtk9/ninja/_cmake_test_compile/build/CMakeFiles/CMakeOutput.log". 
  See also "C:/Users/circleci/AppData/Local/Temp/pip-install-m450xtk9/ninja/_cmake_test_compile/build/CMakeFiles/CMakeError.log". 
  Not searching for unused variables given on the command line. 
  -- The C compiler identification is unknown 
  CMake Error at CMakeLists.txt:3 (ENABLE_LANGUAGE): 
    The CMAKE_C_COMPILER: 
   
      cl 
   
    is not a full path and was not found in the PATH. 
   
    To use the JOM generator with Visual C++, cmake must be run from a shell 
    that can use the compiler cl from the command line.  This environment is 
    unable to invoke the cl compiler.  To fix this problem, run cmake from the 
    Visual Studio Command Prompt (vcvarsall.bat).

pytorch_windows_vs2019_py36_cuda10.1_test1 (3/7)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

CMake Error at CMakeLists.txt:3 (ENABLE_LANGUAGE):

    Tell CMake where to find the compiler by setting either the environment 
    variable "CC" or the CMake cache entry CMAKE_C_COMPILER to the full path to 
    the compiler, or to the compiler name if it is in the PATH. 
   
   
  -- Configuring incomplete, errors occurred! 
  See also "C:/Users/circleci/AppData/Local/Temp/pip-install-9z5tud79/ninja/_cmake_test_compile/build/CMakeFiles/CMakeOutput.log". 
  See also "C:/Users/circleci/AppData/Local/Temp/pip-install-9z5tud79/ninja/_cmake_test_compile/build/CMakeFiles/CMakeError.log". 
  Not searching for unused variables given on the command line. 
  -- The C compiler identification is unknown 
  CMake Error at CMakeLists.txt:3 (ENABLE_LANGUAGE): 
    The CMAKE_C_COMPILER: 
   
      cl 
   
    is not a full path and was not found in the PATH. 
   
    To use the JOM generator with Visual C++, cmake must be run from a shell 
    that can use the compiler cl from the command line.  This environment is 
    unable to invoke the cl compiler.  To fix this problem, run cmake from the 
    Visual Studio Command Prompt (vcvarsall.bat).

pytorch_windows_vs2019_py36_cuda10.1_test2 (4/7)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

RuntimeError: test_autograd failed!

  File "test_autograd.py", line 6451, in <module> 
    run_tests() 
  File "C:\Users\circleci\project\build\win_tmp\build\torch\testing\_internal\common_utils.py", line 278, in run_tests 
    import xmlrunner 
ModuleNotFoundError: No module named 'xmlrunner' 
Traceback (most recent call last): 
  File "run_test.py", line 711, in <module> 
    main() 
  File "run_test.py", line 704, in main 
    raise RuntimeError(message) 
RuntimeError: test_autograd failed! 
 
(base) circleci@PACKER-5ECD3242 C:\Users\circleci\project\test>if ERRORLEVEL 1 exit /b 1  
+ cleanup
+ retcode=1
+ set +x

pytorch_windows_vs2019_py36_cuda10.1_on_cpu_test2 (5/7)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

RuntimeError: test_autograd failed!

  File "test_autograd.py", line 6451, in <module> 
    run_tests() 
  File "C:\Users\circleci\project\build\win_tmp\build\torch\testing\_internal\common_utils.py", line 278, in run_tests 
    import xmlrunner 
ModuleNotFoundError: No module named 'xmlrunner' 
Traceback (most recent call last): 
  File "run_test.py", line 711, in <module> 
    main() 
  File "run_test.py", line 704, in main 
    raise RuntimeError(message) 
RuntimeError: test_autograd failed! 
 
(base) circleci@PACKER-5ECD3249 C:\Users\circleci\project\test>if ERRORLEVEL 1 exit /b 1  
+ cleanup
+ retcode=1
+ set +x

pytorch_windows_vs2019_py36_cuda10.1_on_cpu_test1 (6/7)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

CMake Error at CMakeLists.txt:3 (ENABLE_LANGUAGE):

    Tell CMake where to find the compiler by setting either the environment 
    variable "CC" or the CMake cache entry CMAKE_C_COMPILER to the full path to 
    the compiler, or to the compiler name if it is in the PATH. 
   
   
  -- Configuring incomplete, errors occurred! 
  See also "C:/Users/circleci/AppData/Local/Temp/pip-install-qhf9i0pf/ninja/_cmake_test_compile/build/CMakeFiles/CMakeOutput.log". 
  See also "C:/Users/circleci/AppData/Local/Temp/pip-install-qhf9i0pf/ninja/_cmake_test_compile/build/CMakeFiles/CMakeError.log". 
  Not searching for unused variables given on the command line. 
  -- The C compiler identification is unknown 
  CMake Error at CMakeLists.txt:3 (ENABLE_LANGUAGE): 
    The CMAKE_C_COMPILER: 
   
      cl 
   
    is not a full path and was not found in the PATH. 
   
    To use the JOM generator with Visual C++, cmake must be run from a shell 
    that can use the compiler cl from the command line.  This environment is 
    unable to invoke the cl compiler.  To fix this problem, run cmake from the 
    Visual Studio Command Prompt (vcvarsall.bat).

pytorch_macos_10_13_py3_test (7/7)

Step: "Test" (full log | diagnosis details | 🔁 rerun)

Jun 11 21:32:27 RuntimeError: test_cuda failed!

Jun 11 21:32:27   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/librosa/util/utils.py", line 15, in <module> 
Jun 11 21:32:27     from .decorators import deprecated 
Jun 11 21:32:27   File "/Users/distiller/workspace/miniconda3/lib/python3.7/site-packages/librosa/util/decorators.py", line 9, in <module> 
Jun 11 21:32:27     from numba.decorators import jit as optional_jit 
Jun 11 21:32:27 ModuleNotFoundError: No module named 'numba.decorators' 
Jun 11 21:32:27 Traceback (most recent call last): 
Jun 11 21:32:27   File "test/run_test.py", line 711, in <module> 
Jun 11 21:32:27     main() 
Jun 11 21:32:27   File "test/run_test.py", line 704, in main 
Jun 11 21:32:27     raise RuntimeError(message) 
Jun 11 21:32:27 RuntimeError: test_cuda failed! 
Jun 11 21:32:27 + cleanup 
Jun 11 21:32:27 + retcode=1 
Jun 11 21:32:27 + set +x

ci.pytorch.org: 2 failed

Failed: pr/caffe2-pytorch-linux-xenial-rocm3.3-py3.6-test
Failed: pr/pytorch-linux-xenial-rocm3.3-py3.6

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker or post in the (internal) Dr. CI Users group.

See how this bot performed.

This comment has been revised 26 times.

ssnl

the generator for DataLoader should also be used for base_seed in iterators

ssnl

can we test the base_seed reproducibility by doing something similar to test_worker_seed?

ssnl · 2020-06-12T03:47:53Z

test/test_dataloader.py

+        dataset = SynchronizedSeedDataset(num_workers, batch_size, num_workers)
+        dataloader = DataLoader(dataset, batch_size=batch_size, num_workers=num_workers, generator=torch.Generator().manual_seed(42))
+        actual_seeds = set(int(batch) for batch in dataloader)
+        expected_seeds = set(6909045637428952499 + i for i in range(num_workers))


don't do this. seeding is only guaranteed to be consistent on the same device & version. just do two runs and compare results

ssnl

merge when green. thanks!

albanD · 2020-06-12T14:53:44Z

Windows CI is having a big hiccup because of some pip issues. Don't worry about those :)

albanD

Thanks for the PR.

facebook-github-bot

@albanD is landing this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

ssnl · 2020-06-15T18:06:35Z

Hi @ShawnZhong , do you mind also updating the notes at https://pytorch.org/docs/stable/data.html#randomness-in-multi-process-data-loading https://pytorch.org/docs/stable/notes/faq.html#my-data-loader-workers-return-identical-random-numbers about this too? Thanks :)

Summary: Fix pytorch#39572 Add `generator=` kwarg for DataLoader & random samplers cc: SsnL, deeppatel4557, albanD, mitar Pull Request resolved: pytorch#39737 Differential Revision: D22019132 Pulled By: albanD fbshipit-source-id: 835e08b86c5396bc0b0e41057661306b15394d6e

ShawnZhong changed the title ~~Add generator= kwarg for DataLoader & random samplers~~ [WIP] Add generator= kwarg for DataLoader & random samplers Jun 9, 2020

pytorchbot added the open source label Jun 9, 2020

ShawnZhong changed the title ~~[WIP] Add generator= kwarg for DataLoader & random samplers~~ Add generator= kwarg for DataLoader & random samplers Jun 9, 2020

ShawnZhong marked this pull request as ready for review June 9, 2020 19:50

ShawnZhong requested a review from apaszke as a code owner June 9, 2020 19:50

ssnl requested changes Jun 9, 2020

View reviewed changes

albanD self-requested a review June 9, 2020 20:36

albanD added triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module module: nn Related to torch.nn labels Jun 9, 2020

ShawnZhong marked this pull request as draft June 10, 2020 21:02

ShawnZhong changed the title ~~Add generator= kwarg for DataLoader & random samplers~~ [WIP] Add generator= kwarg for DataLoader & random samplers Jun 10, 2020

ShawnZhong force-pushed the dataloader_gen branch from a210e86 to 7ca2056 Compare June 10, 2020 23:33

ShawnZhong marked this pull request as ready for review June 11, 2020 17:42

ShawnZhong changed the title ~~[WIP] Add generator= kwarg for DataLoader & random samplers~~ Add generator= kwarg for DataLoader & random samplers Jun 11, 2020

ShawnZhong requested a review from ssnl June 11, 2020 17:43

ssnl reviewed Jun 11, 2020

View reviewed changes

ShawnZhong marked this pull request as draft June 12, 2020 02:35

ShawnZhong added 7 commits June 11, 2020 21:35

Update dataloader.py

1bacee6

Update sampler.py

9609dcb

Update sampler.py

da5aafd

Add tests

4a7ed87

bug fix

a481793

fix test

24bd52b

Set _base_seed

d249c04

ShawnZhong force-pushed the dataloader_gen branch from 7ca2056 to d249c04 Compare June 12, 2020 02:36

test_worker_seed_reproducibility

58deb08

ssnl reviewed Jun 12, 2020

View reviewed changes

Update test_worker_seed_reproducibility

ec821db

ssnl approved these changes Jun 12, 2020

View reviewed changes

Fix flake8

d95ff17

albanD approved these changes Jun 12, 2020

View reviewed changes

facebook-github-bot reviewed Jun 12, 2020

View reviewed changes

ShawnZhong marked this pull request as ready for review June 12, 2020 16:46

facebook-github-bot closed this in c8c53c8 Jun 15, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add `generator=` kwarg for DataLoader & random samplers #39737

Add `generator=` kwarg for DataLoader & random samplers #39737

Uh oh!

ShawnZhong commented Jun 9, 2020 •

edited

Loading

Uh oh!

dr-ci bot commented Jun 9, 2020 •

edited

Loading

Uh oh!

ssnl left a comment

Uh oh!

ssnl left a comment

Uh oh!

ssnl Jun 12, 2020

Uh oh!

ShawnZhong Jun 12, 2020

Uh oh!

ssnl left a comment

Uh oh!

albanD commented Jun 12, 2020

Uh oh!

albanD left a comment

Uh oh!

facebook-github-bot left a comment

Uh oh!

ssnl commented Jun 15, 2020

Uh oh!

Uh oh!

Add generator= kwarg for DataLoader & random samplers #39737

Add generator= kwarg for DataLoader & random samplers #39737

Uh oh!

Conversation

ShawnZhong commented Jun 9, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dr-ci bot commented Jun 9, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💊 CI failures summary and remediations

🕵️ 7 new failures recognized by patterns

pytorch_windows_vs2019_py36_cpu_test2 (1/7)

pytorch_windows_vs2019_py36_cpu_test1 (2/7)

pytorch_windows_vs2019_py36_cuda10.1_test1 (3/7)

pytorch_windows_vs2019_py36_cuda10.1_test2 (4/7)

pytorch_windows_vs2019_py36_cuda10.1_on_cpu_test2 (5/7)

pytorch_windows_vs2019_py36_cuda10.1_on_cpu_test1 (6/7)

pytorch_macos_10_13_py3_test (7/7)

ci.pytorch.org: 2 failed

Uh oh!

ssnl left a comment

Choose a reason for hiding this comment

Uh oh!

ssnl left a comment

Choose a reason for hiding this comment

Uh oh!

ssnl Jun 12, 2020

Choose a reason for hiding this comment

Uh oh!

ShawnZhong Jun 12, 2020

Choose a reason for hiding this comment

Uh oh!

ssnl left a comment

Choose a reason for hiding this comment

Uh oh!

albanD commented Jun 12, 2020

Uh oh!

albanD left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

ssnl commented Jun 15, 2020

Uh oh!

Uh oh!

Add `generator=` kwarg for DataLoader & random samplers #39737

Add `generator=` kwarg for DataLoader & random samplers #39737

ShawnZhong commented Jun 9, 2020 •

edited

Loading

dr-ci bot commented Jun 9, 2020 •

edited

Loading