Commits on Apr 21, 2021

1. Update numpy and pytorch seeding for dataloader and multiple processe… …

   …s per machine.
   
   Summary:
   *Current state:*
   
   Currently we set different seeds per-nodes, but the same seed among all training processes on a node. However, each of our Dataloader process seeds are all different each epoch, but non-deterministic.
   
   *Proposed State:*
   
   Different random number seeds for each dist_rank, and different, deterministic, for each Dataloader process seeds per epoch.
   
   https://fb.quip.com/hVIcAahpVLo2
   
   *Effects:*
   
   Fixes randomization for a few losses, hooks, and trunks. Fixes randomization when using the Fork multiprocessing option for transformations. Fixes collapse of seeds to 0 when config seed set to 0.
   
   There are 3 changes summarized as:
   1. Use dist_rank instead of node_id for seeding training processes.
   2. Create worker_init_fn to manually set numpy seed for dataloader workers.
   3. Add knowledge of training process seed to sampler.
   
   Differential Revision: D27784137
   
   fbshipit-source-id: 023c1ded41a4ed7c0b2caaedca5b5d1999d1ce42

   

   iseessel authored and facebook-github-bot committed Apr 21, 2021
   Configuration menu

   • View commit details
   • Copy full SHA for d9ed96e
   • Browse repository at this point

   Copy the full SHA

   d9ed96e View commit details

   Browse the repository at this point in the history