Fix repeated samples after graceful node shutdown

Signed-off-by: Mohamed Yousef <myb@imachines.com>
horovod · Mar 19, 2022 · 81f9a5f · 81f9a5f
1 parent 7b5346e
commit 81f9a5f
Showing 1 changed file with 1 addition and 0 deletions.
diff --git a/horovod/torch/elastic/sampler.py b/horovod/torch/elastic/sampler.py
@@ -106,6 +106,7 @@ def reset(self):
         self.total_size = self.num_samples * self.num_replicas
 
     def __iter__(self):
+        self.reset()
         self.indices = self.remaining_indices[:]
 
         # add extra samples to make it evenly divisible