DistributedSampler can't shuffle the dataset

[elk-cloner]

🐛 Bug

Information

I'm trying to fine-tune BERT model using run_language_modeling.py.

Language I am using the model on is Persian:

The problem arises when using:

• the official example scripts: (give details below)
• my own modified scripts: (give details below)

The tasks I am working on is:

• an official GLUE/SQUaD task: (give the name)
• my own task or dataset: (give details below)

But according to this issue there is a bug in torch.utils.data.distributed.DistributedSampler so that during different epochs shuffling operation doesn't work properly(it's not working).
To solve this problem: according to pytorch official example here, we should add train_sampler.set_epoch(epoch) before each new epoch at this line

To reproduce

Steps to reproduce the behavior:

1. compare batches between different epoch like mentioned issue

Expected behavior

Environment info

• transformers version: transformers==2.8.0
• Platform: Ubuntu 18.04
• Python version: 3.7
• PyTorch version (GPU?): torch==1.4.0 (Yes)
• Tensorflow version (GPU?): tensorflow-gpu==2.1.0 (Yes)
• Using GPU in script?: Yes
• Using distributed or parallel set-up in script?: distributed

[julien-c]

I think you are right

[vlevit]

Isn't there the same issue in other places?
E.g. in trainer.py:

transformers/src/transformers/trainer.py

Lines 305 to 307 in 97a3754

	for epoch in train_iterator:
	epoch_iterator = tqdm(train_dataloader, desc="Iteration", disable=self.args.local_rank not in [-1, 0])
	for step, inputs in enumerate(epoch_iterator):

[julien-c]

I forgot to re-add this in Trainer when merging #3800

It's on my todo-list, but feel free to open a PR if you can do it faster than I can

[vlevit]

Great. Personally I've not yet upgraded to the newer version with trainer.py, so I'll leave it for you, thanks.