Update examples/documentation for torch.distributed.run

[BramVanroy]

Most examples now read local rank from a CLI argument --local_rank. This ensures for it to work when using the torch launch utility torch.distributed.launch. However, since recent torch versions, launch is deprecated. The new suggested way to run distributed code from CLI is with torch.distributed.run. The difference from before is that instead of automatically passing CLI arguments, it sets the rank as an environment variable. The examples therefore need to be updated to read the ENV variable instead of the CLI argument.

See https://pytorch.org/docs/stable/elastic/run.html#launcher-api for more.

[BramVanroy]

I missed that the training arguments already have this covered:

transformers/src/transformers/training_args.py

Lines 681 to 685 in bcc3f7b

	# Handle --use_env option in torch.distributed.launch (local_rank not passed as an arg then).
	# This needs to happen before any call to self.device or self.n_gpu.
	env_local_rank = int(os.environ.get("LOCAL_RANK", -1))
	if env_local_rank != -1 and env_local_rank != self.local_rank:
	self.local_rank = env_local_rank

Documentation may need an update though (see below).

[LysandreJik]

Hi @BramVanroy! Let us know if this isn't sufficiently documented so that we may patch the issue

[BramVanroy]

@LysandreJik I think the code in the TrainingArgs is fine. Documentation may need an update as it refers to launch in a couple of places. A quick search gives me mentions of torch.distributed.launch in these documents that may need to be replaced by torch.distributed.run. You'd also need to check whether --nproc_per_node is still an accepted argument for that command because I am not sure. So it is not only a find-and-replace update!

• https://huggingface.co/transformers/performance.html
• https://huggingface.co/transformers/main_classes/deepspeed.html
• https://huggingface.co/transformers/debugging.html

[LysandreJik]

Maybe of interest to @sgugger

[sgugger]

I wouldn't update the documentation as it won't work for older versions of PyTorch and I would prefer to only put one command vs If your PyTorch version is this then xxx else yyy. Let's wait a bit more :-)

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[sgugger]

My previous comment still stands :-)

[BramVanroy]

Definitely. I think it's a good idea to keep bumping this to open, so that others can also find this issue easily, and it can serve as a reminder that as soon as launch is obsolete, the docs need an update. It's a tiny thing and easy to forget, so an open issue may help to remember as time goes on.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.