Allow args to be optional in deepspeed.initialize

[jeffra]

Before this PR args passed into deepspeed.initialize must be an object with [get|set]attribute support. This PR allows users to initialize deepspeed via deepspeed.initialize(model=model, config_params=config_dict). This assumes that local_rank is set as an environment variable (which is already done via deepspeed/pytorch.distributed launchers and/or from deepspeed.init_distributed).

Can only do the above call with kwargs, positional args need to be backwards compatible to support deepspeed.initialize(args, model). NOTE: this change does not require changes to existing code that use deepspeed.initialize with args.

This PR allows hugging face and lightning to not have to pass a SimpleNamespace object to deepspeed.initialize which was previously a hack to get deepspeed working.

For hugging face integration this means they can (optionally) remove the SimpleNamespace object passed to local rank, since it will be set since deepspeed.init_distributed is already called prior to deepspeed.initialize which ensures the local rank is set properly. https://github.com/huggingface/transformers/blob/256482ac9285c467fb97ca3b1b693a4de1d0ac60/src/transformers/integrations.py#L409-L414 /cc @stas00

For lightning integration I don't believe it is using deepspeed.init_distributed for launching but if they assume torch.distributed launcher then this is fine, see: https://github.com/pytorch/pytorch/blob/master/torch/distributed/launch.py#L253. They can also remove SimpleNamespace here as well then. https://github.com/PyTorchLightning/pytorch-lightning/blob/efda48faab666b78bb5f71bff4a0838f9b82dee5/pytorch_lightning/plugins/training_type/deepspeed.py#L243 /cc @SeanNaren

In either case hf/lightning could set or assert os.environ['LOCAL_RANK'] = self.local_rank before calling deepspeed.initialize.

[stas00]

That is indeed much simpler. Thank you, @jeffra!

We will have to wait for your new release to implement that so that we can require that version number.

But probably need to wait for @cli99 to sort out the issue w/ non-DeepSpeed optim/scheduler before the new release if it's not too far.

[SeanNaren]

Nice changes @jeffra! Regarding lightning not using deepspeed.init_distributed, I think because this uses auto_mpi discovery as default, which required mpi4py to be installed, hence we just do a torch.distributed init process group ourselves. I may be incorrect now, will check :)

[jeffra]

Nice changes @jeffra! Regarding lightning not using deepspeed.init_distributed, I think because this uses auto_mpi discovery as default, which required mpi4py to be installed, hence we just do a torch.distributed init process group ourselves. I may be incorrect now, will check :)

Oh I see, interesting. deepspeed.init_distributed will only use the auto_mpi discovery code path (which requires mpi4py) if the training process(es) were launched without the deepspeed launcher or the torch distributed launcher.