How to run distributed training with ignite in a nvidia-docker container?

[songkq]

Environment:

1. nvidia-docker container: Ubuntu 16.04 with 4 2080Ti GPUs
2. Framework version: PyTorch 1.2
3. Ignite version: 0.2.0
4. CUDA version: 10.1
5. NCCL version: 2.1.4
6. Python version: 3.6.8
7. GCC version: 7.4.0

Question:
Hello, I'd like to use ignite for distributed training. According to the mnist_dist.py example, I need to lauch two terminals in one machine that each terminal runs one command to start training shown as the following.

However, the docker container only allows me lauching one terminal, I wonder if there is an alternative with running distributed training in a nvidia-docker container?
For instance, I could run 4 docker containers with a GPU, i.e four nodes have one GPU, respectively.
Then On the Node i:
open a terminal and run the example on the GPU 0 (process rank i):
python mnist_dist.py --world_size 4 --rank i --gpu 0 --dist_method='tcp://IP_OF_NODE0:FREEPORT'
Thus can I run distributed training with the four nodes? Is there any kind advice?

[alxlampe]

In my company, we have the same one-terminal restriction.
I am using nohup (allows you to close the shell and keep the job running), redirect the command line output to a file and send the process to the background:
$ nohup COMMAND > output.txt &

https://www.tecmint.com/run-linux-command-process-in-background-detach-process/

[vfdev-5]

@ipScore in case of single machine, multiple GPUs you can also use built-in torch distributed launcher :

python -m torch.distributed.launch --nproc_per_node=4 main.py arg1 arg2

See here for more details.

[songkq]

Thanks for your advice, @vfdev-5 . However, when I run the main.py with torch.distributed.launch, it doesn't work and always hangs after calling model = DistributedDataParallel(model, [args.gpu]). It seems that 2 processes have been spwan, however waiting for something to complete.

python3 -u -m torch.distributed.launch --nproc_per_node=2 mnist_dist.py

The following shows the main part of mnist_dist.py, where I only insert some codes to parse the command-line argument: ``--local_rank=LOCAL_PROCESS_RANK`` provided by launch module.

if __name__ == "__main__":
    parser = ArgumentParser()
    parser.add_argument("--batch_size", type=int, default=64,
                        help="input batch size for training (default: 64)")
    parser.add_argument("--val_batch_size", type=int, default=1000,
                        help="input batch size for validation (default: 1000)")
    parser.add_argument("--epochs", type=int, default=10,
                        help="number of epochs to train (default: 10)")
    parser.add_argument("--lr", type=float, default=0.01,
                        help="learning rate (default: 0.01)")
    parser.add_argument("--momentum", type=float, default=0.5,
                        help="SGD momentum (default: 0.5)")
    parser.add_argument("--log_interval", type=int, default=10,
                        help="how many batches to wait before logging training status")
    parser.add_argument("--dist_method", default="file:///home/user/tmp.dat", type=str,
                        help="url or file path used to set up distributed training")
    parser.add_argument("--dist_backend", default="nccl", type=str, help="distributed backend")
    parser.add_argument("--world_size", default=2, type=int, help="Number of GPUs to use.")
    parser.add_argument("--rank", default=0, type=int, help="Used for multi-process training.")
    parser.add_argument("--gpu", default=0, type=int, help="GPU number to use.")

    # --------------------------------------------------------------
    parser.add_argument("--local_rank", default=0, type=int, help="Used for multi-process training.")
    # --------------------------------------------------------------
    args = parser.parse_args()

    args.distributed = args.world_size > 1

    if args.distributed:
        # ----------------------------------------------------------
        args.gpu = args.local_rank
        args.rank = args.local_rank
        # ----------------------------------------------------------

        torch.cuda.set_device(args.gpu)
        torch.distributed.init_process_group(args.dist_backend,
                                             init_method=args.dist_method,
                                             world_size=args.world_size, rank=args.rank)

    run(args.batch_size, args.val_batch_size, args.epochs, args.lr, args.momentum, args.log_interval)

[vfdev-5]

@ipScore yes, probably we need to update mnist_dist.py as it was the first example with distributed training. We are working on adapting ignite to work correctly in distributed settings, so you can take a look at CIFAR10 example in distrib branch.

[songkq]

@vfdev-5 I'll try CIFAR10 Example soon. Thank you~

[songkq]

@Bibonaut Thank you for your advice. I wonder if nohub supports the operations under a nv-docker container as the host machine? I'll try it later.

[songkq]

@vfdev-5 It works well with CIFAR10 example in my docker container. I wonder why two process occupy different memory?

[vfdev-5]

I wonder why two process occupy different memory?

@ipScore it should be related with master and worker processes.

I let you close the issue if the question is answered.