An example of using pytorch distributed data parallel on 1 machine with arbitrary multiple GPUs.

[li-li-github]

Motivation

The existing example pytorch_distributed_simple.py is designed to be used with MPI. There is no example for training a model with arbitrary multiple GPUs on a single machine, where no MPI is needed, which is not trivial to setup. This example should be very flexible that is also applicable to pytorch-lightning in the cases that the existing pytorch_lightning_ddp.py may not work.

Description of the changes

This example uses torch.distributed.init_process_group and torch.distributed.destroy_process_group as well as torch.multiprocessing to distribute jobs that MPI would do in pytorch_distributed_simple.py.

[github-actions]

This pull request has not seen any recent activity.

[toshihikoyanase]

I'm sorry for the delayed response. I think the proposed example seems useful since it enable users to optimize hyperparameters casually compared with pytorch_distributed_simple.py.
Let me share my first comments.

[nzw0301]

Do we need this variable? I thought rank was enough when I implemented similar example
https://gist.github.com/nzw0301/9ba3c837a02539e8bde194f4f2c9d0e5.

[li-li-github]

The code still runs without converting device_id to device but I'm not sure how did your example get the variable rank in

def objective(single_trial):
    trial = optuna.integration.TorchDistributedTrial(single_trial, rank)

I can modify the code so that it only uses device_id, but I think objective still needs an argument for device_id unless I'm missing something.

[nzw0301]

Hmm, indeed. Thank you for your comment. As you did, partial or lambda might be necessary.

[toshihikoyanase]

def objective(single_trial):
trial = optuna.integration.TorchDistributedTrial(single_trial, rank)

The new TorchDistributedTrial in optuna==v3.1.0 will not require rank or device.
It creates a process group with gloo backend only for Optuna's communication.
So, we may be able to update this part after the release of optuna v3.1.0.
https://github.com/optuna/optuna/blob/25a135c26662e5e26f88d7266eb2922703bf72dd/optuna/integration/pytorch_distributed.py#L119

[li-li-github]

Thank you @toshihikoyanase. Indeed having rank argument yields an error with optuna v3.1.0, but without this argument, optuna < 3.1.0 will raise an error (i.e. backward incompatible). I can push a commit to remove rank argument, and perhaps with a comment in the beginning of the code to let users know that this example only works with optuna >= 3.1.0.
Also, do we need to specify optuna version in the requirements.txt?

[nzw0301]

Due to this part, this example does not work with arbitrary multiple GPUs. I believe that torchrun style script (: https://github.com/optuna/optuna-examples/blob/main/pytorch/pytorch_distributed_simple.py) would be more practical in my PyTorch experience since it doesn't need to update the script at all and we can specify the number of gpus from the command line.

[li-li-github]

By arbiturary multiple GPUs, I mean that you can specify exactly which GPUs to use. I understand that in a large scale setting where you have multiple nodes that have multiple GPUs in each node, using MPI (and I guess as well as torchrun because it seems to require MPI) would be definitely a better practice.

This example, however, was made with the mind that you don't need to install any MPI packages. Say, I have a local machine with 4 GPUs and I want to run an experiment on GPU1 & 3 because I want GPU0 to be dedicated to graphics and GPU2 is running other programs. I can just run the code as a normal python code like python pytorch/pytorch_ddp_1machine.py.

I can make DEVICE_IDS an argument so that user can specify which particular GPUs to use from command line.

[nzw0301]

Thanks. I got the point. I should have discussed the necessity of this PR because #150 also addresses the same issue. (I suppose #150 still needs some update to decide rank since it uses mpi's variable.)

In terms of the number of gpus, we can manage it from the command line; For example, https://discuss.pytorch.org/t/how-to-change-the-default-device-of-gpu-device-ids-0/1041/2.

[li-li-github]

In terms of the number of gpus, we can manage it from the command line; For example, https://discuss.pytorch.org/t/how-to-change-the-default-device-of-gpu-device-ids-0/1041/2.

Do you mean passing the environment variables in the command line like this?
CUDA_VISIBLE_DEVICES=1,2 python myscript.py

I was told changing environment variable is generally not a good practice, so I tried avoiding this. Also, with the above example, I guess you might need to configure a default value in the python code.
In the updated commit, I offered a suggestion using --device_ids argument to specify device_ids with a default value 0. Let me know if you like it or what you have in your mind.

[nzw0301]

Do you mean ...

Yes!

I was told changing environment variable is generally not a good practice

Could you share any reference, ideally, PyTorch page? Thanks!

[li-li-github]

Could you share any reference, ideally, PyTorch page? Thanks!

I can't find a good reference but I feel the idea comes from more production or security side after some google search.
To summarize, my understanding is that

• Potentially making debugging harder because the environment change may not be applied immediately.
  https://support.cloud.engineyard.com/hc/en-us/articles/205407508-Environment-Variables-and-Why-You-Shouldn-t-Use-Them
• There're risks of leaking sensitive information
  https://blog.dnsimple.com/2018/05/using-environment-as-configuration/

Again, I was just told by a colleague of mine and I don't really have any experience about it. And apparently, there also seem to be many cases that env variables should be actively used. So I don't have strong opinion about this.
I just thought if there was a less risky way, then I just pick that.

So, if you think using environment variable is better, then I can change the code accordingly.

[nzw0301]

Do we need sqlite for this example? Personally, the example without database would be simpler and easy to run for users thanks to less dependencies.

[li-li-github]

I think just followed the example of pytorch_lightning_ddp.py. I got your point, but I guess the most likely next thing after getting the example code run is probably to find a way to log the tuning history, because in most of the cases, one would want to check the tuning history. I don't mind removing this part, but I also feel it would offer some convenience to users. We can make it optional with an argument, for example.

This probably should be a separate topic, but since you mentioned about database, I wonder if optuna could avoid repeating the same (or very similar) parameters that it tried in the past without saving them to a database. I'm asking this because it seems to me that optuna gets new parameters by referencing to the database (or the tuning history) especially when working with multiple GPUs.

[nzw0301]

pytorch_lightning_ddp.py

I've reviewed this code, so I know a little bit of the background of this example and integration module. Due to technical difficulties, we would only support distributed training with a database. This is why the example code uses SQL. So I don't think we need to follow pytorch_lightning_ddp.py. Another drawback to using SQL for storage is its speed performance. Using SQL is widely documented in Optuna, so I believe that Optuna users can extend the example without database storage to with database storage.

Repeated same/similar params

As an optuna feature, there isn't, but I think we can implement a similar feature: optuna/optuna#1001 (comment) or optuna/optuna#2021 (comment).

[li-li-github]

we would only support distributed training with a database

Do you mean that database is only supported in distributed training but not in other cases? If so, how should users save tuning history?

Another drawback to using SQL for storage is its speed performance.

What would be a better way to store tuning history then? I thought optuna only supports SQL until the latest version 3.1.0.
I also want to know if there're alternative storage options.

As an optuna feature, there isn't,

Do you mean that optuna doesn't have a feature to check parameters it tired in the past?
I think I got my answer. When I run my code without specifying storage, I see it prints out
A new study created in memory with name: no-name-eefeae74-8d1d-4613-9b57-b18177f10da3
So it seems like optuna is storing tuning history in memory instead of SQL, and I guess multiple trials can still refer to this storage in the memory. But I get following errors at the end of tuning if I don't specify storage.

Study statistics: 
  Number of finished trials:  0
  Number of pruned trials:  0
  Number of complete trials:  0
Best trial:
Traceback (most recent call last):
  File ".../optuna-examples/pytorch/pytorch_ddp_1machine.py", line 219, in <module>
    trial = study.best_trial
  File ".../anaconda3/envs/optuna/lib/python3.9/site-packages/optuna/study/study.py", line 159, in best_trial
    return copy.deepcopy(self._storage.get_best_trial(self._study_id))
  File ".../anaconda3/envs/optuna/lib/python3.9/site-packages/optuna/storages/_in_memory.py", line 262, in get_best_trial
    raise ValueError("No trials are completed yet.")
ValueError: No trials are completed yet.

So I guess, since I'm using distributed training, I still need SQL storage anyway?
I have no problem with finishing the entire tuning at pytorch_simple.py in which no storage is given.

[nzw0301]

Do you mean that database is only supported in distributed training but not in other cases? If so, how should users save tuning history?

I mean, pytorch lighting with data distributed training works only with database. Optuna's docs says

For the distributed data parallel training, the version of PyTorchLightning needs to be higher than or equal to v1.5.0. In addition, Study should be instantiated with RDB storage.

Reference: https://optuna.readthedocs.io/en/stable/reference/generated/optuna.integration.PyTorchLightningPruningCallback.html

---

What would be a better way to store tuning history then?

I suppose saving study would be simple to save optimisation history. https://optuna.readthedocs.io/en/stable/faq.html#how-can-i-save-and-resume-studies

Another way is to use optuna's callback feature, such as mlflow or wandb.

---

Do you mean that optuna doesn't have a feature to check parameters it tired in the past?

Yes.

---

So I guess, since I'm using distributed training, I still need SQL storage anyway?

I don't think so because another distributed training code: https://github.com/optuna/optuna-examples/blob/main/pytorch/pytorch_distributed_simple.py, doesn't use database.

[nzw0301]

Sorry, my last answer might be wrong. When we used spawn for ddp training for pytorch-lighting we actually had similar issue. So as you said, swamp-based ddp might require database storage...

[li-li-github]

I suppose saving study would be simple to save optimisation history

Thank you for the information. Good to know that study can be saved as pickle.

swamp-based ddp might require database storage

I see, thanks for the background information.

[nzw0301]

@li-li-github cc: @toshihikoyanase
Thank you for your contributions and comments. Let me summarise my thoughts on this PR. I believe #150 would support nccl backend, which enables us to use multiple GPUs environments. This PR is also motivated using nccl backend (: GPUs), but by using spawn in the script rather than mpirun or torchrun from CLI, which https://github.com/optuna/optuna-examples/blob/main/pytorch/pytorch_distributed_simple.py assumes.

When we use spawn, I suppose we might require a database to store study like PyTorchLigntingCallback because child processes report intermediate values. Thus as TorchDistributedTrial with spawn example (not multiple GPUs, but can be supported), this PR would be helpful. So I would like to suggest supporting other backends if possible because Optuna CI doesn't have GPUs.

[li-li-github]

Thank you for the summary.
I modified the code so that it takes CPU and gloo backend as default. Let me know what you think.

[nzw0301]

Thanks! Before reviewing the updates, I'd like to hear the opinion from @toshihikoyanase. What do you think about #155 (comment)?

[toshihikoyanase]

@nzw0301 @li-li-github Thank you for your insightful discussion.

but by using spawn in the script rather than mpirun or torchrun from CLI, which https://github.com/optuna/optuna-examples/blob/main/pytorch/pytorch_distributed_simple.py assumes.

The official tutorial of PyTorch DDP also employs spawn, and it seems reasonable as a minimum setup. As @li-li-github said in the description, MPI setup is not a trivial task, and I guess more users can try TorchDistributedTrial.

When we use spawn, I suppose we might require a database to store study like PyTorchLigntingCallback because child processes report intermediate values.

Hmm... In my understanding, InMemoryStorage can work with spawn since only the rank-0 process accesses the storage (including writing/reading intermediate values), and the other process receives broadcasted data from rank-0. I confirmed that the example worked with InMemoryStorage except for https://github.com/li-li-github/optuna-examples/blob/926a86faa5114076d77299520d7731f02fcd49bb/pytorch/pytorch_ddp_1machine.py#L224-L239. The parent process could not refer to the storage in the rank-0 process, and the optimization result lost.
So, I think it is reasonable to recommend RDBStorage instead of InMemoryStorage in this example.

Thus as TorchDistributedTrial with spawn example (not multiple GPUs, but can be supported), this PR would be helpful. So I would like to suggest supporting other backends if possible because Optuna CI doesn't have GPUs.

I agree. @li-li-github kindly added support for gloo, and I confirmed that it worked with four processes.

In summary, I'd like to proceed with the review.

[nzw0301]

@toshihikoyanase Thank you for your comments and for checking the InMemoryStorage case! I totally agree with you.

[li-li-github]

Thank you @toshihikoyanase for the detailed information.

InMemoryStorage can work with spawn since only the rank-0 process ...

I got you. It seems that study must be created under the condition of if rank == 0:

I can modify the code such that study is created under rank == 0, and use torch.multiprocessing.Manager to retrieve study after mp.spawn if you guys think that would make it a better example.

In run_optimize

def run_optimize(rank, world_size, device_ids, return_dict):
    ...

    if rank == 0:
        study = optuna.create_study(
            # storage="sqlite:///example.db",
            direction="maximize",
            study_name="pytorch_ddp",
            # load_if_exists=True,
        )
        study.optimize(
            partial(objective, device_id=devi),
            n_trials=N_TRIALS,
            timeout=300,
        )
        return_dict['study'] = study
        
    ...

Before & after mp.spawn

    manager = mp.Manager()
    return_dict = manager.dict()
    mp.spawn(
        run_optimize,
        args=(world_size, device_ids, return_dict),
        nprocs=world_size,
        join=True,
    )
    study = return_dict['study']

[li-li-github]

I just pushed the changes I mentioned above. Please let me know what you guys think! @toshihikoyanase @nzw0301

[github-actions]

This pull request has not seen any recent activity.

[toshihikoyanase]

Since this PR adds a new example file, we need to update the CI config to test it.
Could you add a new step to test this example with CPU next to Run multi-node examples?

optuna-examples/.github/workflows/pytorch.yml

Lines 50 to 56 in 4baba3a

	- name: Run multi-node examples
	run: |
	export OMPI_MCA_rmaps_base_oversubscribe=yes
	STORAGE_URL=sqlite:///example.db
	mpirun -n 2 -- python pytorch/pytorch_distributed_simple.py
	env:
	OMP_NUM_THREADS: 1

[toshihikoyanase]

CI failed due to the network errors. I'll retry the failed jobs after a while.

[nzw0301]

How about renaming the script name, such as pytorch_distributed_spawn.py? I think spawn works with mpi backend without GPUs; the example can support a wider use case.

[toshihikoyanase]

All CI jobs passed. The failure was due to GitHub Actions, not the example code.

[li-li-github]

Could you add a new step to test this example with CPU next to Run multi-node examples?

I have added a test step next to it.

How about renaming the script name, such as pytorch_distributed_spawn.py?

The file name has been changed.

Thank you @nzw0301 and @toshihikoyanase for the above suggestions. I also added an argument --master-port to avoid port number conflict in case the user wants to run more than 1 instance of this code.

[toshihikoyanase]

Thank you for your update. The example was tested in CI jobs as expected.

[toshihikoyanase]

The change almost looks good to me. I have two minor comments.

[li-li-github]

Thank you again for the suggestions. I have accepted all of them.

[github-actions]

This pull request has not seen any recent activity.

[toshihikoyanase]

I'm sorry for the delayed response. I'll approve this PR after the CI jobs pass.

[li-li-github]

Much appreciated!

[toshihikoyanase]

@li-li-github Sorry for bothering you, but a CI job failed due to the update of pytorch lightning integration. #172 resolved this problem, so could you rebase (or merge) the master, please?

AttributeError: 'Trainer' object has no attribute 'strategy'

[li-li-github]

All tests passed after I merged the main branch.

[toshihikoyanase]

Thank you for your update! LGTM!
I deeply appreciate your contribution.