2507 Add Horovod unit tests

[Nic-Ma]

Fixes #2507 .

Description

This PR added the Horovod unit tests for distributed data parallel.

Status

Ready

Types of changes

• Non-breaking change (fix or new feature that would not break existing functionality).
• Breaking change (fix or new feature that would cause existing functionality to change).
• New tests added to cover the changes.
• Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
• Quick tests passed locally by running ./runtests.sh --quick --unittests.
• In-line docstrings updated.
• Documentation updated, tested make html command in the docs/ folder.

[Nic-Ma]

Hi @IsaacYangSLA and @wyli ,

I am trying to add distributed tests based on Horovod environment, but I got a question here:
As Horovod needs to execute its own command in shell to launch the program, how can we launch it in CI automatically?
For example: horovodrun -np 2 python test_evenly_divisible_all_gather_hvd.py
I checked our current distributed tests, seems it uses torch.multiprocessing.Process to launch in a python program instead of runningtorch.dist.launch command.

Thanks in advance.

[Nic-Ma]

/black

[Nic-Ma]

Hi @wyli @IsaacYangSLA , or maybe let's just put this Horovod test in the tests to manually run locally first, then try to integrate into CI later when we have more Horovod tests? Just like what we did for the PyTorch distributed tests before?

Thanks.

[Nic-Ma]

To run this new Horovod test locally, just follow:

1. Install Horovod:
   HOROVOD_NCCL_INCLUDE=/usr/include HOROVOD_NCCL_LIB=/usr/lib/x86_64-linux-gnu HOROVOD_GPU_OPERATIONS=NCCL \ HOROVOD_NCCL_LINK=SHARED pip install --no-cache-dir horovod
2. Execute on 2 GPUs in a single machine:
   horovodrun -np 2 python test_evenly_divisible_all_gather_hvd.py

Thanks.

[wyli]

@IsaacYangSLA could you please help review?