Use nccl in CuPy

[shu65]

Some errors may occur when NCCL versions used for ChainerMN and CuPy are different. So I modify ChainerMN to use nccl in Cupy and remove NCCL wrapper in ChainerMN.

[kuenishi]

Whole sentence would be like this:

Users who want to try ChainerMN in CPU-only environment may skip installation of CuPy. Non-GPU set up may not be performant as GPU-enabled set up, but would would be useful for testing or debugging training program in non-GPU environment such as laptops or CI jobs.

[shu65]

Thank you for your comment. I will fix it.

[kuenishi]

Current version of ChainerMN does not need --no-nccl flag for CPU-only environment at installation any more. It would be just ignored.

[shu65]

I will fix it.

[kuenishi]

I think CuPy must be properly installed with NCCL enabled. I'd recommend adding a sentence how to check CuPy installation, before ChainerMN installation like python -c 'from cupy import nccl' . Probably add after --no-nccl note?

[kuenishi]

Also we'd need update on a nccl check at _base.py :

        if use_nccl and not nccl._available:
            raise RuntimeError(
                'NCCL is not available. '
                'Please confirm that NCCL can be found by dynamic linkers, '
                'and ChainerMN is installed without --no-nccl flag.'
            )

[kuenishi]

My comments are mainly on English, while the code seems nice.

[kuenishi]

Changes are good; while I tested again and unfortunately found potential fix for non-nccl checking:

>>> import chainermn
>>> c = chainermn.create_communicator('pure_nccl')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/kuenishi/src/chainermn/chainermn/communicators/__init__.py", line 77, in create_communicator
    return PureNcclCommunicator(mpi_comm=mpi_comm)
  File "/home/kuenishi/src/chainermn/chainermn/communicators/pure_nccl_communicator.py", line 12, in __init__
    if nccl.get_version() < 2000:
AttributeError: module 'chainermn.nccl' has no attribute 'get_version'
>>> 

This is because of nccl version checking nccl.get_version coming before initializing CommunicatorBase initialization which includes nccl check. If we keep supporting NCCL 1.x support in other communicators than PureNcclCommunicator I'd recommend putting version check later than CommunicatorBase init. Otherwise moving version check to CommunicatorBase init would make sense more.

[shu65]

Thank you for testing. I confirmed this error.
Because the first idea seems to be better, I will adopt it.