FORCE-TERMINATE AT Data unpack would read past end of buffer:-26 - error grpcomm_direct.c

[ddkang]

Environment:

1. Framework: PyTorch
2. Framework version: 1.0.1.post2
3. Horovod version: 0.16.1
4. MPI version: 4.0.0
5. CUDA version: Cuda compilation tools, release 9.0, V9.0.176
6. NCCL version: N/A
7. Python version: 3.5.2
8. OS and version: Ubuntu 16.04

Checklist:

1. Did you search issues to find if somebody asked this question before?

Yes, but there was no resolution: #368

Bug report:

I tried running the pytorch_imagenet_resnet50.py script and got the following error:

--------------------------------------------------------------------------
An internal error has occurred in ORTE:

[[25215,0],1] FORCE-TERMINATE AT Data unpack would read past end of buffer:-26 - error grpcomm_direct.c(359)

This is something that should be reported to the developers.
--------------------------------------------------------------------------
[future5.stanford.edu:12508] [[25215,0],1] ORTE_ERROR_LOG: Data unpack would read past end of buffer in file grpcomm_direct.c at line 355

I made sure to configure and install OpenMPI as in the dockerfile.

[abditag2]

It does seem that your MPI configuration has some issue. Take a look at this and let me know if it provides any help: open-mpi/ompi#4437

[ddkang]

I am not using hwloc, I didn't see other solutions?

[alsrgv]

@ddkang, can you share output of dpkg -l | grep hwloc?

[ddkang]

@alsrgv

future4:

ii  hwloc-nox                              1.11.2-3                                   amd64        Hierarchical view of the machine - non-X version of utilities
ii  libhwloc-dev:amd64                     1.11.2-3                                   amd64        Hierarchical view of the machine - static libs and headers
ii  libhwloc-plugins                       1.11.2-3                                   amd64        Hierarchical view of the machine - plugins
ii  libhwloc5:amd64                        1.11.2-3                                   amd64        Hierarchical view of the machine - shared libs

future5:

ii  hwloc-nox                             1.11.2-3                                   amd64        Hierarchical view of the machine - non-X version of utilities
ii  libhwloc-dev:amd64                    1.11.2-3                                   amd64        Hierarchical view of the machine - static libs and headers
ii  libhwloc-plugins                      1.11.2-3                                   amd64        Hierarchical view of the machine - plugins
ii  libhwloc5:amd64                       1.11.2-3                                   amd64        Hierarchical view of the machine - shared libs

Please let me know if you need anything else!

[alsrgv]

@ddkang, sorry for the delayed response. Looks like this is the same version as the one flagged in open-mpi/ompi#4437 as buggy. Could you try uninstalling hwloc (apt purge hwloc-nox libhwloc-dev libhwloc-plugins libhwloc5) and reinstalling Open MPI?

[ddkang]

That worked, thank you! It would be nice to have this in the documentation.

[alsrgv]

Good idea, added to #1084. Will close this issue.