std::getline Input Issue with in open MPI v5.0 series

[tavetius]

Thank you for taking the time to submit an issue!

Background information

Environment:

OS: Ubuntu 20.04.6 LTS
Compiler: g++ (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0

What version of Open MPI are you using? (e.g., v3.0.5, v4.0.2, git branch name and hash, etc.)

openmpi-5.0.0rc5, openmpi-5.0.0rc7, openmpi-5.0.0rc10, openmpi-5.0.0rc13

However, this works fine with openmpi-4.1.0.

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

./configure --enable-mpi-thread-multiple --with-ft=ulfm
make
sudo make install

Please describe the system on which you are running

• Operating system/version: OS: Ubuntu 20.04.6 LTS
• Computer hardware: Root - Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz, remote hosts - AMD EPYC 7V12 64-Core Processor

SSH works fine. Running command mpirun --allow-run-as-root --hostfile hostfile /home/ubuntu/test/mpi_test
As I said, this works fine with 4.1.0 but doesn't work with any version of 5.

---

Details of the problem

All processes reached the point of "before reading." This means that the processes were started and made it to that point in the program. The root process (Process 0) successfully printed the "Enter a string" prompt and read the input.

However, the processes are not proceeding past the broadcast, and after input and ENTER it just hangs. You need to kill it with ctrl+c and after that all processes managed to proceed past the broadcast point.

mpirun --allow-run-as-root --hostfile hostfile /home/ubuntu/test/mpi_test

[openmpi_getline.cpp.zip](https://github.com/open-mpi/ompi/files/12916363/openmpi_getline.cpp.zip)

[rhc54]

Have you tried removing that cin.sync call? From what I can find, that probably doesn't need to be there and may be causing problems, depending on how the terminal is configured by mpirun:

 cin.sync() would just clear out what's left. The only use I can think of for
sync() that can't be done with ignore is a replacement for system ("PAUSE");

[tavetius]

Yes sure. Results are the same. std::cin, std::getline, cin.getline, none of them works with our without cin.sync.

[ggouaillardet]

does rank 0 run on the same node on which mpirun in invoked?

can you

mpirun --allow-run-as-root --hostfile hostfile -np 1 strace /home/ubuntu/test/mpi_test

and post the last lines before you ctrl+c

[tavetius]

When the host is your own host it works, when hosts are other than your host (lovalhost is not included in hostlist) it doesn't work. But this works with openmpi 4.1.0, doesn't matter localhost is included in hostlist or not.

…

On Tue, Oct 17, 2023, 6:49 PM Gilles Gouaillardet ***@***.***> wrote: does rank 0 run on the same node on which mpirun in invoked? can you mpirun --allow-run-as-root --hostfile hostfile -np 1 strace /home/ubuntu/test/mpi_test and post the last lines before you ctrl+c — Reply to this email directly, view it on GitHub <#11990 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMJVNKBC5AEJVYZAHOFUKHDX72LH3AVCNFSM6AAAAAA6CEGNJCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRWGU3TONBVGI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[jsquyres]

Can you provide a small example program that shows the problem?

[ggouaillardet]

you can simply use cat:

if running rank 0 on the same host, it works as expected

$ echo foo | mpirun -np 1 cat
foo

if running rank 0 on an other host, it hangs

$ echo foo | mpirun -np 1 --host otherhost:1 cat

[ggouaillardet]

@rhc54 the issue is indeed trivial to reproduce as long as rank 0 does not run on the same node that prterun:

$ echo foo | prterun -np 1 --host otherhost:1 cat

I attached prterun under the debugger, and found that push_stdin() from src/mca/iof/hnp/iof_hnp.c is invoked (so far so good) but prte_mca_iof_hnp_component.procs is an empty list.

    /* do we already have this process in our list? */
    PMIX_LIST_FOREACH(proct, &prte_mca_iof_hnp_component.procs, prte_iof_proc_t)

Could you please have a look?

[rhc54]

Guess I was confused by the following in the original error report:

The root process (Process 0) successfully printed the "Enter a string" prompt and read the input.

From what you are saying, rank 0 doesn't actually read the input. I can take a look - fwiw, if prterun doesn't have any local procs, then IIRC that list should indeed be empty. However, prterun should xcast the input so daemons that do have local procs can pass it to their children.

[ggouaillardet]

Thanks Ralph.

If I read the code correctly, push_stdin() simply returns (and I guess the message is then lost) when the list is empty.

[rhc54]

Fix is in the referenced PR - will need to be backported to PRRTE v3.0 release branch and the OMPI submodule pointer updated