Open MPI was unable to obtain the username in order to create a path

[sjackman]

Due to some unresolved misconfiguration, whoami returns a number rather than a name. This situation causes OpenMPI to give the error:

Open MPI was unable to obtain the username in order to create a path 
for its required temporary directories. This type of error is usually 
caused by a transient failure of network-based authentication services 
(e.g., LDAP or NIS failure due to network congestion), but can also be 
an indication of system misconfiguration. 
Please consult your system administrator about these issues and try 
again.

This issue has been reported here:
https://www.open-mpi.org/community/lists/users/2014/10/25485.php

The workaround is to recompile OpenMPI using --disable-getpwuid. Recompiling OpenMPI is a rather heavy weight workaround, when all that is needed is a temporary directory.

Could the above error be downgraded to a warning instead?

Is there a workaround that doesn't require recompiling OpenMPI?

Thanks!

[tseemann]

I also get this error, which I think is due to LDAP or Active Directory authentication on our system.

If it's only for temporary folders then I expect this could be simplified.

[rhc54]

It depends on the system environment. We no longer use getuid for the tmpdir, though I found that this patch had not been backported to the 2.x series. I'm taking care of that now.

However, we still do use getpwuid when launching via rsh/ssh unless you explicitly disable it. The rationale is simply that we shouldn't be attempting to ssh a daemon onto a remote node if the system has no idea who you are.

Looking around the code, I can't find any reference to this error message in the 2.x code base, so I'm assuming you are referring to some earlier series? Could you please clarify what version of OMPI you are talking about?

[sjackman]

The version of open-mpi in Homebrew is 1.10.2 I see there's a 1.10.3 release as well.

[rhc54]

The 1.10 series is essentially closed as it has been replaced by the 2.0 series. So I won't be backporting this patch to 1.10 - it will just go into an upcoming 2.x release as scheduled by the release managers for that series.

[sjackman]

Thanks!

However, we still do use getpwuid when launching via rsh/ssh unless you explicitly disable it. The rationale is simply that we shouldn't be attempting to ssh a daemon onto a remote node if the system has no idea who you are.

I understand that it will still fail then when using ssh and getpwuid fails? What does it do when configured with --disable-getpwuid? Does that also disable ssh?

Why is getpwuid required to ssh? I haven't found that to be the case with openssh.

Edit: getpwuid is required to ssh:

❯❯❯ ssh foo
No user exists for uid 1078
❯❯❯ echo $?
255

Can you think of any simple hack/workaround to use open-mpi with ssh on a system where getpwuid does not work? I've got a feeling that I'm going to be heading down the bunny hole of configuring openldap and glibc. Ugh.

[ggouaillardet]

@sjackman ssh does require getpwuid, so if your system does not support getpwuid, you have to use an other command.
for example, you can

mpirun --mca orte_rsh_agent /usr/bin/mysshthatdoesnotrequiregetpwuid ...

that being said, that will not work because of the following issue

fwiw, i am surprised getpwuid does not work on your system and you have Open MPI use ssh to spawn its daemons.

@rhc54 i hit a case that sits in the middle ...
i built Open MPI in 32 bits mode on a 64 bits system, and it seems nss plugins are only available in 64 bits. bottom line, ssh works just fine (it is a 64 bits app), but plm/rsh fails to getpwuid (it is a 32 bits library)

as a proof of concept, the following inline patch does the trick

diff --git a/orte/mca/plm/rsh/plm_rsh_module.c b/orte/mca/plm/rsh/plm_rsh_module.c
index ed8b7f2..cd03fcf 100644
--- a/orte/mca/plm/rsh/plm_rsh_module.c
+++ b/orte/mca/plm/rsh/plm_rsh_module.c
@@ -1458,11 +1458,13 @@ static int setup_shell(orte_plm_rsh_shell_t *rshell,
             /* This user is unknown to the system. Therefore, there is no reason we
              * spawn whatsoever in his name. Give up with a HUGE error message.
              */
-            orte_show_help( "help-plm-rsh.txt", "unknown-user", true, (int)getuid() );
-            return ORTE_ERR_FATAL;
+            // orte_show_help( "help-plm-rsh.txt", "unknown-user", true, (int)getuid() );
+            // return ORTE_ERR_FATAL;
+            param = NULL;
+        } else {
+            param = p->pw_shell;
+            local_shell = find_shell(p->pw_shell);
         }
-        param = p->pw_shell;
-        local_shell = find_shell(p->pw_shell);
     }
 #endif

i am unsure of how we should deal with this specific case.

• do nothing (this is a site issue)
• do not fail if getpwuid fails
• add a MCA param not to fail if getpwuid fails

any thoughts ? @jsquyres ?

[rhc54]

FWIW: I went ahead and modified the rsh/ssh code so it doesn't fail if getpwuid fails. We use the same code for both rsh and ssh, so even though ssh won't work, we should let the rsh case continue to try.

I'll add the new change to the release PR when it passes CI.

[sjackman]

Thanks! I'd like to get ssh up and working, but this provides me a fallback plan.