MTT errors only on master

[adrianreber]

My MTT runs on a x86_64 and a ppc64 system (both standalone, not connected to each other). Two tests from the ibm test suite are failing. The failure does not happen on the v2.x branch, but only on the master branch. The tests are failing with gcc 5.3.1 as well as with gcc-6.0 (pre-release):

https://mtt.open-mpi.org/index.php?do_redir=2277
https://mtt.open-mpi.org/index.php?do_redir=2278

The failure is in so far a bit strange as it claims that the binaries do not exist;

--------------------------------------------------------------------------
mpirun was unable to find the specified executable file, and therefore
did not launch the job.  This error was first reported for process
rank 0; it may have occurred for other processes as well.

NOTE: A common cause for this error is misspelling a mpirun command
      line parameter option (remember that mpirun interprets the first
      unrecognized command line token as the executable).

Node:       rhlx01
Executable: intercomm_create
--------------------------------------------------------------------------2 total processes failed
to start

The test, however, exists on the filesystem and can also be executed standalone.

~/mtt/mtt-scratch/installs/ompi-nightly-master--gcc_warnings--dev-3647-g361f931/tests/ibm/ibm/dynamic $ file intercomm_create
intercomm_create: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=da53297ab42c0c3bec7bbe9a0bb61784d9a49999, not stripped
~/mtt/mtt-scratch/installs/ompi-nightly-master--gcc_warnings--dev-3647-g361f931/tests/ibm/ibm/dynamic $ ./intercomm_create 
b: MPI_Intercomm_create( intra, 0, intra, MPI_COMM_NULL, 201, &inter) [rank 1]
a: MPI_Intercomm_create( ab_intra, 0, ac_intra, 1, 201, &inter) (0)
c: MPI_Intercomm_create( MPI_COMM_WORLD, 0, intra, 0, 201, &inter) [rank 1]
a: intercomm_create (0)
a: barrier on inter-comm - before
c: intercomm_create (0)
c: barrier on inter-comm - before
b: intercomm_create (0)
b: barrier on inter-comm - before
a: barrier on inter-comm - after
c: barrier on inter-comm - after
b: barrier on inter-comm - after
a: intercomm_merge(0) (0) [rank 0]
c: intercomm_merge(0) (0) [rank 0]
b: intercomm_merge(1) (0) [rank 2]
a: barrier (0)
c: barrier (0)
b: barrier (0)

As it only seems to happen on master it doesn't look like a configuration error. If it is a MTT configuration error I am happy to get a hint how to correctly configure MTT for those two tests.

[ggouaillardet]

@adrianreber
MTT does

mpirun ./dynamic/intercomm_create

and you likely do

cd dynamic
mpirun ./intercomm_create

here is a trivial program

#include <stdio.h>

#include <mpi.h>

int main(int argc, char *argv[]) {
    MPI_Init(&argc, &argv);
    printf("running: %s\n", argv[0]);
    MPI_Finalize();
    return 0;
}

when i

mpirun -np 1 ./a.out

on v1.10 and v2.x, output is

running: ./a.out

but on master, output is

running: a.out

note the missing ./

@rhc54 any idea of what could have caused the change ?
is #1451 really necessary or does it fix a problem that should have never happened ?

[ggouaillardet]

@rhc54
got it, 4d0cc27

among other things

commit 4d0cc27eb7c96a3ad50abf309b6f3186e2f05f58
Author: Ralph Castain <rhc@open-mpi.org>
Date:   Sat Mar 5 21:47:42 2016 -0800

    Update the singularity support to match that of the latest singularity master. Remove the restriction on shared memory components by instructing singularity to not isolate the PID space. Add a new schizo API to allow setting up the original app_context. Ensure the container is installed prior to execution.

diff --git a/orte/tools/orterun/orterun.c b/orte/tools/orterun/orterun.c
index d63008e..6dde2ed 100644
--- a/orte/tools/orterun/orterun.c
+++ b/orte/tools/orterun/orterun.c
@@ -1670,6 +1676,8 @@ static int create_app(int argc, char* argv[],
         rc = ORTE_ERR_NOT_FOUND;
         goto cleanup;
     }
+    free(app->argv[0]);
+    app->argv[0] = opal_basename(app->app);

     /* if this is a Java application, we have a bit more work to do. Such
      * applications actually need to be run under the Java virtual machine

if i remove these two lines, it works as before, and #1451 is useless.
what was the rationale for using the basename of the app ?