OPAL ERROR: Bad parameter in file ../../opal/dss/dss_load_unload.c while running

[rolfv]

See the example below to see this error message. I only observe this running Open MPI 2.0.0. Even though the message pops out, everything runs fine. I did not see this while running on a single node. I also did not see it while running with -np 2 on two nodes.

[rvandevaart@drossetti-ivy4 pt2pt]$ mpirun --version
mpirun (Open MPI) 2.0.0a1

Report bugs to http://www.open-mpi.org/community/help/
[rvandevaart@drossetti-ivy4 pt2pt]$ mpirun -np 4 -host drossetti-ivy4,drossetti-ivy5 send
[drossetti-ivy5.nvidia.com:13465] OPAL ERROR: Bad parameter in file ../../opal/dss/dss_load_unload.c at line 293
[drossetti-ivy5.nvidia.com:13464] OPAL ERROR: Bad parameter in file ../../opal/dss/dss_load_unload.c at line 293
It is not unusual for this test to take several minutes,
particularly on slow networks.
Please be patient.
NOTICE: Using max message size: 10485760
Progress: [==============================================================]
[rvandevaart@drossetti-ivy4 pt2pt]$ 

[rhc54]

I'm trying to remember when it was, but I do know we fixed something like that recently. Just may not have made it across yet

[ggouaillardet]

@rolfv i could not reproduce this
which btls are you using ?

[rolfv]

I am seeing this with self,sm,tcp or self,smcuda,openib. Here is an example from my MTT results.

mpirun --host
drossetti-ivy4,drossetti-ivy4,drossetti-ivy4,drossetti-ivy5,drossetti-ivy5,drossetti-ivy5 -np 6
--mca btl sm,tcp,self --mca btl_openib_warn_default_gid_prefix 0 --prefix
/ivylogin/home/rvandevaart/mtt-git/scratches/mtt-scratch-1/installs/do8o/install ./c_hello 

[drossetti-ivy5.nvidia.com:03490] OPAL ERROR: Bad parameter in file dss/dss_load_unload.c at line
293
[drossetti-ivy5.nvidia.com:03491] OPAL ERROR: Bad parameter in file dss/dss_load_unload.c at line
293
[drossetti-ivy5.nvidia.com:03490] OPAL ERROR: Bad parameter in file dss/dss_load_unload.c at line
293
[drossetti-ivy5.nvidia.com:03489] OPAL ERROR: Bad parameter in file dss/dss_load_unload.c at line
293[drossetti-ivy5.nvidia.com:03491] OPAL ERROR: Bad parameter in file dss/dss_load_unload.c at line
293
[drossetti-ivy5.nvidia.com:03489] OPAL ERROR: Bad parameter in file dss/dss_load_unload.c at line
293
Hello, C world!  I am 0 of 6
Hello, C world!  I am 1 of 6Hello, C world!  I am 2 of 6
Hello, C world!  I am 3 of 6
Hello, C world!  I am 4 of 6
Hello, C world!  I am 5 of 6

[ggouaillardet]

@rolfv here is a one-off patch for the v2.x branch

diff --git a/ompi/proc/proc.c b/ompi/proc/proc.c
index 2e61705..ba38f25 100644
--- a/ompi/proc/proc.c
+++ b/ompi/proc/proc.c
@@ -142,12 +142,12 @@ int ompi_proc_complete_init(void)
     uint16_t u16, *u16ptr;

     OPAL_THREAD_LOCK(&ompi_proc_lock);
-    u16ptr = &u16;

     OPAL_LIST_FOREACH(proc, &ompi_proc_list, ompi_proc_t) {
         if (OMPI_CAST_RTE_NAME(&proc->super.proc_name)->vpid != OMPI_PROC_MY_NAME->vpid) {
             /* get the locality information - all RTEs are required
             * to provide this information at startup */
+            u16ptr = &u16;
             OPAL_MODEX_RECV_VALUE_OPTIONAL(ret, OPAL_PMIX_LOCALITY, &proc->super.proc_name, &u16ptr, OPAL_UINT16);
             if (OPAL_SUCCESS != ret) {
                 proc->super.proc_flags = OPAL_PROC_NON_LOCAL;

@rhc54 master is a bit different ...
i fixed a similar error (i did not run into though ...) at 57ecce4
i did not fully investigate (only code review), but so far, one option is that
OPAL_MODEX_RECV_VALUE_OPTIONAL always success on master, and an other option
is that such stuff is done in ompi_proc_complete_init_single (no loop, so u16ptr is always set```

let me know how you want to move forward (e.g. i can make a PR for this one off patch, or i can make a PR with some more backports from master)

[rhc54]

I’m not sure I understand how that could have an impact, but I’d let Rolf test it and let’s see. Since we aren’t seeing the problem on master, bringing 2.x closer to master makes sense.

On Sep 28, 2015, at 7:50 PM, Gilles Gouaillardet notifications@github.com wrote:

@rolfv https://github.com/rolfv here is a one-off patch for the v2.x branch

diff --git a/ompi/proc/proc.c b/ompi/proc/proc.c
index 2e61705..ba38f25 100644
--- a/ompi/proc/proc.c
+++ b/ompi/proc/proc.c
@@ -142,12 +142,12 @@ int ompi_proc_complete_init(void)
uint16_t u16, *u16ptr;

 OPAL_THREAD_LOCK(&ompi_proc_lock);

• u16ptr = &u16;

OPAL_LIST_FOREACH(proc, &ompi_proc_list, ompi_proc_t) {
if (OMPI_CAST_RTE_NAME(&proc->super.proc_name)->vpid != OMPI_PROC_MY_NAME->vpid) {
/* get the locality information - all RTEs are required
* to provide this information at startup */

•        u16ptr = &u16;
       OPAL_MODEX_RECV_VALUE_OPTIONAL(ret, OPAL_PMIX_LOCALITY, &proc->super.proc_name, &u16ptr, OPAL_UINT16);
       if (OPAL_SUCCESS != ret) {
           proc->super.proc_flags = OPAL_PROC_NON_LOCAL;
  

  @rhc54 https://github.com/rhc54 master is a bit different ...
  i fixed a similar error (i did not run into though ...) at 57ecce4 57ecce4
  i did not fully investigate (only code review), but so far, one option is that
  OPAL_MODEX_RECV_VALUE_OPTIONAL always success on master, and an other option
  is that such stuff is done in ompi_proc_complete_init_single (no loop, so u16ptr is always set```

let me know how you want to move forward (e.g. i can make a PR for this one off patch, or i can make a PR with some more backports from master)

—
Reply to this email directly or view it on GitHub #935 (comment).

[rhc54]

@ggouaillardet Just saw your commit to trunk, and I agree that your patch is required. Please PR it over to 2.x.

Thanks
Ralph

[ggouaillardet]

@rhc54 done in open-mpi/ompi-release#617

[rolfv]

The one off patch from above has fixed the issue I was seeing. No more messages are seen so that looks good to me.

[rolfv]

Just checked my overnight runs and this error is gone on v2.x. I am therefore closing the issue. Thanks for the fix.