MPI_T_finalize() gets SEGV when called

[rolfv]

This problem happens on the master only. Issue is not seen on 1.8 series.

[rvandevaart@drossetti-ivy5 nv]$ mpicc -Wall -g simplest_tool.c -o simplest_tool
[rvandevaart@drossetti-ivy5 nv]$ gdb ./simplest_tool
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6_4.1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.


This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /ivylogin/home/rvandevaart/tests/ompi-tests-git/nv/simplest_tool...done.
(gdb) run
Starting program: /ivylogin/home/rvandevaart/tests/ompi-tests-git/nv/simplest_tool 
[Thread debugging using libthread_db enabled]
[drossetti-ivy5.nvidia.com:27022] [[INVALID],INVALID] ORTE_ERROR_LOG: Error in file ../../../../../orte/mca/ess/pmi/ess_pmi_component.c at line 83
[New Thread 0x7ffff53db700 (LWP 27025)]
[New Thread 0x7fffeb2e2700 (LWP 27027)]
MPI_T_init_thread() ret=0, provided=0
MPI_T_cvar_get_num ret=0, num_cvar=1210
MPI_T_pvar_get_num ret=0, num_pvar=2
[Thread 0x7fffeb2e2700 (LWP 27027) exited]
[Thread 0x7fffec2e3700 (LWP 27026) exited]
[Thread 0x7ffff53db700 (LWP 27025) exited]

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff6f1dce4 in var_destructor (var=0x6ed260) at ../../../../opal/mca/base/mca_base_var.c:1735
1735            NULL != var->mbv_storage->stringval) {
(gdb) where
#0  0x00007ffff6f1dce4 in var_destructor (var=0x6ed260) at ../../../../opal/mca/base/mca_base_var.c:1735
#1  0x00007ffff6f19901 in opal_obj_run_destructors (object=0x6ed260) at ../../../../opal/class/opal_object.h:446
#2  0x00007ffff6f1bc41 in mca_base_var_finalize () at ../../../../opal/mca/base/mca_base_var.c:1041
#3  0x00007ffff6edb550 in opal_finalize_util () at ../../opal/runtime/opal_finalize.c:83
#4  0x00007ffff7ad3258 in ompi_mpi_finalize () at ../../ompi/runtime/ompi_mpi_finalize.c:443
#5  0x00007ffff7afbda5 in PMPI_Finalize () at pfinalize.c:46
#6  0x0000000000400820 in main (argc=1, argv=0x7fffffffe548) at simplest_tool.c:27
(gdb) 

[rolfv]

Here is the program:

/* Tools interface test */

#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>

void tool_test(void);

int main(int argc, char* argv[])
{
    MPI_Init(&argc, &argv);
    tool_test();
    MPI_Finalize();
    return 0;
}

void tool_test(void) {
  int ret;
  int num_cvar, num_pvar;
  int provided;

  ret =  MPI_T_init_thread(MPI_THREAD_SINGLE, &provided);

  printf("MPI_T_init_thread() ret=%d, provided=%d\n", ret, provided);
  ret = MPI_T_cvar_get_num(&num_cvar);
  printf("MPI_T_cvar_get_num ret=%d, num_cvar=%d\n", ret, num_cvar);
  ret = MPI_T_pvar_get_num(&num_pvar);
  printf("MPI_T_pvar_get_num ret=%d, num_pvar=%d\n", ret, num_pvar);

  /* If you comment this out, then the test will pass. */
  MPI_T_finalize();  
}

[hppritcha]

@hjelmn assigning this to Nathan.

[hjelmn]

More that likely this is due to a variable value being freed out from under the MCA variable system. Let me see if I can track down the culprit.

[hjelmn]

Hmm, the responsible component does not appear in my build. Can you print out the value of var->mbv_full_name?

[rolfv]

(gdb) up
#2  0x00007ff865d26e30 in var_destructor (var=0x26c5cb0) at ../../../../opal/mca/base/mca_base_var.c:1735
1735            NULL != var->mbv_storage->stringval) {
(gdb) print var
$1 = (mca_base_var_t *) 0x26c5cb0
(gdb) print var->mbv_full_name
$2 = 0x26c5e00 "mtl_ofi_provider"
(gdb) 

[jsquyres]

@yburette Can you have a look? It looks like the storage for this MCA parameter is being deleted too early.

[hjelmn]

Yup. No strings associated with an MCA variable should never be freed outside the MCA system.

[jsquyres]

@yburette This might fix the problem...?

diff --git a/ompi/mca/mtl/ofi/mtl_ofi_component.c b/ompi/mca/mtl/ofi/mtl_ofi_com
index b6004a7..40aee0c 100644
--- a/ompi/mca/mtl/ofi/mtl_ofi_component.c
+++ b/ompi/mca/mtl/ofi/mtl_ofi_component.c
@@ -161,7 +161,7 @@ ompi_mtl_ofi_component_init(bool enable_progress_threads,
      */
     domain_attr.threading        = FI_THREAD_ENDPOINT;
     domain_attr.control_progress = FI_PROGRESS_AUTO;
-    fabric_attr.prov_name        = ompi_mtl_ofi.provider_name;
+    fabric_attr.prov_name        = strdup(ompi_mtl_ofi.provider_name);
     hints.domain_attr            = &domain_attr;
     hints.fabric_attr            = &fabric_attr;

[jsquyres]

Fixed in 030e9bc.