Using memory after free

[jsquyres]

As noticed by Jenkins / @miked-mellanox in open-mpi/ompi-release#747 (comment), there is a case of using memory after it has been freed in the v2.x branch (I notice that this is not occurring on master). I'm able to replicate this problem in my own local setup with:

$ mpirun -np 1 valgrind examples/hello_c

It doesn't seem to be causing problems with MTT runs, but we might be just getting lucky.

Here's a snipit of the valgrind output.

19:42:03 ==8414== Invalid read of size 4
19:42:03 ==8414==    at 0x3D6980A7B0: pthread_mutex_unlock (in /lib64/libpthread-2.12.so)
19:42:03 ==8414==    by 0x5341576: opal_libevent2022_event_del (event.c:2210)
19:42:03 ==8414==    by 0x6E57FD4: pdes (usock.c:316)
19:42:03 ==8414==    by 0x6E5A7B2: pmix_obj_run_destructors (pmix_object.h:449)
19:42:03 ==8414==    by 0x6E5C1F4: OPAL_PMIX_PMIX1XX_PMIx_Finalize (pmix_client.c:413)
19:42:03 ==8414==    by 0x6C24EAB: pmix1_client_finalize (pmix1_client.c:130)
19:42:03 ==8414==    by 0x6A1D599: rte_finalize (ess_pmi_module.c:412)
19:42:03 ==8414==    by 0x4FE459B: orte_finalize (orte_finalize.c:72)
19:42:03 ==8414==    by 0x4C6A105: ompi_mpi_finalize (ompi_mpi_finalize.c:440)
19:42:03 ==8414==    by 0x4C9A8B0: PMPI_Finalize (pfinalize.c:44)
19:42:03 ==8414==    by 0x40088F: main (hello_c.c:24)
19:42:03 ==8414==  Address 0x5c8a9f0 is 16 bytes inside a block of size 40 free'd
19:42:03 ==8414==    at 0x4A06484: free (vg_replace_malloc.c:468)
19:42:03 ==8414==    by 0x5341BA6: opal_libevent2022_event_base_free (event.c:790)
19:42:03 ==8414==    by 0x6E5C19A: OPAL_PMIX_PMIX1XX_PMIx_Finalize (pmix_client.c:407)
19:42:03 ==8414==    by 0x6C24EAB: pmix1_client_finalize (pmix1_client.c:130)
19:42:03 ==8414==    by 0x6A1D599: rte_finalize (ess_pmi_module.c:412)
19:42:03 ==8414==    by 0x4FE459B: orte_finalize (orte_finalize.c:72)
19:42:03 ==8414==    by 0x4C6A105: ompi_mpi_finalize (ompi_mpi_finalize.c:440)
19:42:03 ==8414==    by 0x4C9A8B0: PMPI_Finalize (pfinalize.c:44)
19:42:03 ==8414==    by 0x40088F: main (hello_c.c:24)
19:42:03 ==8414== 
19:42:03 ==8414== Invalid read of size 4
19:42:03 ==8414==    at 0x3D6980A360: __pthread_mutex_unlock_full (in /lib64/libpthread-2.12.so)
19:42:03 ==8414==    by 0x5341576: opal_libevent2022_event_del (event.c:2210)
19:42:03 ==8414==    by 0x6E57FD4: pdes (usock.c:316)
19:42:03 ==8414==    by 0x6E5A7B2: pmix_obj_run_destructors (pmix_object.h:449)
19:42:03 ==8414==    by 0x6E5C1F4: OPAL_PMIX_PMIX1XX_PMIx_Finalize (pmix_client.c:413)
19:42:03 ==8414==    by 0x6C24EAB: pmix1_client_finalize (pmix1_client.c:130)
19:42:03 ==8414==    by 0x6A1D599: rte_finalize (ess_pmi_module.c:412)
19:42:03 ==8414==    by 0x4FE459B: orte_finalize (orte_finalize.c:72)
19:42:03 ==8414==    by 0x4C6A105: ompi_mpi_finalize (ompi_mpi_finalize.c:440)
19:42:03 ==8414==    by 0x4C9A8B0: PMPI_Finalize (pfinalize.c:44)
19:42:03 ==8414==    by 0x40088F: main (hello_c.c:24)
19:42:03 ==8414==  Address 0x5c8a9f0 is 16 bytes inside a block of size 40 free'd
19:42:03 ==8414==    at 0x4A06484: free (vg_replace_malloc.c:468)
19:42:03 ==8414==    by 0x5341BA6: opal_libevent2022_event_base_free (event.c:790)
19:42:03 ==8414==    by 0x6E5C19A: OPAL_PMIX_PMIX1XX_PMIx_Finalize (pmix_client.c:407)
19:42:03 ==8414==    by 0x6C24EAB: pmix1_client_finalize (pmix1_client.c:130)
19:42:03 ==8414==    by 0x6A1D599: rte_finalize (ess_pmi_module.c:412)
19:42:03 ==8414==    by 0x4FE459B: orte_finalize (orte_finalize.c:72)
19:42:03 ==8414==    by 0x4C6A105: ompi_mpi_finalize (ompi_mpi_finalize.c:440)
19:42:03 ==8414==    by 0x4C9A8B0: PMPI_Finalize (pfinalize.c:44)
19:42:03 ==8414==    by 0x40088F: main (hello_c.c:24)

[rhc54]

@jsquyres if it is solely on 2.x, then syncing the PMIx library with what is on trunk should resolve the problem. My guess is that 2.x is simply lagging behind the master in terms of pmix updates.

[hppritcha]

Looks like 7434c47 and 30ba00e and it looks like there's some activity in 267ca8f as well that should be brought over. Looks like there are multiple commits in pr #969 that need to come over.

I think we're going to need to do an audit soon of all the pmix/ompi pmix associated code commits on master that have not made it in to v2.x yet. Otherwise what's going to start happening is we find some it ever more difficult to patch back valgrind/coverity fixes in to v2.x without a lot of git archeological digging.

[rhc54]

agreed. I plan to provide a PR that will include the final PMIx 1.1.0 release, plus any associated changes.

[jsquyres]

@rhc54 Do you have a timeline for syncing a new PMIx to v2.x?

[jsquyres]

This issue is now effectively a duplicate of #1218. Closing.