oshmem proc valgrind problem (?)

[jsquyres]

As noted on open-mpi/ompi-release#747, there appears to be a problem in oshmem freeing memory...?

@miked-mellanox Can you guys look into this one?

Invalid read of size 8
16:31:22 ==27367==    at 0x4FD7BE3: ompi_proc_world (proc.c:332)
16:31:22 ==27367==    by 0x4FDE5F1: ompi_mpi_finalize (ompi_mpi_finalize.c:142)
16:31:22 ==27367==    by 0x4C3311E: oshmem_shmem_finalize (oshmem_shmem_finalize.c:74)
16:31:22 ==27367==    by 0x4C35BAD: shmem_finalize (pshmem_finalize.c:34)
16:31:22 ==27367==    by 0x4C295CE: ??? (in /scrap/jenkins/jenkins/jobs/gh-ompi-release-pr/workspace/ompi_install1/lib/liboshmem.so.8.0.1)
16:31:22 ==27367==    by 0x4C58D80: ??? (in /scrap/jenkins/jenkins/jobs/gh-ompi-release-pr/workspace/ompi_install1/lib/liboshmem.so.8.0.1)
16:31:22 ==27367==    by 0x3D69035E21: exit (in /lib64/libc-2.12.so)
16:31:22 ==27367==    by 0x3D6901ED23: (below main) (in /lib64/libc-2.12.so)
16:31:22 ==27367==  Address 0xbb4b148 is 72 bytes inside a block of size 144 free'd
16:31:22 ==27367==    at 0x4A06484: free (vg_replace_malloc.c:468)
16:31:22 ==27367==    by 0x5AA605E: opal_free (malloc.c:184)
16:31:22 ==27367==    by 0x4C2FCB3: oshmem_proc_finalize (proc.c:198)
16:31:22 ==27367==    by 0x4C332C1: _shmem_finalize (oshmem_shmem_finalize.c:152)
16:31:22 ==27367==    by 0x4C330E2: oshmem_shmem_finalize (oshmem_shmem_finalize.c:69)
16:31:22 ==27367==    by 0x4C35BAD: shmem_finalize (pshmem_finalize.c:34)
16:31:22 ==27367==    by 0x4C295CE: ??? (in /scrap/jenkins/jenkins/jobs/gh-ompi-release-pr/workspace/ompi_install1/lib/liboshmem.so.8.0.1)
16:31:22 ==27367==    by 0x4C58D80: ??? (in /scrap/jenkins/jenkins/jobs/gh-ompi-release-pr/workspace/ompi_install1/lib/liboshmem.so.8.0.1)
16:31:22 ==27367==    by 0x3D69035E21: exit (in /lib64/libc-2.12.so)
16:31:22 ==27367==    by 0x3D6901ED23: (below main) (in /lib64/libc-2.12.so)

[mike-dubman]

@jsquyres - the issue is fixed in open-mpi/ompi-release#748 (comment)

[jsquyres]

Got it; thanks @miked-mellanox

[jsquyres]

Re-opening: @miked-mellanox the PR you cite is for v1.10. This valgrind issue came up on a PR for v2.x.

[mike-dubman]

@jsquyres -PR #747 referenced in this ticket has v1.10.0 label. probably wrong PR number?

[jsquyres]

I had the wrong link at first, sorry -- it's open-mpi/ompi-release#747.

[mike-dubman]

the valgrind errors failing this PR are as following and unrelated to SHMEM:

19:42:03 ==8414== Invalid read of size 4
19:42:03 ==8414==    at 0x3D6980A7B0: pthread_mutex_unlock (in /lib64/libpthread-2.12.so)
19:42:03 ==8414==    by 0x5341576: opal_libevent2022_event_del (event.c:2210)
19:42:03 ==8414==    by 0x6E57FD4: pdes (usock.c:316)
19:42:03 ==8414==    by 0x6E5A7B2: pmix_obj_run_destructors (pmix_object.h:449)
19:42:03 ==8414==    by 0x6E5C1F4: OPAL_PMIX_PMIX1XX_PMIx_Finalize (pmix_client.c:413)
19:42:03 ==8414==    by 0x6C24EAB: pmix1_client_finalize (pmix1_client.c:130)
19:42:03 ==8414==    by 0x6A1D599: rte_finalize (ess_pmi_module.c:412)
19:42:03 ==8414==    by 0x4FE459B: orte_finalize (orte_finalize.c:72)
19:42:03 ==8414==    by 0x4C6A105: ompi_mpi_finalize (ompi_mpi_finalize.c:440)
19:42:03 ==8414==    by 0x4C9A8B0: PMPI_Finalize (pfinalize.c:44)
19:42:03 ==8414==    by 0x40088F: main (hello_c.c:24)
19:42:03 ==8414==  Address 0x5c8a9f0 is 16 bytes inside a block of size 40 free'd
19:42:03 ==8414==    at 0x4A06484: free (vg_replace_malloc.c:468)
19:42:03 ==8414==    by 0x5341BA6: opal_libevent2022_event_base_free (event.c:790)
19:42:03 ==8414==    by 0x6E5C19A: OPAL_PMIX_PMIX1XX_PMIx_Finalize (pmix_client.c:407)
19:42:03 ==8414==    by 0x6C24EAB: pmix1_client_finalize (pmix1_client.c:130)
19:42:03 ==8414==    by 0x6A1D599: rte_finalize (ess_pmi_module.c:412)
19:42:03 ==8414==    by 0x4FE459B: orte_finalize (orte_finalize.c:72)
19:42:03 ==8414==    by 0x4C6A105: ompi_mpi_finalize (ompi_mpi_finalize.c:440)
19:42:03 ==8414==    by 0x4C9A8B0: PMPI_Finalize (pfinalize.c:44)
19:42:03 ==8414==    by 0x40088F: main (hello_c.c:24)
19:42:03 ==8414== 
19:42:03 ==8414== Invalid read of size 4
19:42:03 ==8414==    at 0x3D6980A360: __pthread_mutex_unlock_full (in /lib64/libpthread-2.12.so)
19:42:03 ==8414==    by 0x5341576: opal_libevent2022_event_del (event.c:2210)
19:42:03 ==8414==    by 0x6E57FD4: pdes (usock.c:316)
19:42:03 ==8414==    by 0x6E5A7B2: pmix_obj_run_destructors (pmix_object.h:449)
19:42:03 ==8414==    by 0x6E5C1F4: OPAL_PMIX_PMIX1XX_PMIx_Finalize (pmix_client.c:413)
19:42:03 ==8414==    by 0x6C24EAB: pmix1_client_finalize (pmix1_client.c:130)
19:42:03 ==8414==    by 0x6A1D599: rte_finalize (ess_pmi_module.c:412)
19:42:03 ==8414==    by 0x4FE459B: orte_finalize (orte_finalize.c:72)
19:42:03 ==8414==    by 0x4C6A105: ompi_mpi_finalize (ompi_mpi_finalize.c:440)
19:42:03 ==8414==    by 0x4C9A8B0: PMPI_Finalize (pfinalize.c:44)
19:42:03 ==8414==    by 0x40088F: main (hello_c.c:24)
19:42:03 ==8414==  Address 0x5c8a9f0 is 16 bytes inside a block of size 40 free'd
19:42:03 ==8414==    at 0x4A06484: free (vg_replace_malloc.c:468)
19:42:03 ==8414==    by 0x5341BA6: opal_libevent2022_event_base_free (event.c:790)
19:42:03 ==8414==    by 0x6E5C19A: OPAL_PMIX_PMIX1XX_PMIx_Finalize (pmix_client.c:407)
19:42:03 ==8414==    by 0x6C24EAB: pmix1_client_finalize (pmix1_client.c:130)
19:42:03 ==8414==    by 0x6A1D599: rte_finalize (ess_pmi_module.c:412)
19:42:03 ==8414==    by 0x4FE459B: orte_finalize (orte_finalize.c:72)
19:42:03 ==8414==    by 0x4C6A105: ompi_mpi_finalize (ompi_mpi_finalize.c:440)
19:42:03 ==8414==    by 0x4C9A8B0: PMPI_Finalize (pfinalize.c:44)
19:42:03 ==8414==    by 0x40088F: main (hello_c.c:24)

[mike-dubman]

@jsquyres - it was my mistake to post SHMEM related valgrind output which was belonging to other PR. This ticket can be closed, and new one opened for pmixX valgring errors from PR open-mpi/ompi-release#747

[jsquyres]

Ok, got it -- there are actual valgrind issues for open-mpi/ompi-release#747.

But isn't there also an oshmem valgrind issue? I.e., is this issue still valid?

[mike-dubman]

oshmem valgrind issue exists only in v1.10 (pr submitted), we forgot to port fix from master :(

[jsquyres]

Ah, ok -- it was just an errant paste. Now I got it. 😄 Ok, I'll close this. Thanks!