intel/MPI_Waitsome_p_f fails with assert

[mike-dubman]

salloc -p pvegas --ntasks-per-node 5 -N 12 /hpc/mtr_scrap/users/mtt/scratch/fca/20150222_203007_11848_81648_vegas19/installs/ixdj/install/bin/mpirun -np 60 -mca btl_openib_if_include mlx5_0:1  -x MXM_IB_PORTS=mlx5_0:1 --map-by node --display-map --report-bindings   /hpc/mtr_scrap/users/mtt/scratch/fca/20150222_203007_11848_81648_vegas19/installs/ixdj/tests/intel/intel_tests/src/MPI_Waitsome_p_f
...
 MPITEST_INFO (         0): Starting test MPI_Waitsome_p
MPI_Waitsome_p_f: ../../../../opal/class/opal_list.h:600: opal_list_prepend: Assertion `0 == item->opal_list_item_refcount' failed.
 MPITEST_ERROR(         0): MPI_WAITSOME returned        18
 MPITEST_ERROR(         0): MPI_ERR_IN_STATUS: error code in status
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 0 on node vegas19 exited on signal 11 (Segmentation fault).

[mike-dubman]

same for MPI_Waitany_f test

[jsquyres]

Can you post the backtrace? All this shows is that an assertion fails somewhere in the code.

[mike-dubman]

no backtrace or core file generated :(
will try to extract re-generate one.

is there way that mpirun can intercept segv and print stacktrace?

[goodell]

@miked-mellanox Sounds like your ulimit -c is not set properly, or you have bad settings for some core file-related entry in /proc/sys: http://man7.org/linux/man-pages/man5/core.5.html

[jsquyres]

@miked-mellanox It's an assertion failure, so mpirun won't catch it.

[bosilca]

This might be of interest: I can't replicate this issue on a non-IB system.

On Mon, Feb 23, 2015 at 2:51 PM, Jeff Squyres notifications@github.com
wrote:

@miked-mellanox https://github.com/miked-mellanox It's an assertion
failure, so mpirun won't catch it.

—
Reply to this email directly or view it on GitHub
#414 (comment).

[jsquyres]

@miked-mellanox What PML are you using for this test -- yalla or ob1?

[mike-dubman]

tried both - same result.it is reproducing on two nodes, ppn=1 as well. and what is strange, that it may work or fail depending on the length of cmd line param passed to mpirun under slurm ....
(occasionally works when some unrelated mca params added or removed)
@yosefe is looking into it

[jsquyres]

@miked-mellanox Per #424, did this end up being a yalla issue (and not an ob1 issue)?

[mike-dubman]

we are looking into ob1 now to rule it out.
it was reproducing with ob1 as well