Assertion `0 != r_mkey->len' failed in spml_yoda.c:1113:mca_spml_yoda_get

[bertwesarg]

I have now a reproducer which fails for me with 1.8.4 and dev-791-g03c2adf but works with 1.8.2. SHMMAX is:

$ cat /proc/sys/kernel/shmmax
268435456

config.log and ompi_info --all is attached for both 1.8.2 and dev-791-g03c2adf.

$ oshcc -W -Wall -o shmem_hello_world shmem_hello_world.c
$ oshrun -np 2 -x SHMEM_SYMMETRIC_HEAP_SIZE=$((32*1024*1024)) ./shmem_hello_world
Hello from PE 2 of 2
Hello from PE 1 of 2
shmem_hello_world: ../../../../../oshmem/mca/spml/yoda/spml_yoda.c:1113: mca_spml_yoda_get: Assertion `0 != r_mkey->len' failed.
[tudtug:25168] *** Process received signal ***
[tudtug:25168] Signal: Aborted (6)
[tudtug:25168] Signal code:  (-6)
shmem_hello_world: ../../../../../oshmem/mca/spml/yoda/spml_yoda.c:1113: mca_spml_yoda_get: Assertion `0 != r_mkey->len' failed.
[tudtug:25169] *** Process received signal ***
[tudtug:25169] Signal: Aborted (6)
[tudtug:25169] Signal code:  (-6)

Backtrace for PE 0 is:

#0  0x00007ffff749f0d5 in __GI_raise (sig=<optimized out>)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x00007ffff74a283b in __GI_abort () at abort.c:91
#2  0x00007ffff7497d9e in __assert_fail_base (fmt=<optimized out>,
    assertion=0x7fffd38c2658 "0 != r_mkey->len",
    file=0x7fffd38c21f8 "../../../../../oshmem/mca/spml/yoda/spml_yoda.c",
    line=<optimized out>, function=<optimized out>) at assert.c:94
#3  0x00007ffff7497e42 in __GI___assert_fail (
    assertion=0x7fffd38c2658 "0 != r_mkey->len",
    file=0x7fffd38c21f8 "../../../../../oshmem/mca/spml/yoda/spml_yoda.c",
    line=1113, function=0x7fffd38c26e0 "mca_spml_yoda_get") at assert.c:103
#4  0x00007fffd38c0d61 in mca_spml_yoda_get (src_addr=0xff004100, size=8,
    dst_addr=0x7fffffffc9f0, src=1)
    at ../../../../../oshmem/mca/spml/yoda/spml_yoda.c:1113
#5  0x00007fffd36b5f19 in _algorithm_f_recursive_doubling (group=0xc3fb90,
    target=0xff0020d8, source=0xff0000d0, nlong=4, pSync=0xff004100)
    at ../../../../../oshmem/mca/scoll/basic/scoll_basic_collect.c:472
#6  0x00007fffd36b4bfd in mca_scoll_basic_collect (group=0xc3fb90,
    target=0xff0020d8, source=0xff0000d0, nlong=4, pSync=0xff004100,
    nlong_type=true, alg=2)
    at ../../../../../oshmem/mca/scoll/basic/scoll_basic_collect.c:93
#7  0x00007ffff7a76b2c in _shmem_collect (target=0xff0020d8,
    source=0xff0000d0, nbytes=4, PE_start=0, logPE_stride=0, PE_size=2,
    pSync=0xff004100, array_type=true) at pshmem_collect.c:87
#8  0x00007ffff7a77236 in pshmem_fcollect32 (target=0xff0020d8,
    source=0xff0000d0, nelems=1, PE_start=0, logPE_stride=0, PE_size=2,
    pSync=0xff004100) at pshmem_collect.c:115
#9  0x00000000004009d8 in main ()

[bertwesarg]

Attachments are here:
https://gist.github.com/bertwesarg/0833dd47af181a5d304b

[bertwesarg]

Just done. I just needed to figure it out how it is done in github. If you need the config.log and ompi_info --all for a master build, please say so.

[jsquyres]

Thanks. Gist is the normal way.

[bertwesarg]

We are reaching 2 weeks of inactivity because of this regression! Any progress in the background?

Thanks.

[jsquyres]

@miked-mellanox @jladd-mlnx Have you guys seen this bug yet, perchance?

[mike-dubman]

yep, @igor-ivanov is looking into it and update on progress

[jsquyres]

Thanks.

[ompiteam-mtt]

I am able to reproduce issue on v1.8.4.
Investigation demonstrates that this issue came with following commit:

commit 3443c1d5e5c388623d9deac59fce3656d4e618d2
Author: Alex Mikheev <alexm@mellanox.com>
Date:   Sun Nov 16 17:20:20 2014 +0200

    OSHMEM: spml yoda: use flags to check if btl is RDMA capable

diff --git a/oshmem/mca/spml/yoda/spml_yoda.c b/oshmem/mca/spml/yoda/spml_yoda.c
index d37c5a6..ccf55b5 100644
--- a/oshmem/mca/spml/yoda/spml_yoda.c
+++ b/oshmem/mca/spml/yoda/spml_yoda.c
@@ -414,7 +414,7 @@ sshmem_mkey_t *mca_spml_yoda_register(void* addr,
         mkeys[i].spml_context = yoda_context;

         yoda_context->registration = NULL;
-        if (NULL != ybtl->btl->btl_prepare_src) {
+        if (ybtl->btl->btl_flags & MCA_BTL_FLAGS_RDMA) {

             /* initialize convertor for source descriptor*/
             opal_convertor_copy_and_prepare_for_recv(proc_self->super.proc_conv

I can not find initial explanation for this change. Probably I missed discussion.

This issue relates at least following configuration as spml=yoda btl=vader,self
It an be launched explicitly

oshrun -np 2 -x SHMEM_SYMMETRIC_HEAP_SIZE=$((32*1024*1024)) -mca spml yoda -mca pml ob1 -mca btl vader,self ./shmem_hello_world

Current configuration sets btl_vader_flags as 0x8 whereas MCA_BTL_FLAGS_RDMA macro is 0x6 as a result condition above is FALSE (it is not processed).

At the moment I can not understand if yoda logic should be changed to process this case or it can be solved by mca variables/configuration.

[jsquyres]

Who left that last comment? Please do not login and use the ompiteam-mtt account for doing actual Github work -- it is a utility account.

[igor-ivanov]

It was me. Sorry