BTL/SM component frag header confusion

[TroyMitchell911]

When initializing the free_list, the payload_buffer_size is:
mca_btl_sm_component.max_inline_send + sizeof(mca_btl_sm_frag_t)

In the init function, base.super.ptr(item->ptr) is assigned as payload_ptr:

// in opal_free_list.c
/* allocate the rest from the mpool (or use memalign/malloc) */
payload_ptr = (unsigned char *) flist->fl_mpool->mpool_alloc(flist->fl_mpool,
                                                             buffer_size,
                                                             align, 0);
item->ptr = payload_ptr;

So the memory layout looks like this:
+-----------------------------+
| struct btl_sm_frag | <- base.super.ptr
| ... |
+-----------------------------+
| max_inline_send_size bytes |
| (payload buffer) |
| ... |
+-----------------------------+
In the frag_constructor function:

frag->hdr = (mca_btl_sm_hdr_t *) frag->base.super.ptr;

So hdr directly points to base.super.ptr(this memory size is sizeof(struct mca_btl_sm_frag _t)?
Then why, during init, is it not + sizeof(struct mca_btl_sm_hdr_t)?

If I'm wrong, please let me know.

Best regards,
Troy

[TroyMitchell911]

If this is indeed the issue, I'd be happy to submit a PR.

[bosilca]

Nice catch. mca_btl_sm_frag_t is the local struct keeping track of the fragment, while mca_btl_sm_hdr_t is on the wire header exchanged between peers (header that holds a pointer back to the originating fragment). So you are right, the size of the freelist item should technically be mca_btl_sm_component.max_inline_send + sizeof(mca_btl_sm_hdr_t). We're wasting 200 bytes for every fragment (sizeof(mca_btl_sm_frag_t) - sizeof(mca_btl_sm_hdr_t).

[TroyMitchell911]

Nice catch. mca_btl_sm_frag_t is the local struct keeping track of the fragment, while mca_btl_sm_hdr_t is on the wire header exchanged between peers (header that holds a pointer back to the originating fragment). So you are right, the size of the freelist item should technically be mca_btl_sm_component.max_inline_send + sizeof(mca_btl_sm_hdr_t). We're wasting 200 bytes for every fragment (sizeof(mca_btl_sm_frag_t) - sizeof(mca_btl_sm_hdr_t).

Thanks for your reply, I will submit a PR to fix this issue. :)
Have a nice day, bosilca.