crash when sending very large messages

[mattijsjanssens]

We're occasionally seeing assert message of the form

ips_proto.c:1646: (scb->payload_size & 0x3) == 0

which seem to originate from somewhere in the network stack (e.g. https://github.com/intel/opa-psm2/blob/master/ptl_ips/ips_proto.c#L1957) when the size is not a multiple of 4.

• this only happens occasionally
• and only for extremely large messages (not sure but could be 100Mb or even Gb)
• and only on omnipath, either with IntelMPI+ofi or ompi+psm2

Is this a known problem? We don't pad our mpi messages to be multiple of 4 bytes. Should we? If so why does it not show up on ordinary usage (i.e. smaller messages).

[aravindksg]

We have seen this problem appear with message sizes that are not DW multiple before, but the issue was fixed. (as of PSM2 version :PSM2_10.2-235)
Also- the line numbers you posted above do not match:
ips_proto.c:1646 where your execution is failing and current location of assert (in latest PSM2 master) is ips_proto.c: 1957. Could you clarify if you are actually using the latest PSM2 version from GitHub or a different PSM2 version (either from distro or from IFS)? If it is indeed an older version, could you please update to latest GitHub master and retry?

[aravindksg]

We have seen this problem appear with message sizes that are not DW multiple before, but the issue was fixed. (as of PSM2 version :PSM2_10.2-235)
Also- the line numbers you posted above do not match:
ips_proto.c:1646 where your execution is failing and current location of assert (in latest PSM2 master) is ips_proto.c: 1957. Could you clarify if you are actually using the latest PSM2 version from GitHub or a different PSM2 version (either from distro or from IFS)? If it is indeed an older version, could you please update to latest GitHub master and retry?

[mattijsjanssens]

Thanks for the answer. I will check.

[mattijsjanssens]

Thanks for the answer. I will check.

[rwmcguir]

Can this issue be closed or is this still a problem?

[rwmcguir]

Can this issue be closed or is this still a problem?

[mattijsjanssens]

From what I'm told it can probably be closed. Many thanks for the feedback. Mattijs

…

On 26 April 2018 at 17:25, Russell McGuire ***@***.***> wrote: Can this issue be closed or is this still a problem? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#23 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AL-eP6QymF_FyJLLpO6gOuuHgwRZwwWAks5tsfT2gaJpZM4RshQt> .

[mattijsjanssens]

From what I'm told it can probably be closed. Many thanks for the feedback. Mattijs

…

On 26 April 2018 at 17:25, Russell McGuire ***@***.***> wrote: Can this issue be closed or is this still a problem? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#23 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AL-eP6QymF_FyJLLpO6gOuuHgwRZwwWAks5tsfT2gaJpZM4RshQt> .

[rwmcguir]

Thank you for confirming.

[rwmcguir]

Thank you for confirming.