MPI_Sendrecv_replace needlessly limits message byte-size

[albandil]

(This problem is version-agnostic, present e.g. in 4.1.0 but also in the current Git.)

The function MPI_Sendrecv_replace first creates a temporary contiguous buffer into which it moves the local data for sending and in this way decouples the send-receive operation by sending from that buffer and receiving into the original one.

This requires that the data to be sent are first converted to the MPI_PACKED (1-byte) datatype and that the associated count (or packed_size to be more specific) parameter is adequately increased. To perform the send-receive, MPI_Sendrecv_replace then simply calls MPI_Sendrecv operating on the two buffers.

However, to call MPI_Sendrecv the 32-bit integer API is used, which puts a needless internal limit on the actual byte-size (packed_size) of the message.

Given how simple the code of MPI_Sendrecv is---particularly when the parameter checking is omitted given that this would be an internal call---couldn't the code be duplicated in MPI_Sendrecv_replace, too, to avoid this limit?

Such a limit is not present in MPICH, which apparently uses internal 64-bit integer API in such places.

[ggouaillardet]

Thanks for the report!

I am not sure I fully understand the solution you suggest.

A potentially simpler option would be (only when needed?) to create an intermediate datatype of contiguous MPI_PACKED with the same size as datatype, and then PMPI_Sendrecv(..., count, ..., count)
An other option on the long run is to wait for the upcoming MPI embiggenment to be implemented, and use these primitives (no ETA yet though)

@bosilca any thoughts/advices ?

[albandil]

What I meant is that MPI_Sendrecv_replace currently calls MPI_Sendrecv (using 32-bit integer API), which calls mca_pml_whatever_irecv and mca_pml_whatever_send (using 64-bit integer API), so avoiding the constraining intermediate call to MPI_Sendrecv and directly calling ..._irecv and ..._send from MPI_Sendrecv_replace would be one way.

But if MPI_Type_contiguous is applicable here (to keep the count unchanged), it sounds indeed as a much cleaner solution.

[bwbarrett]

I'm not sure I agree with the proposed solution, either. Yes, that would work around the current limitation. But it would also result in creating a temporary buffer of at least 4 GiB. That seems like trading one unexpected behavior for another. A more advanced segmented copy protocol is almost certainly required here if we want to avoid unexpected behavior.

[albandil]

The huge allocation is already there, if I understand the code correctly:

ompi/ompi/mpi/c/sendrecv_replace.c

Line 122 in eba1796

rc = PMPI_Alloc_mem(packed_size, MPI_INFO_NULL, &iov.iov_base);

[bosilca]

With the current PML API we can't do anything better than wasting memory and going through the intermediate packed buffer. We could use the partitioned communication API to handle this, it could be interesting.

[ggouaillardet]

Thanks @bosilca and @bwbarrett for your feedback.

Given the facts that we already make the huge memory allocation, data can be silently truncated, and I do not see a clear fix happening in a near future (at least for the release branches), I fixed that issue as initially suggested by @albandil

A simple improvement would be not to allocate and sendrecv all count at once, but allocate a "reasonable" number of datatypes, and send/recv in a loop. That would not handle the case when there is only a few number of a huge datatype though.

[bosilca]

There will be issues with the matching tags and the ordering of communications. You will need to use internal tags to avoid polluting the user tag-space, and then you will need to properly separate them in a non-overlapping ranges (to avoid issues when multiple sendrecv_replace are running in the same time).

[ggouaillardet]

oh, you mean if one task calls MPI_Sendrecv_replace() and the other one simply call something else such as MPI_Send() ?

[bosilca]

That's one problematic scenario. Also threaded applications, or the non-blocking version of the call (which I heard is scheduled for MPI 4.0) will have similar issues.