Improve C++ API for data transfers to and from streams

[Mellich]

Currently, all ACCL calls like send/recv require a reference to a BaseBuffer object as mandatory argument. When streaming is used as input or output, this requires to create and pass a BaseBuffer which is not used at all.

CCLO *ACCL::send(BaseBuffer &srcbuf, unsigned int count, unsigned int dst, unsigned int tag, communicatorId comm_id, bool from_fpga, streamFlags stream_flags, dataType compress_dtype, bool run_async, std::vector<CCLO *> waitfor)

There are different options to change/extend the API to overcome this issue:

• Pass the buffers as pointer. This would allow to use a nullptr for the argument. This would already be handled by ACCL::prepare_call but it would change the signatures for all calls.
• Create specific overloads of the calls for streaming that do not use the input buffer and implicitly set the input streaming flag
• introduce additional calls ACCL::stream_send, ACCL::stream_recv ... that explicitly support streaming and only take the required arguments as input.

I personally would favor the last option because it is most explicit. However, I am not sure if it is flexible enough to support further enhancements of ACCL in the future since we would basically remove control over the streaming flags by the user.

I would also argue for a stream_recv call because the current way of handling communication via streams as one-sided communication is ambiguous. What happens in cases, where two ranks send data to the same destination rank? The order the messages will arrive is not defined and there is no way to find out the current sender on the receiving side. With that, we need to use barriers to enforce the correct order. With the stream_recv calls we would be able to define the order how the messages are received and handled by the user kernel.

I think all this needs further discussion to find the best solution. Maybe there are also technical restrictions i am not aware of.

[quetric]

All valid points to consider, but I'll touch this first:

I would also argue for a stream_recv call because the current way of handling communication via streams as one-sided communication is ambiguous. What happens in cases, where two ranks send data to the same destination rank? The order the messages will arrive is not defined and there is no way to find out the current sender on the receiving side. With that, we need to use barriers to enforce the correct order. With the stream_recv calls we would be able to define the order how the messages are received and handled by the user kernel.

I think all this needs further discussion to find the best solution. Maybe there are also technical restrictions i am not aware of.

The way stream_put works is that the received data bypasses all the RX logic and goes straight to the kernel stream, in an attempt to provide the lowest-latency delivery possible. A side-effect of this is that the metadata of the received message is not recorded anywhere, therefore it's not possible to sync on it. Think of it as the equivalent of a RDMA write - the writer can't know the write has finished.

To synchronize, you could follow up a stream_put with a send on the sender, and have the receiver wait on a matching recv. But from a performance point of view, you might as well do a (send) + (recv with RES_STREAM), which gives you both synchronization and certainty about the origin of the data in your stream.

Alternatively, senders could agree to target a distinct stream ID on their respective receivers in the stream_put operation, which means the sender can be identified by inspecting the TDEST field of the stream.