[Draft] DSL design for next version #519
Conversation
| channel = Channel(dst_rank, src_rank, channel_type) | ||
| sem.acquire(tb=1) | ||
| channel.put(other_peer_chunk, dst_chunk, tb=1) | ||
| ``` |
There was a problem hiding this comment.
is this code pipelining "copy data from input buffer to scratch buffer" with "transfer data from scratch buffer to other peers". if so, which variable is the scratch buffer?
sem.release(tb=0) and sem.acquire(tb=1) ensures that copy to dst_chunk occures before put operation, right?
please provide more detailed explanation about the code
| ```python | ||
| sem = Rank.Semaphore(rank=rank, size=1) | ||
| rank = Rank(src_rank) | ||
| with Loop.iteration(unit=2**20, num_chunks=1) as iter: |
There was a problem hiding this comment.
please add more explanation about the code. e.g., what is unit?
| # The dst_chunk and src_chunk sizes should match the num_chunks parameter in the loop context. | ||
| rank.copy(dst_chunk, src_chunk, tb=0, iter_context=iter) | ||
| sem.release(tb=0) | ||
| channel = Channel(dst_rank, src_rank, channel_type) |
There was a problem hiding this comment.
can we define channel before loop?
|
|
||
|
|
||
| Here is the example for two ranks allreduce. Which achieve non-zero copy and use nvls. We use 3 thread-blocks to do the allreduce. | ||
| The first thread-block is used to copy data from input buffer to scratch buffer, the second thread-block is used to do allreduce in scratch buffer, and the third thread-block is used to copy data from scratch buffer to output buffer. The thread-blocks are synchronized by semaphores. |
There was a problem hiding this comment.
please add a line by line explanation of the code
| ``` | ||
|
|
||
| ## All2All support | ||
| For now, DSL only support static all2all algorithm. For all2allv support, we need to get the send/recv size at the runtime. It may require some placeholder at the Json execution plan and relace to the real size at the runtime. If we could make chunk size be variable, we could use the same way to support all2allv. |
There was a problem hiding this comment.
--> "Currently, the DSL only supports the static all2all algorithm. To support all2allv, we need to obtain the send/receive sizes at runtime. This may require using placeholders in the JSON execution plan, which would be replaced with the actual sizes during execution. If we can make the chunk size variable, the same approach could be used to support all2allv."
No description provided.