Analyze overlapped P2P memory transfer and computing

[andreaskoepf]

Create an ipynb to analyze in PyTorch the peer-to-peer (between two GPUs) memory transfer and computing in parallel. Dummy computation could for example be some larger matmuls in a loop. Create notebooks folder and place the file there.

Goal should be to demonstrate that memory transfer and computation can run (to some degree) overlapped.

• Old nvidia blog post: How to Overlap Data Transfers in CUDA C/C++
• cuda steams in pytorch, see CUDA streams
• Slides: Data Transfer and CUDA Streams
• Slides: CUDA STREAMS BEST PRACTICES AND COMMON PITFALLS

Quote from the ring-attention paper:
"If the computation time exceeds the time required for transferring key-value blocks, this results in no additional communication cost. This overlapping mechanism applies to both forward and backward passes of our approach since the same operations and techniques can be used"

[eternalops]

first one's link: https://developer.nvidia.com/blog/how-overlap-data-transfers-cuda-cc/

[andreaskoepf]

zhuzilin/ring-flash-attention#9