where the code related to communication and calculation of overlap of one algorithm？

[chst1994]

I didn't find code related to communication and calculation of overlap，can you show me an example? Thanks！

[wangraying]

The overlapping mechanism is implemented in backward hook. During the backward process, when tensors in a bucket are ready, a background thread (implemented in rust) will start communication (performing comm op on the bucket), while the main thread can continue the backward process. In this way, successive buckets' communication and computation can overlap together.