Example for overlap communication with sequence parallel

[vivekgoe]

I am looking to optimize performance for sequence parallel implementation using DistributedAttention by overlapping communication and compute. I see that there is some code for this available under "sp_overlap_comm" flag in

DeepSpeed/deepspeed/sequence/layer.py

Line 311 in 02bbf50

class DistributedAttention(torch.nn.Module):

Is this feature (sp_overlap_com) functional? If yes, is there an example or blog about this which I can use to understand the feature. Will really appreciate it if someone can help answering this. Thanks.

[inkcherry]

hi @vivekgoe , plz see deepspeedai/Megatron-DeepSpeed#415

[vivekgoe]

@inkcherry Thanks for sharing PR, will check this.