Add support for sequence parallelism

[Quentin-Anthony]

Add sequence parallelism from upstream megatron

• https://github.com/search?q=repo%3ANVIDIA%2FMegatron-LM+sequence_parallel&type=code
• https://arxiv.org/pdf/2205.05198.pdf

[guozhiyao]

Hi, do you have any progress on sequence parallelism ?

[Quentin-Anthony]

Not yet. We haven't been able to find someone with the time to implement this yet.

[NeverNude]

I'd be interested in taking this on, if help is still needed here.

[StellaAthena]

I'd be interested in taking this on, if help is still needed here.

That would be awesome! Someone recently dropped by our discord to volunteer to tackle this, was that you? If not, you should drop by our discord server to coordinate.

[NeverNude]

Yes that was me! NerevarTheCaptain on discord.

[Quentin-Anthony]

@NeverNude How's this going?

[NeverNude]

Apologies, I haven't been able to get to this yet, should have an update within the next 7-10 days

…

On Mon, May 1, 2023 at 9:47 PM Quentin Anthony ***@***.***> wrote: @NeverNude <https://github.com/NeverNude> How's this going? — Reply to this email directly, view it on GitHub <#812 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMTVBSRQ4P2WVJSBNGHJKDXECGUVANCNFSM6AAAAAAVSZQMOA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Quentin-Anthony]

Apologies, I haven't been able to get to this yet, should have an update within the next 7-10 days
…
On Mon, May 1, 2023 at 9:47 PM Quentin Anthony @.> wrote: @NeverNude https://github.com/NeverNude How's this going? — Reply to this email directly, view it on GitHub <#812 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMTVBSRQ4P2WVJSBNGHJKDXECGUVANCNFSM6AAAAAAVSZQMOA . You are receiving this because you were mentioned.Message ID: @.>

Any update? Anything we can help with?

[NeverNude]

Apologies, life got way busier than I expected. I am hoping to get to this in the next couple of weeks, but feel free to assign it to somebody else, I don't want to block progress unnecessarily here.

[Quentin-Anthony]

@NeverNude -- Are you still interested in heading this up? Do you have an early version that you can put in a draft PR so that we can add other contributors?

[NeverNude]

@Quentin-Anthony you should assign it to somebody else at this point. I am unlikely to find the time soon to be able to tackle this, apologies.

[Quentin-Anthony]

@NeverNude -- No worries!

All - we are re-opening this issue to contributors. Reach out if you're interested in tackling it!