Synchronize lora's merge, unmerge, etc. modifications to lora's tp_layer.

[zhangsheng377]

There was a previous commit that moved the merge and other functions in LoraLayer to Linear, etc., but the LoraParallelLinear in tp_layer was missed.

[zhangsheng377]

@BenjaminBossan

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[BenjaminBossan]

Thanks for this update. Could you please run make style?

[zhangsheng377]

Thanks for this update. Could you please run make style?

Ok. Sorry, I forgot it.

[BenjaminBossan]

The PR is failing on Python 3.8 because of type annotation syntax. Adding from __future__ import annotations should fix that. I assume that you tried out that the changes you made to the megatron layer work?

[BenjaminBossan]

This should not be necessary, right?

[zhangsheng377]

The PR is failing on Python 3.8 because of type annotation syntax. Adding from __future__ import annotations should fix that. I assume that you tried out that the changes you made to the megatron layer work?

The merge func is that I copyed from Linear.merge

But I forgot to add the from __future__ import annotations

[BenjaminBossan]

I don't really have experience with megatron, so I'm not sure if these methods would just work when copied 1:1. Just to be sure, did you test with your setup that nothing breaks with these changes? If yes, I think we can merge and in the future try to be more careful to keep tp_layer.py in sync.