[Trainer] LoRA feature enhancement

[lynnliu030]

SkyRL currently supports LoRA training when using FSDP+vLLM. However, there still exist several features, integrations, and performance improvements that we will aggressively iterate on. The goal is to extend LoRA support across the backends SkyRL supports and ensure LoRA training has much greater throughput and memory efficiency relative to full parameter fine tuning.

If you have any other desired LoRA features, please feel free to comment on this issue!

TODOs:

• Support LoRA for the Megatron training backend and support MoE models
• Support fast in-memory sync of LoRA weights to vLLM.
• Explore similar quantized LoRA variants (e.g. QLoRA?)

Later

• Support LoRA for SGLang

[benji-cannot-code]

Would love to help with this if needed!