Add Documentation for LoraConfig Parameters

[sebdg]

Hi team,

I've added detailed documentation for the LoraConfig parameters in your project. This includes explanations for the key parameters and their impacts on the LoRA training process, making it easier for everyone to understand and use.

Here's a summary of what has been documented:

• r: Rank of low-rank decomposition, balancing training efficiency and performance.
• lora_alpha: Scaling factor for low-rank matrices, affecting convergence speed and stability.
• lora_dropout: Dropout probability for regularization, preventing overfitting.
• weight_decay: Regularization technique to penalize large weights, promoting generalization.
• learning_rate: Controls the step size for parameter updates, balancing training speed and stability.
• use_rslora: Enables Rank-Stabilized LoRA for better performance.
• loftq_config: Configuration for LoftQ quantization, optimizing model efficiency and performance.
• gradient_accumulation_steps: The number of steps to accumulate gradients before performing a backpropagation update.
• target_modules: Specifies which model modules LoRA is applied to, providing flexibility for different architectures.

I believe these additions will be beneficial for the project and general users.
Feel free to take parts of this to the wiki or docs, and if there is anything else you want to be documented let me know.
Looking forward to your feedback!

Best regards,
Sebastien

[danielhanchen]

Great work thanks!