Multiple GPU Lora training is not working.

[leonary]

I have successfully configured two A40 graphics cards to perform Lora training. During training, both cards are observed to be utilized, but the training speed does not improve significantly. The time required for training is almost the same as using a single card, and the number of epochs increases from 1 to 2. Furthermore, the training results achieved with two cards (the capability displayed by Lora) are even worse than those obtained with a single card.
I would like to know if it is possible to accelerate Lora training using multiple cards. If so, what should I do? Apart from setting the Accelerate config, are there any additional steps required?

[kohya-ss]

In multiple GPU training, the number of the images multiplied by GPU count is trained with single step. So it is recommended to use --max_train_epochs for training same amount as the single GPU training.

For the result of LoRA, I think it may be overfitted by multiple GPU training.