Training time of Unpaired Day2night

[wenyi-li]

Thanks for your impressive work! I'm curious about the training time required for the Unpaired Day2Night model on the BDD100K.

It took me around 1 day on a single A100 GPU for only 1k steps. This training time seems pretty long, and I have a feeling there might be some problems causing it to take so much time. Could you help me figure out what's going on?

Here are some of the training parameters:
--learning_rate="1e-5" --max_train_steps=25000
--train_batch_size=1 --gradient_accumulation_steps=1
--enable_xformers_memory_efficient_attention --validation_steps 250
--lambda_gan 0.5 --lambda_idt 1 --lambda_cycle 1

[ShenZheng2000]

I'm also training BDD100K Day2Night. It takes about 16 hours on four 4090 GPUs to complete 25,000 training steps.