Training details (MoGe and MoGe-2)

[karimknaebel]

Hi @EasternJournalist could you confirm or deny some of the hyperparameters that were used for training? This is what I gathered from various sources. In particular, I suspect that 1 million training steps for MoGe (v1) is not actually correct? What about the batch size of MoGe-2?

Thanks in advance!

	MoGe	MoGe-2
batch size	256 (paper)	128 (comment)
steps	~160k (comment)	120k (paper)
base LR	1e-4/1e-5 configs/train/v1.json	1e-4/1e-5 (paper)
LR schedule	"more conservative than 2"	half every 25k steps (paper)
GPUs	64 V100 #9	32 A100 (paper)
time	one week #9	5 days (paper)

[karimknaebel]

@EasternJournalist could you please quickly let me know the batch size for MoGe-2? I think I've figured out the rest. thanks 🙏🏽

[EasternJournalist]

Hi. The number of training steps for MoGe-1 is about 160K, and the batch size of MoGe-2 is 128. The LR schedule is the updated solution, which leads to faster convergence. When MoGe-1 was trained, we were actually using a more conservative LR decay strategy which requires more training steps.

[karimknaebel]

many thanks :) i've updated the table for reference

[karimknaebel]

what about low_resolution_training_steps=50_000 and lr_warmup=1000, are those correct?

[ZhouJankin]

Hi @EasternJournalist ,

I have a question regarding the batch size setting.
When you mention a batch size of 128, does this refer to the total effective batch size, i.e.

batch_size_total = batch_size_forward × gradient_accumulation_steps × accelerator.num_processes

or does it mean that batch_size_forward itself is set to 128?

I’m asking because in my experiments, using batch_size_forward = 4 already consumes around 50 GB of GPU memory, so I want to make sure I’m interpreting the batch size correctly.

Looking forward to your clarification.
Thanks in advance!