Question About Training with Variable Length Videos

[xiao10ma]

Thank you for the excellent work on this model. I have a question regarding the training process for handling videos of varying lengths. Could you please clarify the distribution of video durations within your training dataset?

1. Since the model need to generate different length of videos, I'm interested in the proportion of videos with different frame counts.
2. Additionally, was a uniform learning rate applied across all training samples, or was it adjusted based on the video length?
3. Lastly, could you share the learning rate and the training steps?

Thanks in advance!

[xiao10ma]

By the way, I have one more question. I noticed the OmniWorld dataset was released recently. Did you use it to fine-tune a base model like CogVideo, and did you experiment with any other models as well?

[HaoyiZhu]

Thanks for your interest. During training, we just uniformly select the number of frames among [17, 25, 33, 41]. The learning rate is not related to the number of frames; we just keep a cosine learning scheduler across training steps. The max learning rate is 2e-5, and the min learning rate is 2e-7. The total number of training steps of all stages is about 20k.

[HaoyiZhu]

We have trained a second version of Aether on the OmniWorld dataset. We are currently conducting evaluations and plan to release the results within 1 or 2 months. Stay tuned!

[xiao10ma]

Thank you so much!