-
Notifications
You must be signed in to change notification settings - Fork 367
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
R2 setup in the paper #96
Comments
R1 uses the original DiT model with the original trainer, as in the official implementation, with just the VAE removed and normalization adjusted. This setup has a bunch of important differences to the trainer setup used for HDiT (discrete vs. continuous time, sigma distribution, ...). To enable the most fair comparison possible, these should ideally be matched, so we implemented DiT in the same trainer as HDiT, simply putting one of Kat's discrete-time DDPM wrappers around the original implementation and training with the same settings as HDiT, yielding R2. That, however, resulted in a substantially worse model (probably due to multiple reasons compounding - the adapting to continuous time while still keeping the conditioning network that expects discrete time, mismatched hyperparameters, ...). So we also added a version that uses all the same transformer blocks and conditioning network ("mapping network") as HDiT but matches the structure (blocks, width, depth, ...) of DiT exactly, yielding R3. As this performs basically equal to the original DiT, it's reasonable to assume that R2 performed badly due to a mismatch in network vs. training setup, with R3 enabling comparisons w.r.t. model structure. Finally, having R3 enables training DiT for a fair comparison with Soft-Min-SNR (as there's complex interactions between it and the EDM preconditioner, sigma sampling schedules, ..., which having a base setup that's matched regarding them with R3 vs hour HDiT ablations makes them substantially easier to compare), yielding R4. |
Ok so if i get it right the trainer used in HDiT uses EDM instead of DDPM parametrization? Is there other changes? Thanks! |
Thanks for the clarifications! |
Hi,
I'm not sure i get from the hourglass paper what does the R2 trainer setup corresponds to. What is changed versus R1?
And since it underperform, why is it kept for R3 and R4? Wouldn't R1 work better?
Thanks!
The text was updated successfully, but these errors were encountered: