The time step t is not used?

[PanXiebit]

unleashing-transformers/models/transformer.py

Line 130 in 3e5f564

def forward(self, idx, t=None):

The denoise_fn is Transformer, but in the forward of this model, the time step t is not used?

[samb-t]

Hi, yes that's correct. In our early experiments we tested the impact of conditioning the Transformer on time but didn't find it to have much of an effect.

My guess as to why this doesn't matter so much in the discrete absorbing diffusion case is that it's relatively simple for the model to figure out what the time step is based on the number of input tokens that are masks. Whereas in the continuous Gaussian case it's much less obvious how noisy the inputs are.

Hope this helps!

[PanXiebit]

@samb-t Thanks a lot! This really helps.