Why can model_pred simply subtract noise to fit targets=x0?

[zhangqizky]

https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_sdxl.py#L1079
here, when prediction_type=="sample", means UNet predict x0, and the targets also set to x0, but why the model_pred need to subtract noise? I don't much undstand here... Could anybody help to explain?

[sayakpaul]

I don't fully understand the question. How would you get the target sample otherwise?

[zhangqizky]

I'm not sure otherwise it should be. I just don't understand, what "model_pred = model_pred - noise " phsically mean? It should mean "predicted-x0", but the training target is "GT-x0", after training, the model ouput should be "predicted-x0", I don't understand why need to subtract noise. I think should be sth like this , not simply subtract? I didn't find any explain in ddpm-paper aboud "model_pred - noise", so I got confused....

[christopher-beckham]

@zhangqizky @sayakpaul I don't really understand it either. Based on these two lines:

• https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_sdxl.py#L1126
• https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_sdxl.py#L1064

we have model_pred = model_pred - epsilon, which I will just re-write to model_pred = unet(x_noised) - epsilon, where x_noised = add_noise(x, sigma, epsilon) (which would be e.g. x + sigma*epsilon for Euler schedulers) and epsilon is the original prescaled noise, i.e. epsilon ~ N(0,1). Since we're using a mean squared error loss, under a prediction type of "sample" it would obviously be:

loss = || model_pred - x ||^2
     = || (unet(x_noised) - epsilon) - x||^{2}

which means that unet(x_noised) = x + epsilon at optimality (to make the loss zero). This is an odd and I've never seen this formulation before. Either:

• (1) The unet should directly predict the original sample, i.e. unet(x_noised) = x, then the loss is ||unet(x_noised) - x||^2. This would truly correspond to the prediction type being "sample".
• (2) or you define the denoised sample in terms of unet being an epsilon predictor, i.e. x0_predicted = x_noised - sigma*unet(x_noised), and then do the minimise loss ||x0_predicted - x||^2. However this is sort of a moot point since it implies the prediction type should be "epsilon" and we're not doing that anyway.

It looks like most schedulers don't support prediction_type == 'sample', but for the ones that do it is assumed that (1) is the case, e.g. see here and here.

[zhangqizky]

您的邮件已收到，我会尽快回复，谢谢。张琦

[sasasqt]

any updates on this? @bghira

[zhangqizky]

您的邮件已收到，我会尽快回复，谢谢。张琦