Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding input pertubation #3293

Closed
isamu-isozaki opened this issue Apr 30, 2023 · 6 comments
Closed

Adding input pertubation #3293

isamu-isozaki opened this issue Apr 30, 2023 · 6 comments
Labels
stale Issues that haven't received updates

Comments

@isamu-isozaki
Copy link
Contributor

isamu-isozaki commented Apr 30, 2023

Input perturbation is a very simple change proposed here where we add noise during training to make the trainer robust to the error accumulation during inference. This got state-of-the-art on the celeb dataset as can be seen here

I made a pr here

@xiaohu2015
Copy link
Contributor

@isamu-isozaki Do this trick related to noise offset?

@isamu-isozaki
Copy link
Contributor Author

isamu-isozaki commented May 17, 2023

@xiaohu2015 Good question! And very similar implementation but the idea/what they do is very different.

Noise offset adds another normal distribution to reduce the low-frequency noise to make stable diffusion able to generate images that it originally can't. Implementation-wise this is

noise += args.noise_offset * torch.randn(
    (latents.shape[0], latents.shape[1], 1, 1), device=latents.device
)
....
noisy_latents = noise_scheduler.add_noise(latents, noise, timesteps)
....
loss = F.mse_loss(model_pred.float(), noise.float(), reduction="mean")

For input perturbation, the idea is to make the model's training robust to an error during inference(some random noise at x_t). To do this we do

new_noise = noise + args.input_pretubation*torch.randn_like(noise)
...
noisy_latents = noise_scheduler.add_noise(latents, new_noise, timesteps)
...
loss = F.mse_loss(model_pred.float(), noise.float(), reduction="mean")

so, in input perturbation, the objective is to ignore the new noise added while for offset noise, it's included for more info.

Let me know if that makes sense!

@xiaohu2015
Copy link
Contributor

@isamu-isozaki another work is https://arxiv.org/abs/2305.08891

@isamu-isozaki
Copy link
Contributor Author

@xiaohu2015 ohh I was planning to read that paper. Thanks for the send!

@github-actions
Copy link

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

@github-actions github-actions bot added the stale Issues that haven't received updates label Jun 12, 2023
@isamu-isozaki
Copy link
Contributor Author

Closing since merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
stale Issues that haven't received updates
Projects
None yet
Development

No branches or pull requests

2 participants