Where in the code is SDEdit?

[jmugan]

Hi, love the project. I'm trying to run this on a custom dataset, and I can't see where the image is inserted into the diffusion process as described in equation 6 in the paper. I see the textual inversion but not that. Can someone point me to where that is? Thanks!

[brandontrabucco]

Hello Jonathan,

Thanks for your interest in our code! SDEdit is implemented inside the diffusers StableDiffusionImg2ImgPipeline class:
https://huggingface.co/docs/diffusers/v0.26.3/en/api/pipelines/stable_diffusion/img2img#diffusers.StableDiffusionImg2ImgPipeline. The key arguments to this pipeline's call method for generation are image and strength, the latter of which controls the extent the images are modified.

This pipeline is used in the code for augmentation here:

da-fusion/semantic_aug/augmentations/textual_inversion_upstream.py

Line 62 in dbcdc9c

StableDiffusionImg2ImgPipeline)

-Brandon

[brandontrabucco]

Also, when loading textual inversion embeddings, the diffusers Stable Diffusion pipelines will fail silently if a Textual Inversion embedding doesn't exist in the folder the code points to:

da-fusion/semantic_aug/augmentations/textual_inversion_upstream.py

Line 76 in dbcdc9c

embeds_list = glob(embed_path + '/**/learned_embeds.bin')

Make sure to use the fine_tune_upstream.py script to train these embeddings first.

Let me know how else I can help!

-Brandon

[jmugan]

Thanks! Still working through it.