add PAG support for SD architecture #8725

shauray8 · 2024-06-28T00:03:35Z

What does this PR do?

Adds PAG (Perturbed-Attention Guidance) support for SD models (StableDiffusionPAGPipeline). Continuation of #7944

Fixes #8710 (partially)

Before submitting

Did you read the contributor guideline?
Did you read our philosophy doc (important for complex PRs)?
Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

@yiyixuxu
Anyone in the community is free to review the PR once the tests have passed.

for the wonderers, here are some of my results I found during testing

Comparison between activation layers	Comparison between PAG and no-PAG

I thought attention applied on the latter layers should give out much better quality and applying on middle layers would be much faster.

Usage [SD+PAG]

from diffusers import AutoPipelineForText2Image
from diffusers.utils import load_image
import torch

pipeline = AutoPipelineForText2Image.from_pretrained(
    "Lykon/DreamShaper",
    enable_pag=True,
    pag_applied_layers = ["mid", "up.block_1.attentions_0"],
    torch_dtype=torch.float16
)
pipeline.enable_model_cpu_offload()


pag_scales =  [0.0, 3.0]
guidance_scales = [0.0, 2.0]

grid = []
for pag_scale in pag_scales:
    for guidance_scale in guidance_scales:
        generator = torch.Generator(device="cpu").manual_seed(0)
        images = pipeline(
            prompt="a polar bear sitting in a chair drinking a milkshake",
            negative_prompt="deformed, ugly, wrong proportion, low res, bad anatomy, worst quality, low quality",
            num_inference_steps=30,
            guidance_scale=guidance_scale,
            generator=generator,
            pag_scale=pag_scale,
            height=512,
            width=512,
        ).images
        images[0]

        grid.append(images[0])

# save the grid
from diffusers.utils import make_image_grid
make_image_grid(grid, rows=len(pag_scales), cols=len(guidance_scales)).save("sample.png")

shauray8 · 2024-06-28T00:05:38Z

src/diffusers/pipelines/pag/pipeline_pag_sd.py

+        **kwargs,
+    ):
+        deprecation_message = "`_encode_prompt()` is deprecated and it will be removed in a future version. Use `encode_prompt()` instead. Also, be aware that the output format changed from a concatenated tensor to a tuple."
+        deprecate("_encode_prompt()", "1.0.0", deprecation_message, standard_warn=False)


@yiyixuxu should I remove all the deprecate messages, I think this is long deprecated

I think we can remove this method here since it is not used in the pipeline

a-r-r-o-w · 2024-06-28T09:26:46Z

src/diffusers/pipelines/pag/pipeline_pag_sd.py

+
+        return prompt_embeds, negative_prompt_embeds
+
+    def encode_image(self, image, device, num_images_per_prompt, output_hidden_states=None):


Could you add # Copied from comments at every method that requires it in similar fashion to how it's done in other pipelines?

Ahh, forgot to do that, done!

Bhavay-2001 · 2024-06-28T16:27:18Z

Hi @shauray8, I just have one query. How did you compare the StableDiffusionPipeline and added the support for PAG. How did you figure out where to add the lines for PAG support. I am facing a bit difficulty in that.
Thanks

yiyixuxu

thanks! very nice work!
I think there are some deprecated method from SD1.5 that we do not need to add in PAG, other than that, it is perfect!

yiyixuxu · 2024-06-28T17:05:00Z

src/diffusers/pipelines/pag/pipeline_pag_sd.py

+        **kwargs,
+    ):
+        deprecation_message = "`_encode_prompt()` is deprecated and it will be removed in a future version. Use `encode_prompt()` instead. Also, be aware that the output format changed from a concatenated tensor to a tuple."
+        deprecate("_encode_prompt()", "1.0.0", deprecation_message, standard_warn=False)


I think we can remove this method here since it is not used in the pipeline

yiyixuxu · 2024-06-28T17:08:45Z

src/diffusers/pipelines/pag/pipeline_pag_sd.py

+        return image, has_nsfw_concept
+
+    # Copied from diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline.decode_latents
+    def decode_latents(self, latents):


we can remove this method here if it is not used by this pipeline :)

yiyixuxu · 2024-06-28T17:10:11Z

src/diffusers/pipelines/pag/pipeline_pag_sd.py

+        callback_on_step_end_tensor_inputs: List[str] = ["latents"],
+        pag_scale: float = 3.0,
+        pag_adaptive_scale: float = 0.0,
+        **kwargs,


Suggested change

**kwargs,

can remove this if not used

yiyixuxu · 2024-06-28T17:10:50Z

src/diffusers/pipelines/pag/pipeline_pag_sd_xl.py

@@ -76,7 +76,7 @@
        >>> pipe = AutoPipelineForText2Image.from_pretrained(
        ...     "stabilityai/stable-diffusion-xl-base-1.0",
        ...     torch_dtype=torch.float16,
-        ...     enabe_pag=True,
+        ...     enable_pag=True,


thank you!!

yiyixuxu · 2024-06-28T17:16:51Z

hi @Bhavay-2001
if you are working on StableDiffusionControlNetPAGImg2ImgPipeline, no? so I think you can:

copy over the code from StableDiffusionControlNetImg2ImgPipeline as a starting point
compare the code between StableDiffusionXLControlNetPAGImg2ImgPipelineand StableDiffusionXLControlNetImg2ImgPipeline, understand the change we introduced for PAG and apply same logic to StableDiffusionControlNetPAGImg2ImgPipeline

HuggingFaceDocBuilderDev · 2024-06-28T17:20:03Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

yiyixuxu · 2024-06-28T17:24:33Z

can get the CI to pass too: run make style and make fix-copies

shauray8 · 2024-06-29T12:18:46Z

@yiyixuxu removed the methods mentioned above, with all the changes necessary for the CI to go green.

yiyixuxu · 2024-06-29T19:26:27Z

@shauray8 thanks for your contribution!

shauray8 added 7 commits June 27, 2024 15:37

add pag to sd pipelines

7dbc3de

auto pipeline ad

b22c268

tests and fixes

ad0f46d

docs update

61b2039

format

6789156

fix

25be53a

docs

5a93620

shauray8 commented Jun 28, 2024

View reviewed changes

sayakpaul requested review from yiyixuxu and asomoza June 28, 2024 02:42

a-r-r-o-w reviewed Jun 28, 2024

View reviewed changes

shauray8 added 2 commits June 28, 2024 20:12

add copied from

2e02f9b

reformated

ba53d07

yiyixuxu mentioned this pull request Jun 28, 2024

Add PAG support to SD1.5 #8710

Open

5 tasks

yiyixuxu approved these changes Jun 28, 2024

View reviewed changes

shauray8 added 2 commits June 29, 2024 17:45

remove deprecated methods

1e5fbbc

CI fixes

35a8c93

yiyixuxu merged commit 8690e8b into huggingface:main Jun 29, 2024
14 of 15 checks passed

shauray8 deleted the pag_sd15 branch July 4, 2024 10:22

a-r-r-o-w mentioned this pull request Jul 8, 2024

add PAG support for SD Controlnet Img2Img #8810

Open

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add PAG support for SD architecture #8725

add PAG support for SD architecture #8725

shauray8 commented Jun 28, 2024 •

edited

Loading

shauray8 Jun 28, 2024

yiyixuxu Jun 28, 2024

a-r-r-o-w Jun 28, 2024 •

edited

Loading

shauray8 Jun 28, 2024

Bhavay-2001 commented Jun 28, 2024

yiyixuxu left a comment

yiyixuxu Jun 28, 2024

yiyixuxu Jun 28, 2024

yiyixuxu Jun 28, 2024

yiyixuxu Jun 28, 2024

yiyixuxu commented Jun 28, 2024

HuggingFaceDocBuilderDev commented Jun 28, 2024

yiyixuxu commented Jun 28, 2024

shauray8 commented Jun 29, 2024

yiyixuxu commented Jun 29, 2024


		return prompt_embeds, negative_prompt_embeds

		def encode_image(self, image, device, num_images_per_prompt, output_hidden_states=None):

add PAG support for SD architecture #8725

add PAG support for SD architecture #8725

Conversation

shauray8 commented Jun 28, 2024 • edited Loading

What does this PR do?

Before submitting

Who can review?

shauray8 Jun 28, 2024

Choose a reason for hiding this comment

yiyixuxu Jun 28, 2024

Choose a reason for hiding this comment

a-r-r-o-w Jun 28, 2024 • edited Loading

Choose a reason for hiding this comment

shauray8 Jun 28, 2024

Choose a reason for hiding this comment

Bhavay-2001 commented Jun 28, 2024

yiyixuxu left a comment

Choose a reason for hiding this comment

yiyixuxu Jun 28, 2024

Choose a reason for hiding this comment

yiyixuxu Jun 28, 2024

Choose a reason for hiding this comment

yiyixuxu Jun 28, 2024

Choose a reason for hiding this comment

yiyixuxu Jun 28, 2024

Choose a reason for hiding this comment

yiyixuxu commented Jun 28, 2024

HuggingFaceDocBuilderDev commented Jun 28, 2024

yiyixuxu commented Jun 28, 2024

shauray8 commented Jun 29, 2024

yiyixuxu commented Jun 29, 2024

shauray8 commented Jun 28, 2024 •

edited

Loading

a-r-r-o-w Jun 28, 2024 •

edited

Loading