Add IP-Adapter to StableDiffusionXLControlNetImg2ImgPipeline

[antoine-scenario]

Add IP-Adapter to StableDiffusionXLControlNetImg2ImgPipeline as it was missing so far.

Before submitting

• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[charchit7]

@antoine-scenario run make quality

[antoine-scenario]

@antoine-scenario run make quality

done 👍

[patrickvonplaten]

@yiyixuxu could you take a look here?

[yiyixuxu]

looking great! thank you :)

[yiyixuxu]

can we fix the test? you need to add the optional image_encoder component (None) here

diffusers/tests/pipelines/controlnet/test_controlnet_sdxl_img2img.py

Line 130 in 2db73f4

components = {

[antoine-scenario]

can we fix the test? you need to add the optional image_encoder component (None) here

diffusers/tests/pipelines/controlnet/test_controlnet_sdxl_img2img.py

Line 130 in 2db73f4

components = {

@yiyixuxu I did the fix you suggested but the test still fails 😕

[yiyixuxu]

ok can we find out what's the difference? I think you can add some print lines here to find out (i.e. print out pipe.components.keys()) and init_components.keys())

diffusers/tests/pipelines/test_pipelines_common.py

Line 570 in 79c380b

self.assertTrue(set(pipe.components.keys()) == set(init_components.keys()))

[antoine-scenario]

ok can we find out what's the difference? I think you can add some print lines here to find out (i.e. print out pipe.components.keys()) and init_components.keys())

diffusers/tests/pipelines/test_pipelines_common.py

Line 570 in 79c380b

self.assertTrue(set(pipe.components.keys()) == set(init_components.keys()))

do you have a command to run the test locally ? I followed the instructions from the doc and tried pytest tests/pipelines/test_pipelines_common.py but everything is being skipped.

Otherwise in the CI the print doesn't show.

[yiyixuxu]

@antoine-scenario
oh you should run this test file pytest tests/pipelines/controlnet/test_controlnet_sdxl_img2img.py

[antoine-scenario]

@antoine-scenario oh you should run this test file pytest tests/pipelines/controlnet/test_controlnet_sdxl_img2img.py

oh yes you're right ! everything seems to be fine by now 😇

[yiyixuxu]

cc @sayakpaul for a final review

[sayakpaul]

Very nice.

Brainstorming, not related to the PR: I wonder if we should work on a tutorial showing the different use cases of IP Adapters in light of various pipelines for which it's supported.

IP Adapters are getting increasingly popular, so I definitely think it's a good idea. @stevhliu WDYT?

[charchit7]

Very nice.

Brainstorming, not related to the PR: I wonder if we should work on a tutorial showing the different use cases of IP Adapters in light of various pipelines for which it's supported.

IP Adapters are getting increasingly popular, so I definitely think it's a good idea. @stevhliu WDYT?

Yes, IP Adapters quality is really good.

[stevhliu]

IP Adapters are getting increasingly popular, so I definitely think it's a good idea. @stevhliu WDYT?

We currently have a section for IP-Adapters here where we show off how to use it with various pipelines and for different tasks as well, so I don't necessarily think we need a new doc for it. But I think we can spin this section out into a standalone doc to increase visibility and keep adding to it. 🙂