[Diffusers] Add text-guided image to image #223

patrickvonplaten · 2023-04-04T08:53:14Z

@Narsil @osanseviero would be great to get a review here :-)

…ommunity into add_img2img_diffusers

patrickvonplaten · 2023-04-18T10:50:47Z

@Narsil would be amazing to get a quick review here to better see what still needs to be done.

I rebased the PR after https://github.com/huggingface/api-inference-community/pull/230/files was merged.

docker_images/diffusers/app/pipelines/image_to_image.py

Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>

Narsil · 2023-04-18T16:19:40Z

@patrickvonplaten the model seems incorrect :

{"error":"unknown error","warnings":["There was an inference error: The size of tensor a (148) must match the size of tensor b (37) at non-singleton dimension 3"]}

https://huggingface.co/hf-internal-testing/tiny-controlnet

Did I do something wrong ?

patrickvonplaten · 2023-04-18T18:21:19Z

@patrickvonplaten the model seems incorrect :
{"error":"unknown error","warnings":["There was an inference error: The size of tensor a (148) must match the size of tensor b (37) at non-singleton dimension 3"]}
https://huggingface.co/hf-internal-testing/tiny-controlnet

Did I do something wrong ?

I'll check tomorrow!

osanseviero

Very cool! Thanks a lot 🔥

docker_images/diffusers/app/pipelines/image_to_image.py

osanseviero · 2023-04-18T19:47:24Z

docker_images/diffusers/app/pipelines/image_to_image.py

+            with open(config_file, "r") as f:
+                config_dict = json.load(f)
+
+            is_controlnet = config_dict.get("_class_name", None) == "ControlNetModel"


_class_name could be added to the ModelInfo object so you don't need to load the whole config here. Internal PR for that https://github.com/huggingface/moon-landing/pull/6067

Ok waiting for #6067

docker_images/diffusers/app/pipelines/image_to_image.py

osanseviero · 2023-04-18T19:53:41Z

docker_images/diffusers/tests/test_api.py

-TESTABLE_MODELS: Dict[str, str] = {"text-to-image": "CompVis/ldm-text2im-large-256"}
+TESTABLE_MODELS: Dict[str, str] = {
+    "text-to-image": "hf-internal-testing/tiny-stable-diffusion-pipe",
+    "image-to-image": "hf-internal-testing/tiny-controlnet",


Would be great to add a second test for image-to-image that is not controlNet. An example of testing multiple models can be seen here, but it will need some slight changes in the tests https://github.com/huggingface/api-inference-community/blob/main/docker_images/speechbrain/tests/test_api.py#L11

docker_images/diffusers/tests/test_api_image_to_image.py

patrickvonplaten · 2023-04-19T08:05:50Z

docker_images/diffusers/app/pipelines/image_to_image.py

+                **kwargs,
+            )
+        else:
+            self.ldm = DiffusionPipeline.from_pretrained(


This can load both StableDiffusionInstructPix2PixPipeline, StableDiffusionImg2ImgPipeline and AltDiffusionImg2ImgPipeline depending on how the model_index.json is defined

patrickvonplaten · 2023-04-19T08:45:55Z

@patrickvonplaten the model seems incorrect :
{"error":"unknown error","warnings":["There was an inference error: The size of tensor a (148) must match the size of tensor b (37) at non-singleton dimension 3"]}
https://huggingface.co/hf-internal-testing/tiny-controlnet

Did I do something wrong ?

Sorry there was a problem with the config! Should be fixed now:

from diffusers import ControlNetModel, StableDiffusionControlNetPipeline
import numpy
from PIL import Image

array = numpy.random.rand(64, 64, 3) * 255
image = Image.fromarray(array.astype('uint8')).convert('RGB')

model_id = "hf-internal-testing/tiny-stable-diffusion-pipe-no-safety"  # == tags?.base_model

controlnet = ControlNetModel.from_pretrained("hf-internal-testing/tiny-controlnet")

pipeline = StableDiffusionControlNetPipeline.from_pretrained(model_id, controlnet=controlnet)


out_img = pipeline("hey", image).images[0]

Co-authored-by: Omar Sanseviero <osanseviero@gmail.com>

docker_images/diffusers/tests/test_api_image_to_image.py

patrickvonplaten · 2023-04-19T09:07:41Z

The second main use case (stable diffusion image to image and pix2pix) can be seen in this example:

#!/usr/bin/env python3
from diffusers import UNet2DConditionModel, StableDiffusionInstructPix2PixPipeline
import numpy
from PIL import Image

array = numpy.random.rand(64, 64, 3) * 255
image = Image.fromarray(array.astype('uint8')).convert('RGB')

model_id = "hf-internal-testing/tiny-stable-diffusion-pix2pix"

pipeline = StableDiffusionInstructPix2PixPipeline.from_pretrained(model_id)

out_img = pipeline("hey", image).images[0]

docker_images/diffusers/app/pipelines/image_to_image.py

osanseviero

Thanks a lot! Approving pending the internal PR and change to model_info

osanseviero · 2023-04-19T10:47:13Z

docker_images/common/tests/test_api.py

    "sentence-similarity",
    "fill-mask",
    "table-question-answering",
    "summarization",
    "text2text-generation",
    "text-classification",
-    "text-to-image",
-    "text-to-speech",
-    "token-classification",


The removed ones should be reverted

We should also add image-to-image here

They are duplicates "text-to-image" exists in there already. Cleaning up ;-)

patrickvonplaten · 2023-04-19T10:56:22Z

Ok things work now! @Narsil was so kind to show me all the commands!

./manage.py start hf-internal-testing/tiny-controlnet --gpu

url -X POST --data-binary "@tests/samples/plane.jpg" http://localhost:8000

patrickvonplaten · 2023-04-19T10:57:09Z

@Narsil, feel free to merge once ok for you

Narsil · 2023-04-19T12:57:54Z

Tests are failing uniquely because tiny-controlnet is slow to run on CPU. But it's working.

patrickvonplaten added 2 commits April 4, 2023 10:53

[Diffusers] Add image to image

009ee02

fix some stuff

c1e8bca

patrickvonplaten changed the title ~~[Diffusers] Add image to image~~ [Diffusers] Add text-guided image to image Apr 4, 2023

patrickvonplaten added 3 commits April 4, 2023 11:17

make style

13656ec

make style

2523d3a

Merge branch 'main' of https://github.com/huggingface/api-inference-c…

b921893

…ommunity into add_img2img_diffusers

patrickvonplaten requested review from Narsil and osanseviero April 18, 2023 10:51

Narsil reviewed Apr 18, 2023

View reviewed changes

docker_images/diffusers/app/pipelines/image_to_image.py Outdated Show resolved Hide resolved

patrickvonplaten and others added 2 commits April 18, 2023 16:38

Update docker_images/diffusers/app/pipelines/image_to_image.py

8228bab

Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>

Fixing diffusers.

14404b8

osanseviero reviewed Apr 18, 2023

View reviewed changes

osanseviero mentioned this pull request Apr 18, 2023

Improve Diffusers Examples huggingface/hub-docs#747

Merged

radames mentioned this pull request Apr 18, 2023

Image-to-Image huggingface/huggingface.js#157

Closed

This comment was marked as resolved.

Sign in to view

patrickvonplaten commented Apr 19, 2023

View reviewed changes

Apply suggestions from code review

07c9495

Co-authored-by: Omar Sanseviero <osanseviero@gmail.com>

patrickvonplaten commented Apr 19, 2023

View reviewed changes

docker_images/diffusers/tests/test_api_image_to_image.py Outdated Show resolved Hide resolved

patrickvonplaten commented Apr 19, 2023

View reviewed changes

docker_images/diffusers/tests/test_api_image_to_image.py Outdated Show resolved Hide resolved

Apply suggestions from code review

1692bc1

patrickvonplaten added 3 commits April 19, 2023 11:15

up

b996948

up

d3ad2b2

make style

a7c64ee

patrickvonplaten commented Apr 19, 2023

View reviewed changes

docker_images/diffusers/app/pipelines/image_to_image.py Outdated Show resolved Hide resolved

Apply suggestions from code review

6a764c2

osanseviero approved these changes Apr 19, 2023

View reviewed changes

finalize

de62de4

patrickvonplaten requested a review from Narsil April 19, 2023 10:56

Narsil merged commit fcf17c8 into main Apr 19, 2023

Narsil deleted the add_img2img_diffusers branch April 19, 2023 12:58

coyotte508 mentioned this pull request Apr 23, 2023

SAM on JS? (Segment Anything Model) huggingface/huggingface.js#161

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Diffusers] Add text-guided image to image #223

[Diffusers] Add text-guided image to image #223

patrickvonplaten commented Apr 4, 2023 •

edited

Loading

patrickvonplaten commented Apr 18, 2023

Narsil commented Apr 18, 2023 •

edited

Loading

patrickvonplaten commented Apr 18, 2023

osanseviero left a comment

osanseviero Apr 18, 2023

patrickvonplaten Apr 19, 2023

osanseviero Apr 18, 2023

patrickvonplaten Apr 19, 2023

This comment was marked as resolved.

patrickvonplaten Apr 19, 2023 •

edited

Loading

patrickvonplaten commented Apr 19, 2023 •

edited

Loading

patrickvonplaten commented Apr 19, 2023

osanseviero left a comment

osanseviero Apr 19, 2023

patrickvonplaten Apr 19, 2023

patrickvonplaten commented Apr 19, 2023

patrickvonplaten commented Apr 19, 2023

Narsil commented Apr 19, 2023

[Diffusers] Add text-guided image to image #223

[Diffusers] Add text-guided image to image #223

Conversation

patrickvonplaten commented Apr 4, 2023 • edited Loading

patrickvonplaten commented Apr 18, 2023

Narsil commented Apr 18, 2023 • edited Loading

patrickvonplaten commented Apr 18, 2023

osanseviero left a comment

Choose a reason for hiding this comment

osanseviero Apr 18, 2023

Choose a reason for hiding this comment

patrickvonplaten Apr 19, 2023

Choose a reason for hiding this comment

osanseviero Apr 18, 2023

Choose a reason for hiding this comment

patrickvonplaten Apr 19, 2023

Choose a reason for hiding this comment

This comment was marked as resolved.

patrickvonplaten Apr 19, 2023 • edited Loading

Choose a reason for hiding this comment

patrickvonplaten commented Apr 19, 2023 • edited Loading

patrickvonplaten commented Apr 19, 2023

osanseviero left a comment

Choose a reason for hiding this comment

osanseviero Apr 19, 2023

Choose a reason for hiding this comment

patrickvonplaten Apr 19, 2023

Choose a reason for hiding this comment

patrickvonplaten commented Apr 19, 2023

patrickvonplaten commented Apr 19, 2023

Narsil commented Apr 19, 2023

patrickvonplaten commented Apr 4, 2023 •

edited

Loading

Narsil commented Apr 18, 2023 •

edited

Loading

patrickvonplaten Apr 19, 2023 •

edited

Loading

patrickvonplaten commented Apr 19, 2023 •

edited

Loading