From 8c6181d2dadc3b509df66c26195ea1b46d627b57 Mon Sep 17 00:00:00 2001 From: yiyixuxu Date: Thu, 23 Feb 2023 11:21:22 -1000 Subject: [PATCH 1/3] convert img2img.mdx into notebook doc --- docs/source/en/using-diffusers/img2img.mdx | 80 ++++++++++++++++++++-- 1 file changed, 74 insertions(+), 6 deletions(-) diff --git a/docs/source/en/using-diffusers/img2img.mdx b/docs/source/en/using-diffusers/img2img.mdx index c64d677686ba..ed4adaaf165d 100644 --- a/docs/source/en/using-diffusers/img2img.mdx +++ b/docs/source/en/using-diffusers/img2img.mdx @@ -12,7 +12,17 @@ specific language governing permissions and limitations under the License. # Text-Guided Image-to-Image Generation -The [`StableDiffusionImg2ImgPipeline`] lets you pass a text prompt and an initial image to condition the generation of new images. +[[open-in-colab]] + +The [`StableDiffusionImg2ImgPipeline`] lets you pass a text prompt and an initial image to condition the generation of new images. This tutorial shows how to use it for text-guided image-to-image generation with Stable Diffusion model. + +Before you begin, make sure you have all the necessary libraries installed: + +```python +!pip install diffusers==0.11.1 transformers ftfy accelerate +``` + +Get started by creating a [`StableDiffusionImg2ImgPipeline`] with a pretrained Stable Diffusion model. ```python import torch @@ -21,25 +31,83 @@ from PIL import Image from io import BytesIO from diffusers import StableDiffusionImg2ImgPipeline +``` + +Load the pipeline -# load the pipeline +```python device = "cuda" pipe = StableDiffusionImg2ImgPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16).to( device ) +``` + +Download an initial image and preprocess it so we can pass it to the pipeline. -# let's download an initial image +```python url = "https://raw.githubusercontent.com/CompVis/stable-diffusion/main/assets/stable-samples/img2img/sketch-mountains-input.jpg" response = requests.get(url) init_image = Image.open(BytesIO(response.content)).convert("RGB") init_image.thumbnail((768, 768)) +init_image +``` + +![img](https://huggingface.co/datasets/YiYiXu/test-doc-assets/resolve/main/image_2_image_using_diffusers_cell_8_output_0.jpeg) +Define the prompt and run the pipeline. + +```python prompt = "A fantasy landscape, trending on artstation" +``` + + + +`strength` is a value between 0.0 and 1.0, that controls the amount of noise that is added to the input image. Values that approach 1.0 allow for lots of variations but will also produce images that are not semantically consistent with the input. + + + +Let's generate two images with same pipeline and seed, but with different values for `strength` + +```python +generator = torch.Generator(device=device).manual_seed(1024) +image = pipe(prompt=prompt, image=init_image, strength=0.75, guidance_scale=7.5, generator=generator).images[0] +``` + +```python +image +``` + +![img](https://huggingface.co/datasets/YiYiXu/test-doc-assets/resolve/main/image_2_image_using_diffusers_cell_13_output_0.jpeg) + + +```python +image = pipe(prompt=prompt, image=init_image, strength=0.5, guidance_scale=7.5, generator=generator).images[0] +image +``` + +![img](https://huggingface.co/datasets/YiYiXu/test-doc-assets/resolve/main/image_2_image_using_diffusers_cell_14_output_1.jpeg) -images = pipe(prompt=prompt, image=init_image, strength=0.75, guidance_scale=7.5).images -images[0].save("fantasy_landscape.png") +As you can see, when using a lower value for `strength`, the generated image is more closer to the original `image` + +Now let's use a different scheduler - [LMSDiscreteScheduler](https://huggingface.co/docs/diffusers/api/schedulers#diffusers.LMSDiscreteScheduler) + +```python +from diffusers import LMSDiscreteScheduler + +lms = LMSDiscreteScheduler.from_config(pipe.scheduler.config) +pipe.scheduler = lms ``` -You can also run this example on colab [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/image_2_image_using_diffusers.ipynb) + +```python +generator = torch.Generator(device=device).manual_seed(1024) +image = pipe(prompt=prompt, image=init_image, strength=0.75, guidance_scale=7.5, generator=generator).images[0] +``` + +```python +image +``` + +![img](https://huggingface.co/datasets/YiYiXu/test-doc-assets/resolve/main/image_2_image_using_diffusers_cell_19_output_0.jpeg) From c7d137139682fc49aaeb104cf2d0f99204eda283 Mon Sep 17 00:00:00 2001 From: yiyixuxu Date: Thu, 23 Feb 2023 11:22:56 -1000 Subject: [PATCH 2/3] fix --- docs/source/en/using-diffusers/img2img.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/en/using-diffusers/img2img.mdx b/docs/source/en/using-diffusers/img2img.mdx index ed4adaaf165d..d36ea272fbe1 100644 --- a/docs/source/en/using-diffusers/img2img.mdx +++ b/docs/source/en/using-diffusers/img2img.mdx @@ -18,7 +18,7 @@ The [`StableDiffusionImg2ImgPipeline`] lets you pass a text prompt and an initia Before you begin, make sure you have all the necessary libraries installed: -```python +```bash !pip install diffusers==0.11.1 transformers ftfy accelerate ``` From e67c575317e072dc318e88bc4bdf3163568fea24 Mon Sep 17 00:00:00 2001 From: YiYi Xu Date: Sun, 5 Mar 2023 17:34:43 -1000 Subject: [PATCH 3/3] Update docs/source/en/using-diffusers/img2img.mdx Co-authored-by: Patrick von Platen --- docs/source/en/using-diffusers/img2img.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/en/using-diffusers/img2img.mdx b/docs/source/en/using-diffusers/img2img.mdx index d36ea272fbe1..9f05366bb122 100644 --- a/docs/source/en/using-diffusers/img2img.mdx +++ b/docs/source/en/using-diffusers/img2img.mdx @@ -19,7 +19,7 @@ The [`StableDiffusionImg2ImgPipeline`] lets you pass a text prompt and an initia Before you begin, make sure you have all the necessary libraries installed: ```bash -!pip install diffusers==0.11.1 transformers ftfy accelerate +!pip install diffusers transformers ftfy accelerate ``` Get started by creating a [`StableDiffusionImg2ImgPipeline`] with a pretrained Stable Diffusion model.