unCLIP image variation #1781

williamberman · 2022-12-20T06:11:37Z

Adds an unclip image variation pipeline

Converting the text to image pipeline to image variation

I uploaded the pipeline to https://huggingface.co/fusing/karlo-image-variations-diffusers if you want to skip this

From the diffusers root directory:

$ python scripts/convert_unclip_txt2img_to_image_variation.py --dump_path <path to save model>

Using the model

import os
os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8"

from diffusers import UnCLIPImageVariationPipeline
import torch
import random
import numpy as np
import PIL
from PIL import Image


def set_seed(seed: int):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
set_seed(0)

torch.backends.cuda.matmul.allow_tf32 = False
torch.use_deterministic_algorithms(True)
torch.set_printoptions(precision=40)

def image_grid(imgs, rows, cols):
    assert len(imgs) == rows*cols

    w, h = imgs[0].size
    grid = Image.new('RGB', size=(cols*w, rows*h))
    
    for i, img in enumerate(imgs):
        grid.paste(img, box=(i%cols*w, i//cols*h))
    return grid


pipe = UnCLIPImageVariationPipeline.from_pretrained("fusing/karlo-image-variations-diffusers")
pipe = pipe.to('cuda')

# See image below to use as input
image = PIL.Image.open("./test.jpg") 

images = pipe(image, num_images_per_prompt=4).images

image_grid(images, 1, 4).save('./out.jpg')

HuggingFaceDocBuilderDev · 2022-12-20T06:16:46Z

The documentation is not available anymore as the PR was closed or merged.

patil-suraj

Looks good to me, thanks a lot for adding the pipeline! Would be nice to add copied from .. comments wherever possible.

src/diffusers/pipelines/unclip/pipeline_unclip_image_variation.py

pcuenca

Awesome!

pcuenca · 2022-12-21T17:01:13Z

src/diffusers/pipelines/unclip/pipeline_unclip_image_variation.py

@@ -0,0 +1,454 @@
+# Copyright 2022 Kakao Brain and The HuggingFace Team. All rights reserved.


Is this so? Or is it just Hugging Face for the code? Just wondering, no idea how those things work!

This is the licensing @patrickvonplaten added to the text to image pipeline. We should probably clarify with him :)

Think it's fine to mention Kakao Brain, since we use their code as a reference when implementing it here.

src/diffusers/pipelines/unclip/pipeline_unclip_image_variation.py

tests/pipelines/unclip/test_unclip.py

pcuenca · 2022-12-21T17:23:13Z

scripts/convert_unclip_txt2img_to_image_variation.py

+import argparse
+
+from diffusers import UnCLIPImageVariationPipeline, UnCLIPPipeline
+from transformers import CLIPImageProcessor, CLIPVisionModelWithProjection
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+
+    parser.add_argument("--dump_path", default=None, type=str, required=True, help="Path to the output model.")
+
+    parser.add_argument(
+        "--txt2img_unclip",
+        default="kakaobrain/karlo-v1-alpha",
+        type=str,
+        required=False,
+        help="The pretrained txt2img unclip.",
+    )
+
+    args = parser.parse_args()
+
+    txt2img = UnCLIPPipeline.from_pretrained(args.txt2img_unclip)
+
+    feature_extractor = CLIPImageProcessor()
+    image_encoder = CLIPVisionModelWithProjection.from_pretrained("openai/clip-vit-large-patch14")
+
+    img2img = UnCLIPImageVariationPipeline(
+        decoder=txt2img.decoder,
+        text_encoder=txt2img.text_encoder,
+        tokenizer=txt2img.tokenizer,
+        text_proj=txt2img.text_proj,
+        feature_extractor=feature_extractor,


This is very informative, but I'm not sure we store this kind of scripts in the repo. The ones in the folder are usually about converting weights from other checkpoints. What do you think @patil-suraj?

Happy to remove and just put in the PR description! lmk @patil-suraj

Yeah, think no need to have this script.

This is better than not having a script at all. Think it's totally fine to leave it here as is. The main purpose of the scripts is really so that the user can convert the checkpoints themselves - I'm fine with the way it is. Better would be to directly convert from the original checkpoint, but for me this is ok as well and def better than not having anything.

patil-suraj

LGTM!
Also, think it would be nice to add a doc page explaining the unCLIP pipelines. It's the first cascaded pipeline in diffusers, so would be nice to document the different components and how they work.

patil-suraj · 2022-12-23T14:20:14Z

src/diffusers/pipelines/unclip/pipeline_unclip_image_variation.py

@@ -0,0 +1,454 @@
+# Copyright 2022 Kakao Brain and The HuggingFace Team. All rights reserved.


Think it's fine to mention Kakao Brain, since we use their code as a reference when implementing it here.

patrickvonplaten

Nice looks good to me!

@pcuenca

* unCLIP image variation * remove prior comment re: @pcuenca * stable diffusion -> unCLIP re: @pcuenca * add copy froms re: @patil-suraj

williamberman force-pushed the unclip_image_variation branch 4 times, most recently from 241d482 to 54176fc Compare December 21, 2022 02:46

unCLIP image variation

12833eb

williamberman force-pushed the unclip_image_variation branch from 32a14e2 to 12833eb Compare December 21, 2022 03:14

williamberman marked this pull request as ready for review December 21, 2022 03:14

williamberman changed the title ~~Unclip image variation~~ unCLIP image variation Dec 21, 2022

williamberman requested review from patil-suraj, anton-l, pcuenca, patrickvonplaten and yiyixuxu December 21, 2022 03:20

patil-suraj approved these changes Dec 21, 2022

View reviewed changes

src/diffusers/pipelines/unclip/pipeline_unclip_image_variation.py Outdated Show resolved Hide resolved

src/diffusers/pipelines/unclip/pipeline_unclip_image_variation.py Show resolved Hide resolved

pcuenca approved these changes Dec 21, 2022

View reviewed changes

williamberman added 3 commits December 21, 2022 09:37

remove prior comment re: @pcuenca

07db35d

stable diffusion -> unCLIP re: @pcuenca

180ee7c

add copy froms re: @patil-suraj

53cee50

williamberman requested review from pcuenca and patil-suraj December 21, 2022 17:57

patil-suraj approved these changes Dec 23, 2022

View reviewed changes

patrickvonplaten approved these changes Dec 24, 2022

View reviewed changes

patrickvonplaten reviewed Dec 24, 2022

View reviewed changes

patrickvonplaten merged commit 53c8147 into huggingface:main Dec 28, 2022

yoonseokjin pushed a commit to yoonseokjin/diffusers that referenced this pull request Dec 25, 2023

unCLIP image variation (huggingface#1781)

686fb04

* unCLIP image variation * remove prior comment re: @pcuenca * stable diffusion -> unCLIP re: @pcuenca * add copy froms re: @patil-suraj

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

unCLIP image variation #1781

unCLIP image variation #1781

williamberman commented Dec 20, 2022 •

edited

Loading

HuggingFaceDocBuilderDev commented Dec 20, 2022 •

edited

Loading

patil-suraj left a comment

pcuenca left a comment

pcuenca Dec 21, 2022

williamberman Dec 21, 2022

patil-suraj Dec 23, 2022

pcuenca Dec 21, 2022

williamberman Dec 21, 2022

patil-suraj Dec 23, 2022

patrickvonplaten Dec 24, 2022

patil-suraj left a comment

patil-suraj Dec 23, 2022

patrickvonplaten left a comment

		@@ -0,0 +1,454 @@
		# Copyright 2022 Kakao Brain and The HuggingFace Team. All rights reserved.

unCLIP image variation #1781

unCLIP image variation #1781

Conversation

williamberman commented Dec 20, 2022 • edited Loading

Converting the text to image pipeline to image variation

Using the model

HuggingFaceDocBuilderDev commented Dec 20, 2022 • edited Loading

patil-suraj left a comment

Choose a reason for hiding this comment

pcuenca left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patil-suraj left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

patrickvonplaten left a comment

Choose a reason for hiding this comment

williamberman commented Dec 20, 2022 •

edited

Loading

HuggingFaceDocBuilderDev commented Dec 20, 2022 •

edited

Loading