OSError: h94/IP-Adapter does not appear to have a file named models\image_encoder\config.json

[tin2tin]

Describe the bug

I'm getting this error:

OSError: h94/IP-Adapter does not appear to have a file named models\image_encoder\config.json. Checkout 'https://huggingface.co/h94/IP-Adapter/main' for available files.

When running the IP-Adapter example from here: https://huggingface.co/docs/diffusers/main/en/using-diffusers/loading_adapters

Reproduction

Run the IP-Adapter example from here: https://huggingface.co/docs/diffusers/main/en/using-diffusers/loading_adapters

import torch
from diffusers import StableDiffusionPipeline, DDIMScheduler
from diffusers.utils import load_image

noise_scheduler = DDIMScheduler(
    num_train_timesteps=1000,
    beta_start=0.00085,
    beta_end=0.012,
    beta_schedule="scaled_linear",
    clip_sample=False,
    set_alpha_to_one=False,
    steps_offset=1
)

pipeline = StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    torch_dtype=torch.float16,
    scheduler=noise_scheduler,
).to("cuda")

pipeline.load_ip_adapter("h94/IP-Adapter", subfolder="models", weight_name="ip-adapter-full-face_sd15.bin")

pipeline.set_ip_adapter_scale(0.7)

image = load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/ai_face2.png")

generator = torch.Generator(device="cpu").manual_seed(33)

image = pipeline(
    prompt="A photo of a girl wearing a black dress, holding red roses in hand, upper body, behind is the Eiffel Tower",
    ip_adapter_image=image,
    negative_prompt="monochrome, lowres, bad anatomy, worst quality, low quality", 
    num_inference_steps=50, num_images_per_prompt=1, width=512, height=704,
    generator=generator,
).images[0]

OSError: h94/IP-Adapter does not appear to have a file named models\image_encoder\config.json. Checkout 'https://huggingface.co/h94/IP-Adapter/main' for available files.

Logs

OSError: h94/IP-Adapter does not appear to have a file named models\image_encoder\config.json. Checkout 'https://huggingface.co/h94/IP-Adapter/main' for available files.

System Info

Win 11
Diffusers 0.25.0

Who can help?

@sayakpaul @yiyixuxu @DN6 @patrickvonplaten

[fabiorigano]

hi @tin2tin, I encountered the same issue on Windows, it is caused by the different convention in path generation.

To solve this, I separately load the CLIPVisionModelWithProjection and pass it to the pipeline as it follows:

from transformers import CLIPVisionModelWithProjection
image_encoder = CLIPVisionModelWithProjection.from_pretrained(
    "h94/IP-Adapter",
    subfolder="models/image_encoder",
    torch_dtype=torch.float16,
).to("cuda")

pipeline = StableDiffusionPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5",
    torch_dtype=torch.float16,
    scheduler=noise_scheduler,
    image_encoder=image_encoder,
).to("cuda")

(Or you also can set pipeline.image_encoder = image_encoder after creating the pipeline.)

Then you can load the IP Adapter