Switch to SD v1.5

[143f86b6-91b7-47e4-808b-18779e22f2dd]

Is it possible to use this in v1.5 instead of v1.4?

[0xdevalias]

This is documented in the README already:

• https://github.com/cloneofsimo/paint-with-words-sd#using-other-fine-tuned-models

---

You can see that pww_load_tools takes a hf_model_path and/or local_model_path here:

paint-with-words-sd/paint_with_words/paint_with_words.py

Lines 89 to 93 in 4c95a11

	def pww_load_tools(
	device: str = "cuda:0",
	scheduler_type=LMSDiscreteScheduler,
	local_model_path: Optional[str] = None,
	hf_model_path: Optional[str] = None,

pww_load_tools is called by paint_with_words, which can also be passed hf_model_path and/or local_model_path, with hf_model_path defaulting to "CompVis/stable-diffusion-v1-4"

paint-with-words-sd/paint_with_words/paint_with_words.py

Lines 213 to 238 in 4c95a11

	def paint_with_words(
	color_context: Dict[Tuple[int, int, int], str] = {},
	color_map_image: Optional[Image.Image] = None,
	input_prompt: str = "",
	num_inference_steps: int = 30,
	guidance_scale: float = 7.5,
	seed: int = 0,
	scheduler_type=LMSDiscreteScheduler,
	device: str = "cuda:0",
	weight_function: Callable = lambda w, sigma, qk: 0.1
	* w
	* math.log(sigma + 1)
	* qk.max(),
	local_model_path: Optional[str] = None,
	hf_model_path: Optional[str] = "CompVis/stable-diffusion-v1-4",
	preloaded_utils: Optional[Tuple] = None,
	unconditional_input_prompt: str = "",
	):
	vae, unet, text_encoder, tokenizer, scheduler = (
	pww_load_tools(
	device,
	scheduler_type,
	local_model_path=local_model_path,
	hf_model_path=hf_model_path,
	)

So it looks like it may be even easier than the README suggests, and you should just be able to pass "runwayml/stable-diffusion-v1-5" directly into the hf_model_path parameter of paint_with_words as it's called in the 'Basic Usage' section:

• https://github.com/cloneofsimo/paint-with-words-sd#basic-usage

Something like this would presumably work:

img = paint_with_words(
    color_context=color_context,
    color_map_image=color_map_image,
    input_prompt=input_prompt,
    num_inference_steps=30,
    guidance_scale=7.5,
    device="cuda:0",
+   hf_model_path="runwayml/stable-diffusion-v1-5"
)

[0xdevalias]

I made a simple colab example just now, you can try it out here: paint-with-words colab

Originally posted by @shreydan in #4 (comment)

You can confirm the above hf_model_path change on the colab mentioned here by editing the cell that has the img = paint_with_words( (first cell after the 'Run Model' heading)

We can also explicitly set the seed parameter here as well, to make comparisons between the 2 models easier.

Original version using SD v1.4

New version using SD v1.5

[hiss-remi]

You can see that pww_load_tools takes a hf_model_path and/or local_model_path here:

One advantage of this method that you can also change the scheduler, which might be desirable for performance or quality issues. (I found recently that the model I'm using often has blatant graphical issues even with a large step count on LMS, but EulerAncestralDiscreteScheduler or EulerDiscreteScheduler do much better. Just don't forget to import your scheduler of choice from diffusers. (Something the example currently doesn't show.)

[cloneofsimo]

Nice note @hiss-remi , ill add this feature as well.