Shouldn't it be "pil_img" instead of "input"?

[PytaichukBohdan]

clip-guided-diffusion/cgd/clip_util.py

Line 59 in 54d273e

pil_img = resize_right.resize(input, out_shape=[smallest_side],

[PytaichukBohdan]

Or even np.array(pil_img)
But still, the code isn't running properly after those changes

Got an error:
RuntimeError: adaptive_avg_pool2d(): Expected input to have non-zero size for non-batch dimensions, but input has sizes [1, 1066, 0, 226] with dimension 2 being empty

[afiaka87]

@PytaichukBohdan

Indeed, thanks for filing an issue! I'll patch it.

[afiaka87]

c81fde0
should have fixed it. let me know if it works for you.

[afiaka87]

@PytaichukBohdan - ah yes, another issue you may be facing is that you have to use multiples of (i believe) 16 (for size < 128) and 32 for the offsets.

[htoyryla]

Still does not work. ResizeRight is expecting either a numpy array or a torch tensor, now it gets a PIL image which does not have shape attribute.

This is what I tried and at least it runs without an error

   t_img = tvf.to_tensor(pil_img)
   t_img = resize_right.resize(t_img, out_shape=(smallest_side, smallest_side),
                                 interp_method=lanczos3, support_sz=None,
                                 antialiasing=True, by_convs=False, scale_tolerance=None)
   batch = make_cutouts(t_img.unsqueeze(0).to(device)) 

I am not sure what was intended here as to the output shape. As it was it made 1024x512 from 1024x1024 original, for image_size 512, now this makes 512x512.

I am not using offsets, BTW.

As to the images produced, can't see much happening, but I guess that is another story. According to my experience guidance by comparing CLIP encoded images is not very useful as such, so I'll probably go my own way to add other ways as to image based guidance. This might depend on the kind of images I work with and how. More visuality than semantics.

PS. I see now that the init image actually means using perceptual losses as guidance, rather than initialising something (like one can do with VQGAN latents for instance). So that's more like what I am after.

[htoyryla]

Or even np.array(pil_img) But still, the code isn't running properly after those changes

Got an error: RuntimeError: adaptive_avg_pool2d(): Expected input to have non-zero size for non-batch dimensions, but input has sizes [1, 1066, 0, 226] with dimension 2 being empty

I tried that also first. I guess it fails as the numpy array has shape (h, w, c) while (I think) (c, h, w) is expected. Using to_tensor takes care of this.