Keep getting Rickrolled

[mokiat]

Hi,

I am trying to generate some AI art with your code but I keep getting broken results and half of them are the Rickroll meme photo.

Following is the command I used to generate this:

python 'scripts/txt2img.py' \
  --ckpt 'sd-v1-3.ckpt' \
  --config "configs/stable-diffusion/v1-inference.yaml" \
  --n_samples 1 \
  --W 256 \
  --H 256 \
  --plms \
  --n_iter 50 \
  --prompt "photo of a cat"

I tried the v1.4 model and get similar results. My videocard is an RTX 2080 Super, 8G VRAM, so I can't try any larger resolutions (they just fail with OOM errors).

I've tried simple commands, as in the README of the repository, or alternative variations of the above command and still get the same result.

At this point in time I am starting to worry if I have downloaded a virus, though I checked the sha values...

Any help would be welcome.

[magnusviri]

You don't have a virus. It's coming from this and this.

[mokiat]

Well, my prompts so far have been the original one a photograph of an astronaut riding a horse, a car, photo of a cat. I think that's it.

Maybe some heuristic isn't working properly and is evaluating it as NSFW (looking at this line:

stable-diffusion/scripts/txt2img.py

Line 92 in 69ae4b3

if has_nsfw_concept[i]:

).

Either way, the results are with much lower quality that what friends are getting. Could it be the 256x256 resolution that is messing everything up?

[mokiat]

OK, so I managed to get it working. All I did was add the following line in scripts/txt2img.py:

model.half()

Not sure what it does, watched it in some YouTube clip. However, it allowed me to generate at 512x512 on my card and now it seems to generate images of fine quality and no rickrolls.

[breadbrowser]

demo here https://huggingface.co/spaces/stabilityai/stable-diffusion.

[magnusviri]

@mokiat, where did you add that line?

[mokiat]

@magnusviri , I added it immediately after this line:

stable-diffusion/scripts/txt2img.py

Line 54 in 69ae4b3

model = instantiate_from_config(config.model)

so that it becomes:

model = instantiate_from_config(config.model)
model.half()

As far as I understand it, this converts the model from 32bit float to 16bit float, which gives some VRAM room to the GPU. Enough so that 512x512 works.

I also figured out how to get img2img to use float16 as well. You need to do the same thing after the following line:

stable-diffusion/scripts/img2img.py

Line 34 in 69ae4b3

model = instantiate_from_config(config.model)

And you also need to change float32 to float16 on the following line:

stable-diffusion/scripts/img2img.py

Line 54 in 69ae4b3

image = np.array(image).astype(np.float32) / 255.0

[Cabbagec]

It's the safety checker's doing. When NSFW features are detected, it replaces the image with rick.jpg. But the safety checker doesn't seem to work very well, it just randomly removes some of the images.

I've added an --unsafe command line flag to disable it temporarily, just add it to end of your command. Also, another side effect is that the safety checker model will not be loaded with this flag on, saving some gigs of your precious VRAM :D

See #79

[mokiat]

@Cabbagec , nice. However, I think in general the model doesn't work well with 256x256. Just look at the cat photos above.
Going with float16 gives users with 8GB video cards a chance to generate at 512x512.
It's interesting to see if the resolution can indeed be pushed further without the safety check consuming RAM.

[mokiat]

I am closing this, as there is a workaround and the reason for the Rickroll image was determined.