Minor suggestions

[woctezuma]

I wanted to try the official code at https://huggingface.co/CompVis/stable-diffusion-safety-checker with:

from diffusers.pipelines.stable_diffusion.safety_checker import StableDiffusionSafetyChecker

model = StableDiffusionSafetyChecker.from_pretrained("CompVis/stable-diffusion-safety-checker").cuda()

But I have noticed that this requires an additional input, which is clip_input in the code below.

@torch.no_grad()
def forward(self, clip_input, images):
    # [...]
    return images, has_nsfw_concepts

So I am a bit confused by this possibly text argument for now... and discovered your repository right afterwards.

---

Note

Edit: This explains the two arguments: https://github.com/huggingface/diffusers/blob/d486f0e84669447b178569ad499eeb86c739b99e/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py#L505-L517

def run_safety_checker(self, image, device, dtype):
    if self.safety_checker is None:
        has_nsfw_concept = None
    else:
        if torch.is_tensor(image):
            feature_extractor_input = self.image_processor.postprocess(image, output_type="pil")
        else:
            feature_extractor_input = self.image_processor.numpy_to_pil(image)
        safety_checker_input = self.feature_extractor(feature_extractor_input, return_tensors="pt").to(device)
        image, has_nsfw_concept = self.safety_checker(
            images=image, clip_input=safety_checker_input.pixel_values.to(dtype)
        )
    return image, has_nsfw_concept

---

The code runs fine, but I would have a few suggestions:

1. is it possible to output a safety score (float) instead of a yes/no answer (bool)?
2. it is possible to apply the model to a batch of images, e.g. 8 images, so as to make the whole process faster?

Thank you for your attention!

[iyume]

1. I'm sorry that I am researching it too. I'm not an expert of CLIP.
2. Yes! Allowed types are described here: https://github.com/huggingface/transformers/blob/2c658b5a4282f2e824b4e23dc3bcda7ef27d5827/src/transformers/image_utils.py#L59

For 2, I typed in my code as Union[Image.Image, List[Image.Image]] which is a mistake, I'll fix it soon.

[woctezuma]

1. You could probably use the cosine similarity (between -1 and +1). cf. https://en.wikipedia.org/wiki/Cosine_similarity
2. Thanks!

[woctezuma]

Actually, len(res["bad_concepts"]) should do the trick based on https://github.com/huggingface/diffusers/blob/d486f0e84669447b178569ad499eeb86c739b99e/src/diffusers/pipelines/stable_diffusion/safety_checker.py#L84C11-L84C77

has_nsfw_concepts = [len(res["bad_concepts"]) > 0 for res in result]

[iyume]

The problem is that the network can't be exactly predict R15, R16, R17, R18. Current safety checker is trained for R15 detection and nobody could promise it to work well in R18 detection. Anymore I need more test result, or train another model.

[woctezuma]

In the end, I have reused some code from another one of my projects, which extracts image features:

• https://github.com/woctezuma/feature-extractor

to build a similar tool, which reports the IDs of the bad concepts:

• https://github.com/woctezuma/stable-diffusion-safety-checker

[woctezuma]

By the way, I have found the semantics behind the "bad concepts":

• https://github.com/LAION-AI/CLIP-based-NSFW-Detector/blob/main/safety_settings.yml

References:

• What are the original NSFW concepts used in the safety checker? huggingface/diffusers#476
• The behavior of safety_checker is ambiguous. huggingface/diffusers#866

[iyume]

Thanks to your share!

As a conclusion, safety checker works well on R15, and I would recommend nsfw_model for R18 which I'm currently using in another project.