Shieldgemma2

[RyanMullins]

What does this PR do?

Fixes # (issue)

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[github-actions]

Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the Ready for review button (at the bottom of the PR page).

[ArthurZucker]

Thanks, small nits but good to go otherwise!

[ain-soph]

@RyanMullins Submit a PR for this branch as bug fixes.
RyanMullins#2

• Fix a bug in readme example snippet (return_tensors="pt" for Processor call)
• Use public image link as example (from https://huggingface.co/google/shieldgemma-2-4b-it)
• Fix typo

[ain-soph]

There is still an logging issue that outputs incorrect info:

from PIL import Image
import requests
from transformers import AutoProcessor, ShieldGemma2ForImageClassification

model_id = "google/shieldgemma-2-4b-it"
model = ShieldGemma2ForImageClassification.from_pretrained(model_id, device_map="auto")
processor = AutoProcessor.from_pretrained(model_id)

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"
image = Image.open(requests.get(url, stream=True).raw)

custom_policies = {
    "key_a": "descrition_a",
    "key_b": "descrition_b",
}

inputs = processor(
    images=[image],
    custom_policies=custom_policies,
    policies=["dangerous", "key_b"],
    return_tensors="pt",
).to(model.device)

output = model(**inputs)
print(output.probabilities)

$ python test.py
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:06<00:00,  3.12s/it]
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.50, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
Keyword argument `custom_policies` is not a valid argument for this processor and will be ignored.
Keyword argument `policies` is not a valid argument for this processor and will be ignored.
tensor([[5.3613e-11, 1.0000e+00],
        [3.7379e-09, 1.0000e+00]], device='cuda:1', grad_fn=<ToCopyBackward0>)

As you can see above, the argument custom_policies and policies are already functioning (the output tensor is only 2 classes now), but still get

Keyword argument `custom_policies` is not a valid argument for this processor and will be ignored.
Keyword argument `policies` is not a valid argument for this processor and will be ignored.

[ain-soph]

And currently it's confusing about the output tensor.

• What's the meaning of each dimension for the output tensor shape?
• What's the meaning of each number and which harmful category is it mapped to? I assume the second column is not necessary since it seems to be 1 - first_column

It would be nice if we have something for explanation in the readme / API doc.

[RyanMullins]

@ain-soph thanks for the contribs. For some reason I can't reply to your comments, but I've added docstrings to address your questions about the shape of the output tensors, and I added an explicit ShieldGemma2ProcessorKwargs to address the logs you saw.

[ghunkins]

When running the default example, an IndexError is raised. Passing in custom_policies fixes that.

Example

from PIL import Image
import requests
from transformers import AutoProcessor, ShieldGemma2ForImageClassification

model_id = "google/shieldgemma-2-4b-it"
model = ShieldGemma2ForImageClassification.from_pretrained(model_id, device_map="auto")
processor = AutoProcessor.from_pretrained(model_id)

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(images=[image], return_tensors="pt").to(model.device)

output = model(**inputs)
print(output.probabilities)

Stack Trace

Loading checkpoint shards: 100%
 2/2 [00:06<00:00,  2.86s/it]
WARNING:accelerate.big_modeling:Some parameters are on the meta device because they were offloaded to the cpu.
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
[<ipython-input-26-290947fc3325>](https://localhost:8080/#) in <cell line: 0>()
     10 image = Image.open(requests.get(url, stream=True).raw)
     11 
---> 12 inputs = processor(images=[image], return_tensors="pt").to(model.device)
     13 
     14 output = model(**inputs)

1 frames
[/usr/local/lib/python3.11/dist-packages/transformers/processing_utils.py](https://localhost:8080/#) in apply_chat_template(self, conversation, chat_template, **kwargs)
   1326 
   1327         if isinstance(conversation, (list, tuple)) and (
-> 1328             isinstance(conversation[0], (list, tuple)) or hasattr(conversation[0], "content")
   1329         ):
   1330             is_batched = True

IndexError: list index out of range

[ArthurZucker]

LGTM! THanks for iterating!

[RyanMullins]

@ghunkins thanks for the bug report!

The problem was that the files on the Hub were out of date and the processor.json was missing the policy_definitions property. I was using updated files locally, which is why I didn't see this until I repro'd your example in Colab. The files are now updated on the Hub.

@ArthurZucker accidentally suggested a local solution to the problem by providing a default set of policies. So this should be fully fixed now and ready for merging.

[ghunkins]

Brilliant, thanks for the amazing work on this @RyanMullins !

[ArthurZucker]

@RyanMullins let's just fix cis! 🤗

[ArthurZucker]

Do you need help on my side for the last remaining tests?

[ain-soph]

nit: I just tested and there is the same warning for policy_definitions.
Some kwargs in processor config are unused and will not have any effect: policy_definitions.