Empty output when using facebook/sam-vit-base with automatic mask generation pipeline

[sunhaozhepy]

System Info

• transformers version: 4.35.0.dev0
• Platform: Linux-5.4.15-1.el7.elrepo.x86_64-x86_64-with-glibc2.27
• Python version: 3.11.4
• Huggingface_hub version: 0.17.3
• Safetensors version: 0.4.0
• Accelerate version: 0.23.0
• Accelerate config: not found
• PyTorch version (GPU?): 2.1.0+cu121 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: yes
• Using distributed or parallel set-up in script?: yes (but I grabbed 1 GPU only to my workspace bash terminal)

Who can help?

@amyeroberts @Narsil

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Following the example notebook of automatic mask generation using SAM model, I used the following code snippet to do image segmentation:

import numpy as np
import matplotlib.pyplot as plt
import gc
from transformers import pipeline
from PIL import Image
import requests

def show_mask(mask, ax, random_color=False):
    if random_color:
        color = np.concatenate([np.random.random(3), np.array([0.6])], axis=0)
    else:
        color = np.array([30 / 255, 144 / 255, 255 / 255, 0.6])
    h, w = mask.shape[-2:]
    mask_image = mask.reshape(h, w, 1) * color.reshape(1, 1, -1)
    ax.imshow(mask_image)
    del mask
    gc.collect()

def show_masks_on_image(raw_image, masks):
  plt.imshow(np.array(raw_image))
  ax = plt.gca()
  ax.set_autoscale_on(False)
  for mask in masks:
      show_mask(mask, ax=ax, random_color=True)
  plt.axis("off")
  plt.show()
  del mask
  gc.collect()

generator = pipeline("mask-generation", model="facebook/sam-vit-base", device=0)

raw_image = Image.open("test.png")

outputs = generator(raw_image, points_per_batch=64)

masks = outputs["masks"]
show_masks_on_image(raw_image, masks)

The original code is provided in the form of a notebook, which I arrange as a script that can be run in the terminal.

Expected behavior

The output is empty, and no mask is shown on the image. When I insert print(outputs) into the code snippet, the output in the terminal is {'masks': [], 'scores': tensor([])}.

[LysandreJik]

cc @rafaelpadilla @younesbelkada

[younesbelkada]

Hi @sunhaozhepy
Thanks for the issue, I have just tried it with a custom image and it seems to work great on my end. Can you maybe share the image with us so that we can reproduce and try to understand what is going on?

[rafaelpadilla]

@sunhaozhepy, I also couldn't reproduce the error.

I tested your code with this image:

img_url = "https://huggingface.co/ybelkada/segment-anything/resolve/main/assets/car.png"
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert("RGB")

Which resulted in:

len(outputs["masks"]), len(outputs["scores"])
# 48, 48

If you share your test.png, we could track down the issue.

[sunhaozhepy]

Hi, thanks for your replies! Here's my image test.png, but I strongly suspect that the image itself is not the most critical issue...

Also, I forgot to mention that for the script to be runnable in the terminal, I added a line plt.savefig("masked_image.png") before del mask to save the masked image. It looks like this:

Basically the same, without masks.

And here's an error image when I run this in my environment (masked image still saved):

Traceback (most recent call last):
  File "/data/jupyter/qbs_lora/sam_auto.py", line 39, in <module>
    show_masks_on_image(raw_image, masks)
  File "/data/jupyter/qbs_lora/sam_auto.py", line 28, in show_masks_on_image
    del mask
        ^^^^
UnboundLocalError: cannot access local variable 'mask' where it is not associated with a value

When I ran this script in colab, neither did I get empty masks, nor the error image.

[rafaelpadilla]

Hi @sunhaozhepy,

This does not seen a problem related to the model nor the transformers library. However, I tried to replicate the error using your image and, by my side, the code finishes its execution normally..

For your given image, 15 masks were found:

len(outputs["scores"]), len(outputs["masks"])
# 15, 15

The mask object only exists if the for loop is executed. So, if your masks list is empty ([]), the mask object is never created, and will raise this error.

This is likely a problem related to your environment.

[sunhaozhepy]

Ok then...Do you have any suggestions for me to fix my environment? Otherwise I can try to do my experiment in Colab or try another model.

[rafaelpadilla]

I checked the SAM example notebook (here) to find if there's a memory-related explanation to delete mask it call GC, but it is a different code.

In your code, first you call del mask inside the function show_mask. Then in show_masks_on_image you're trying todel mask again, but mask was has already been deleted.

Any reason why you are calling del mask and gc.collect() in show_masks_on_image? I think it might not be necessary.

[sunhaozhepy]

It's here.

Never mind the environment, I've continued my experiment on Colab, and the experiment did not turn out to have good outcomes. I'll try something else...