Processor num_crops=0 causing "ValueError: height and width must be > 0"

[hm-ca]

Hi Team,

When I set num_crops=0 I get the error: "ValueError: height and width must be > 0". Full error is below....
my images are square and small (less than 336x336) already and I don't want to crop them any further for training in order to keep the memory footprint in check and to be able to pack as many together in my batch as possible.

Anyway, I feel that num_crops=0 should be an acceptable value.

Moreover, I noticed while displaying the crops returned by the processor that for num_crops=1, the returned crop is always blank which almost double the memory requirements per sample/batch for no good reason. (double because the processor returns the original image and the blank crop)

Also, based on additional and limited testing, only values like num_crops = 4 and 16 appear to return actual non blank crops!

What am I missing?

Thanks!

=================================
Full error is here:
File "/home/ubuntu/content/mycode-finetune-phi3v.py", line 669, in call
inputs = self.processor(prompt, frames, return_tensors='pt')
File "/home/ubuntu/.cache/huggingface/modules/transformers_modules/microsoft/Phi-3-vision-128k-instruct/866d1691437a49af79d5f3ad4a34c1750e08d163/processing_phi3_v.py", line 113, in call
image_inputs = self.image_processor(images, return_tensors=return_tensors)
File "/home/ubuntu/.local/lib/python3.10/site-packages/transformers/image_processing_utils.py", line 551, in call
return self.preprocess(images, **kwargs)
File "/home/ubuntu/.cache/huggingface/modules/transformers_modules/microsoft/Phi-3-vision-128k-instruct/866d1691437a49af79d5f3ad4a34c1750e08d163/image_processing_phi3_v.py", line 245, in preprocess
elems = [HD_transform(im, hd_num = self.num_crops) for im in images]
File "/home/ubuntu/.cache/huggingface/modules/transformers_modules/microsoft/Phi-3-vision-128k-instruct/866d1691437a49af79d5f3ad4a34c1750e08d163/image_processing_phi3_v.py", line 245, in
elems = [HD_transform(im, hd_num = self.num_crops) for im in images]
File "/home/ubuntu/.cache/huggingface/modules/transformers_modules/microsoft/Phi-3-vision-128k-instruct/866d1691437a49af79d5f3ad4a34c1750e08d163/image_processing_phi3_v.py", line 82, in HD_transform
img = torchvision.transforms.functional.resize(img, [new_h, new_w],)
File "/home/ubuntu/.local/lib/python3.10/site-packages/torchvision/transforms/functional.py", line 490, in resize
return F_pil.resize(img, size=output_size, interpolation=pil_interpolation)
File "/home/ubuntu/.local/lib/python3.10/site-packages/torchvision/transforms/_functional_pil.py", line 250, in resize
return img.resize(tuple(size[::-1]), interpolation)
File "/home/ubuntu/.local/lib/python3.10/site-packages/PIL/Image.py", line 2008, in resize
return self._new(self.im.resize(size, resample, box))
ValueError: height and width must be > 0

[leestott]

Hi @hm-ca Sorry could you confirm which sample your trying? Also please include the Python code your trying to execute?

[leestott]

@hm-ca the sample we provide for Finetuning Phi-3 vision is available at https://github.com/microsoft/Phi-3CookBook/blob/main/md/04.Fine-tuning/FineTuning_Vision.md

[hm-ca]

Hi @leestott,

Below is the code to reproduce the 3 processor/crops issues I'm facing:

1. with num_crops = 0, the processor returns the "ValueError: height and width must be > 0" error.
2. with num_crops = 1, the processor created an unnecessary crop that is identical to the global image. Basically doubling the pixel_values size? TBV
3. for many other num_crops values, there are many blank images returned. (see attached image)., again waist of space in pixel_values that will make the batch size during training much bigger without any benefit,

===============================

from transformers import (
    AutoModelForCausalLM,
    AutoProcessor,
    BitsAndBytesConfig,
    Trainer,
    TrainingArguments,
)
from PIL import Image
import requests
from torchvision.transforms import ToPILImage
import matplotlib.pyplot as plt

model_name_or_path = 'microsoft/Phi-3-vision-128k-instruct' # Model name or path to load from

user_prompt = '<|user|>\n'
assistant_prompt = '<|assistant|>\n'
prompt_suffix = "<|end|>\n"
prompt = f"{user_prompt}<|image_1|>\nWhat is shown in this image?{prompt_suffix}{assistant_prompt}"
url = "https://upload.wikimedia.org/wikipedia/commons/thumb/b/bd/Tesla_Motors.svg/240px-Tesla_Motors.svg.png" # 240 × 310 which is smaller than 336x336
print(f">>> Prompt\n{prompt}")
image = Image.open(requests.get(url, stream=True).raw)
print(f"Original Image, size: {image.size}")
display(image)

for num_crops in range(1, 17):  # Iterate over different numbers of crops
    print(f"\nNumber of Crops: {num_crops}")
    print("="*20)

    # Load processor
    processor = AutoProcessor.from_pretrained(model_name_or_path, trust_remote_code=True, num_crops=num_crops)

    inputs = processor(prompt, image, return_tensors="pt").to("cuda:0")
    pixel_values = inputs['pixel_values']
    print(f'pixel_values.shape: {pixel_values.shape}')

    # Display each crop as an image
    fig, axs = plt.subplots(1, pixel_values.shape[1], figsize=(15, 5))
    fig.suptitle(f'Number of Crops: {num_crops}', fontsize=16)

    for crop_idx in range(pixel_values.shape[1]):  # Iterate over crops
        crop_tensor = pixel_values[0, crop_idx].to(torch.float32)  # Convert to float32
        crop_image = ToPILImage()(crop_tensor.cpu())
        
        # Print the crop number and size
        print(f"Crop #{crop_idx + 1}, size: {crop_tensor.shape}")
        
        # Display crop image
        axs[crop_idx].imshow(crop_image)
        axs[crop_idx].set_title(f"Crop #{crop_idx + 1}")
        axs[crop_idx].axis('off')
    
    plt.show()

[hm-ca]

Hi @leestott,

Below is the code to reproduce the 3 processor/crops issues I'm facing:

1. with num_crops = 0, the processor returns the "ValueError: height and width must be > 0" error.
2. with num_crops = 1, the processor created an unnecessary crop that is identical to the global image. Basically doubling the pixel_values size? TBV
3. for many other num_crops values, there are many blank images returned. (see attached image)., again waist of space in pixel_values that will make the batch size during training much bigger without any benefit,

===============================

from transformers import (
    AutoModelForCausalLM,
    AutoProcessor,
    BitsAndBytesConfig,
    Trainer,
    TrainingArguments,
)
from PIL import Image
import requests
from torchvision.transforms import ToPILImage
import matplotlib.pyplot as plt

model_name_or_path = 'microsoft/Phi-3-vision-128k-instruct' # Model name or path to load from

user_prompt = '<|user|>\n'
assistant_prompt = '<|assistant|>\n'
prompt_suffix = "<|end|>\n"
prompt = f"{user_prompt}<|image_1|>\nWhat is shown in this image?{prompt_suffix}{assistant_prompt}"
url = "https://upload.wikimedia.org/wikipedia/commons/thumb/b/bd/Tesla_Motors.svg/240px-Tesla_Motors.svg.png" # 240 × 310 which is smaller than 336x336
print(f">>> Prompt\n{prompt}")
image = Image.open(requests.get(url, stream=True).raw)
print(f"Original Image, size: {image.size}")
display(image)

for num_crops in range(1, 17):  # Iterate over different numbers of crops
    print(f"\nNumber of Crops: {num_crops}")
    print("="*20)

    # Load processor
    processor = AutoProcessor.from_pretrained(model_name_or_path, trust_remote_code=True, num_crops=num_crops)

    inputs = processor(prompt, image, return_tensors="pt").to("cuda:0")
    pixel_values = inputs['pixel_values']
    print(f'pixel_values.shape: {pixel_values.shape}')

    # Display each crop as an image
    fig, axs = plt.subplots(1, pixel_values.shape[1], figsize=(15, 5))
    fig.suptitle(f'Number of Crops: {num_crops}', fontsize=16)

    for crop_idx in range(pixel_values.shape[1]):  # Iterate over crops
        crop_tensor = pixel_values[0, crop_idx].to(torch.float32)  # Convert to float32
        crop_image = ToPILImage()(crop_tensor.cpu())
        
        # Print the crop number and size
        print(f"Crop #{crop_idx + 1}, size: {crop_tensor.shape}")
        
        # Display crop image
        axs[crop_idx].imshow(crop_image)
        axs[crop_idx].set_title(f"Crop #{crop_idx + 1}")
        axs[crop_idx].axis('off')
    
    plt.show()

[leestott]

@ChenRocks Any ideas on this issues/comment?

[ChenRocks]

Hi @hm-ca thanks for reporting. In short, this is the expected behavior. Please see the inline detailed response below.

Hi @leestott,

Below is the code to reproduce the 3 processor/crops issues I'm facing:

1. with num_crops = 0, the processor returns the "ValueError: height and width must be > 0" error.

I'd consider num_crops == 0 a different model (a model not using HD images). Therefore this wasn't supported in the first place. Not sure if this can't be implemented with minimal code change but I'd rather keep things simpler.

2. with num_crops = 1, the processor created an unnecessary crop that is identical to the global image. Basically doubling the pixel_values size? TBV

Yes, there will be one global image and one HD image of 1x1 (which is essentially the same as the global image). I'd assume those who wish to tune num_crops understand this consequence.

3. for many other num_crops values, there are many blank images returned. (see attached image)., again waist of space in pixel_values that will make the batch size during training much bigger without any benefit,

The blank images are paddings, see here. Since the example you used is a near square image, the resulting HD image will always be squared (1x1, 2x2, ... 4x4). The num_crops hyper-parameter specifies the upper bound of total HD patches, i.e., W * H <= num_crops for the W x H HD image (W and H being integers). If you try images with more extreme aspect ratio, say 3:1, you should see the effects.

I hope this answers your questions.

[leestott]

Closing this issue based on explanation and example

[hm-ca]

Thanks @ChenRocks and @leestott for the detailed explanation.

In this case I'll try to filter those additional crops in my code given the smaller image size I'm processing.

Best