Custom dataset

[cyrilzakka]

Hello,

I have a bit of a technical question - I have a dataset of 20 million image caption pairs with a healthy distribution of pngs, jpegs, jpgs and tiffs. Should I convert all of them to JPEG before converting them to a webdataset? Also they're all of different sizes - should I resize them to 224? I'd rather crop them to 224 during training.

[rom1504]

You have indeed 2 choices 1. Prepare data during data loading 2. Prepare data in advance The benefit of preparing data in advance are 1. Saving compute during training: by having uniform sample and shard sizes. 2. Reducing chances of bad data creeping in I'd recommend you * convert it all to jpg * resize all images bigger than smallest length 224 to 224xH or Wx224 You can totally do it all during data loading but it may make things more complicated

…

On Thu, Aug 24, 2023, 22:35 Cyril Zakka, MD ***@***.***> wrote: Hello, I have a bit of a technical question - I have a dataset of 20 million image caption pairs with a healthy distribution of pngs, jpegs, jpgs and tiffs. Should I convert all of them to JPEG before converting them to a webdataset? Also they're all of different sizes - should I resize them to 224? I'd rather crop them to 224 during training. — Reply to this email directly, view it on GitHub <#605>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAR437SJ5TFUVOUVLGSN7P3XW63JLANCNFSM6AAAAAA35R7DBY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[gabrielilharco]

If you do decide to do everything at data loading time, be sure to change this code https://github.com/mlfoundations/open_clip/blob/main/src/training/data.py#L174, so tiff files don't get filtered out. We currently read only from png, jpg, jpeg or webp.

[cyrilzakka]

@rom1504 Thank you! Do you mean that if the width of an image is greater than 224 pixels, and the width is the larger dimension, the image should be resized to have a width of 224 pixels, and the height should be adjusted proportionally to maintain the aspect ratio. And similarly for if height is > 224?

i.e.

width, height = img.size
if width > 224 or height > 224:
                if width > height:
                    new_width = 224
                    new_height = int((new_width / width) * height)
                else:
                    new_height = 224
                    new_width = int((new_height / height) * width)
                img_resized = img.resize((new_width, new_height), Image.ANTIALIAS)
                img_resized.save(new_path, 'JPEG')

[rom1504]

no smallest length, not larger length

eg put an and in your first line

that way if you then center resize to 224x224 you won't need to upscale only crop

[cyrilzakka]

I'm sorry do you mind clarifying the last part a bit?

[rwightman]

@cyrilzakka

• resize longest means longest edge gets scaled to the target size, so the shortest edge will be < than the target size and padding needs to be used, this is wanted if you need to fit the full image into the model, but often scales it down too much for these small sizes like 224, esp if there are large aspect ratios
• resize shortest (the default in pytorch Resize() if you just pass in a single int), resizes the shortest edge to target size, longest edge will be > target so you typically need to crop after for a square output (instead of pad)

I wrote a pair of transforms a little while back that lets you slide between fully shortest and fully longest with a float input, and then a crop or pad fn that fills and crops as needed, works for both square and non-square target sizes...
https://github.com/huggingface/pytorch-image-models/blob/main/timm/data/transforms.py#L256-L342

[rwightman]

It's usually best to resize your dataset to something a step away from your target, to give you a bit of room for augmentation and training at different resolutions w/o rewriting your dataset for each run. This is a tradeoff decision.

If you leave images large though, it can be VERY expensive to resize on the fly, if you had 20MP digital camera images, really large medical scans, it just hammers your CPU to downscale to 224, etc.

Moving to discussions for future reference.

[cyrilzakka]

Thanks @rwightman! I ended up going with something similar to resize longest - I just had trouble understanding the term. Also I did notice a small bug here:

open_clip/src/open_clip/transform.py

Line 34 in f692ec9

self.fn = min if fn == 'min' else min

I think it should be: self.fn = min if fn == 'min' else max instead

[rwightman]

Indeed, but fn is not even used :O looks one of us was part way through some changes...