Implementing image reading functions

[ShahriarRezghi]

Proposal

We must have a function that reads images and converts it to a tensor in order to implement C++ datasets. I have written the said function here with OpenCV. But using OpenCV just to read images is an overkill.

I suggest that we implement functions that read a specific format (for example a function that reads png images using libpng). We can cover the formats that are used in the datasets that will be implemented.

[fmassa]

I think this goes in the right direction.

A few things we should also discuss:

• what should be the format that we will return from the image loading?
  • Should it be CxHxW or HxWxC?
    • The images are natively stored in HxWxC, but the default format we use in PyTorch is CxHxW
  • Should it be uint8 in 0-255 or float in 0-1?
    • we had plenty of complaints about weird handling of to_tensor and to_pil_image in many cases, we should try as much as possible to be consistent.
• Can we make it work on both Windows and Linux / OSX?
  • if yes, then it would be a candidate for potentially replacing PIL in the future
• should we have a separate function for reading and decoding?
  • we might want to have in-memory encoded images as a bytes string, and just decode it when needed
  • this means we might want to have a decode_image function, similar to TF, which is also supported by PIL
• What formats do we want to support? JPEG / PNG / others?
• What about images encoded with other data types, like int16 or float?

[ShahriarRezghi]

Here is what I think:

• Input image: I think that it's better if we use 0-1 float values in our tensors because they are eventually turned into float when training and such. This way other data types like int16 and float can be turned into float. Also I have currently implemented CxHxW. What is the point of keeping HxWxC?
• Separating reading and decoding: I haven't used tensorflow so I don't know what are the benefits of separating reading and decoding of images.
• Image formats: In the link that you provided tensorflow supports BMP, GIF, JPEG, PNG. I think that for now we should support formats that are enough for reading the image of datasets that are implemented.

[fmassa]

I gave it a second thought, and I'm leaning towards the following:

Image format

float 0-1 vs original format

There have been a lot of feature requests from users about extending support for to_tensor and to_pil_image to handle particular edge cases. Those edge cases exist because of the implicit assumption that float should be in 0-1. But this breaks some things when people want to use uint16 images, which have larger range of values.

The original reason why we convert the images to 0-1 format is because we used to feed them directly as is to the network (with sigmoid / tanh activations), so having the image be in 0-1 would help training.

But since a while, we have been performing an image normalization step which consists of dividing by the std of the values. This means that, if we use such normalization, it doesn't matter if the float tensor is in 0-1 or 0-255, all it matters is that the normalization values are compatible with how the images are loaded.

This makes me think that we might want to have the images be returned the original range, be it 0-255 or larger for larger types.
I think we should also return the data in the standard type they have been stored (uint8 for most cases), because it uses less space and many operations can be efficiently performed on uint8 types.

CHW x HWC

I'm not 100% clear on this one. Maybe stick with HWC and let a transform perform the conversion?

reading and decoding

The benefits of separating reading and decoding is that we have more flexibility.
If the API only supports passing a file path, then we can't decode a bytestream that lives (encoded) in memory. I think this is an important requirement.

Image Formats

Agree that we should support the formats that are necessary for reading the images of the datasets that we implement.

[t-vi]

I'm far from an expert but balancing the desire

• to have something that is "standard" (no dependency on file format, backend library, ...) as output by default,
• to be efficient (i.e. have something that is ideally "zero copy" after returning from the reading library), not lose information in preprocessing.

The cleanest way perhaps would be having a "one size fits all" recommended image reading function that returns a fixed dtype / layout. And then maybe "raw_read_someformat". A "raw" option could be another way, but I must admit I cringe thinking about the return dtype depending on an option.

I would probably be OK with non-contiguous layouts to be returned if it reduces the copying going on - as Francisco said, we usually have a normalization step and that would typically deal with non-contiguous inputs.

[ShahriarRezghi]

So adding it all together:

• Supporting the image formats of our datasets.
• Having a simple function that reads files into memory.
• Having a function that reads raw image from memory and keeps type and size and HWC. Also having a function that reads all the formats desired and returns a fixed type and size but actually using the raw functions to read the datasets.

@fmassa @t-vi does this make sense?
I can start working on this after all of this is 100%.

[ShahriarRezghi]

OK. So these are the results and I have started implementing. Closing this.

[jguerin-fb]

what should be the format that we will return from the image loading?
Should it be CxHxW or HxWxC? - match image library format by default (e.g. PIL) with option to specify the format

Should it be uint8 in 0-255 or float in 0-1? This should be uint8. Eventually we'll support quantized/low precision training that won't be float. Also, type conversions are ideally abstracted away / automatically done only when needed.

What formats do we want to support? JPEG / PNG / others? - by default support as many formats as possible, but allow for specific formats to be specified as an optimization.

[fmassa]

I'm reopening this issue until we have the PRs merged in torchvision.

And I agree with @jguerin-fb comments, thanks for the feedback!