pixelBuffer normalization function?

[appsird]

Matt,

Thanks much for the ML library. You always write such elegant code.

Let's assume I have the following, though (as the comment shows), I want to normalize all samples of the pixelBuffer. What is the best means to accomplish this normalization pixel normalization?

let image = UIImage(...)

// Convert the image
if let pixelBuffer = image.pixelBuffer(width: 224, height: 224) {

  // normalize all samples in image.pixelBuffer

  // Make the prediction with Core ML
  if let prediction = try? model.prediction(input: pixelBuffer) {
    print(prediction)
  }
}

[hollance]

Why do you need to do this? It's better to let Core ML handle this for you.

[appsird]

New to this arena, so excuse my lack of understanding.

If my model is normalized via CoreML, input images must be similarly normalized? What is the best/easiest means to normalize a UIImage?

Brian

[hollance]

When you convert your model to Core ML you tell it how to normalize the image (by passing in the appropriate parameters). Then in your app you just use a regular CVPixelBuffer and Core ML takes care of the normalization. Even easier is using Vision, which also resizes the image if necessary.

[manuelcosta74]

Hi @hollance

Google just showed me this thread while searching image normalization and CoreML. Write now i'm in the process to convert a pytorch model through onnx. Original model normalizes each channel individually by subtracting average and dividing by standard deviation.
In this scenario it is not clear yet to me how to do it with coremltools through bias and scale factors. If it is not, @appsird makes makes sense.

[hollance]

The bias is for subtracting, and you can provide a separate bias for each of the 3 channels. The scale is for dividing by the standard deviation.

See also: http://machinethink.net/blog/help-core-ml-gives-wrong-output/

[manuelcosta74]

ok, but that model that i'm using has a standard deviation per channel.

IMAGE_NET_MEAN = [0.485, 0.456, 0.406]
IMAGE_NET_STD = [0.229, 0.224, 0.225]

This means that
normalizedValRed = (red - red_avg) / red_stddev
With bias you can do red - red_avg. Scale, though, i did not figure out yet how to apply it per channel.

[manuelcosta74]

Wait. This makes no sense.
What eventually makes sense is to replace input model and use MLMultiArray with normalized values. In the end probably the best is to keep input as image and add a custom layer to do the normalization.

[hollance]

Yeah it gets a bit trickier. But really I would just use 0.225 for all of them since they're so similar it probably doesn't matter.

[manuelcosta74]

That is a dangerous assumption...
I'm moving now to the snakes world and investigating how to add custom layers to the graph through coremltools. I really don't understand why Apple does not provide this type of normalization. All python frameworks have it and, together with (x-min)/(max - min), (x-avg)/stdev is one of the most common normalizations adopted in data mining methods. Doing it per color channel seems quite reasonable.

[hollance]

You're going to lose more precision due to 16-bit floating point precision issues than because of a difference of 0.001 or 0.004 in the standard deviation. So, I wouldn't lose any sleep over it.

You could probably achieve what you want by adding a ScaleLayerParams as the first layer in the model, although I'm not 100% sure that it accepts a different scale factor per channel.

[manuelcosta74]

@hollance and folks reading this.
There is an example now in coremltools how to scale per channel

https://github.com/apple/coremltools/blob/master/examples/Image_preprocessing_per_channel_scale.ipynb

cheers

[hollance]

Nice, thanks for finding this!

[manuelcosta74]

This was created a few days ago and added to master yesterday. Here you can find the flow

onnx/onnx-coreml#338