Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Normalize input image per color channel #338

Closed
manuelcosta74 opened this issue Sep 20, 2018 · 13 comments

Comments

@manuelcosta74
Copy link

@manuelcosta74 manuelcosta74 commented Sep 20, 2018

Read coremltools documentation, did search through Google and it is not clear to me the "best way" to normalize per color channel. I see that there are bias factors per channel and scale, though, seems to be applied to all.
My model is the same as it was exposed in #337 and they do the following

Python source

from torchvision import transforms


IMAGE_NET_MEAN = [0.485, 0.456, 0.406]
IMAGE_NET_STD = [0.229, 0.224, 0.225]

class Transform:
    def __init__(self):
        normalize = transforms.Normalize(
            mean=IMAGE_NET_MEAN,
            std=IMAGE_NET_STD)

        self._val_transform = transforms.Compose([
            transforms.Resize((224, 224)),
            transforms.ToTensor(),
            normalize])

@property
    def val_transform(self):
    return self._val_transform

...

and before feeding the model

from nima.common import Transform

...

   image = self.transform(image)
    
    image = image.unsqueeze_(0)
    
    image = image.to(device)

So, i see two ways of doing this.

1 - create a custom layer that will do the conversion after the root node

2 - do the normalization in swift or objective-c by transforming CVPixelBuffer

Before putting my hands "dirty" on this, just wanted to know if i'm missing something or this is the reality.

cheers

@manuelcosta74

This comment has been minimized.

Copy link
Author

@manuelcosta74 manuelcosta74 commented Sep 20, 2018

Option 2 should not be as i as said. I think we'll have to replace model input and use MLMultiArray with the normalization processed upstream of the graph.
Don't like much this possibility. It is cleaner to inject something in the graph and keep input clean as image.

@aseemw

This comment has been minimized.

Copy link
Collaborator

@aseemw aseemw commented Sep 20, 2018

The CoreML scale layer can do channel wise multiplication, so can be used for normalizing per color. So you do not need to make a custom layer or normalize the CVPixelBuffer.

The only question is about adding the Scale layer at the beginning of the CoreML model, which can be done either manually using the CoreML builder class or manually adding the layer (since mlmodel is in protobuf format which can be easily edited in python) or if the scale layer is present in the onnx model, the converter should take care of it.

@manuelcosta74

This comment has been minimized.

Copy link
Author

@manuelcosta74 manuelcosta74 commented Sep 20, 2018

Following apple/coremltools#244 and continuing here.

@aseemw suggested to print preprocessing parameters of the converted mlmodel.
https://github.com/apple/coremltools/blob/e8b7655e0ef72a2a40075c6aad3453e253b94fb9/docs/APIExamples.md#printing-the-pre-processing-parameters

I'm converting with code bellow, which was done just to have an idea of how results could look like. Because it is not possible to have image_scale per color channel i used average standard deviation. Results cannot be the same as original model.

import onnx
from onnx_coreml import convert
from onnx import onnx_pb

"""
IMAGE_NET_MEAN = [0.485, 0.456, 0.406]
IMAGE_NET_STD = [0.229, 0.224, 0.225]

"""

scale = 1.0 / (0.226 * 255.0)

args = dict(is_bgr=False, red_bias = -(0.485 * 255.0) * scale  , green_bias = -(0.456 * 255.0) * scale , blue_bia = -(0.406 * 255.0) * scale, image_scale = scale)
        
model_file = open('NIMA_Aesthetics.proto', 'rb')
model_proto = onnx_pb.ModelProto()
model_proto.ParseFromString(model_file.read())
coreml_model = convert(model_proto, image_input_names=['0'], preprocessing_args=args)
coreml_model.save("NIMA_Aesthetics.mlmodel")

Blue channel seems forgotten, which looks like a bug and which is not the issue that was trying to find a solution for.

[featureName: "0"
scaler {
channelScale: 0.0173520743847
greenBias: -2.01769900322
redBias: -2.14601778984
}
]

Although i just have start looking to python deeper this week, it seems that original model does normalization before the graph and we have to be able to do a normalization in coreml with a different scale factor for each channel.

def __init__(self, path_to_model):
    self.transform = Transform().val_transform
    self.model = NIMA(pretrained_base_model=False)
    state_dict = torch.load(path_to_model, map_location=lambda storage, loc: storage)
    self.model.load_state_dict(state_dict)
    self.model = self.model.to(device)
    self.model.eval()

def predict_from_file(self, image_path):
    print("predict_from_file")
    
    image = default_loader(image_path)
    
    print("Image Original", image)
    
    return self.predict(image)

def predict_from_pil_image(self, image):
    image = image.convert('RGB')
    
    return self.predict(image)

def predict(self, image):
    print("predict")
    
    image = self.transform(image)
    
    image = image.unsqueeze_(0)
    
    image = image.to(device)
    
    
    
    with torch.no_grad():
        image = torch.autograd.Variable(image)
        
    prob = self.model(image).data.cpu().numpy()[0] 

Transform class doing normalization

class Transform:
    def __init__(self):
        normalize = transforms.Normalize(
            mean=IMAGE_NET_MEAN,
            std=IMAGE_NET_STD)

        self._val_transform = transforms.Compose([
            transforms.Resize((224, 224)),
            transforms.ToTensor(),
            normalize])

@property
    def val_transform(self):
    return self._val_transform
@manuelcosta74

This comment has been minimized.

Copy link
Author

@manuelcosta74 manuelcosta74 commented Sep 20, 2018

bug on my side... blue channel! :)

@manuelcosta74

This comment has been minimized.

Copy link
Author

@manuelcosta74 manuelcosta74 commented Sep 20, 2018

@aseemw i like the idea of adding scale layer in python.
Do you think that https://github.com/onnx/onnx-coreml/blob/master/tests/_test_utils.py can be a good reference?

@aseemw

This comment has been minimized.

Copy link
Collaborator

@aseemw aseemw commented Sep 21, 2018

just to confirm: is the per channel scale layer present in the exported ONNX model? (you can use Netron to visualize the onnx model)

@manuelcosta74

This comment has been minimized.

Copy link
Author

@manuelcosta74 manuelcosta74 commented Sep 21, 2018

I don't think so. Input = [float32[1,3,224,224]], first layer has a convolution with a kernel 3x3.
onnx model link is available here

https://drive.google.com/file/d/1By-J2aiz9oDYKcq32x8Uxkb0IVyqGqx-/view?usp=sharing

@manuelcosta74

This comment has been minimized.

Copy link
Author

@manuelcosta74 manuelcosta74 commented Sep 25, 2018

@aseemw just tried the example that you provided and i'm getting an error here

coremltools.models.MLModel(spec)

Error

File "<ipython-input-3-fe9385dae025>", line 1, in <module> runfile('###/python/onnx-coreml/tests/onnx2coreml.py', wdir='###/python/onnx-coreml/tests')

File "/anaconda2/envs/Pytorch2CoreML/lib/python2.7/site-packages/spyder_kernels/customize/spydercustomize.py", line 668, in runfile execfile(filename, namespace)

File "/anaconda2/envs/Pytorch2CoreML/lib/python2.7/site-packages/spyder_kernels/customize/spydercustomize.py", line 100, in execfile
builtins.execfile(filename, *where)

File "###/python/onnx-coreml/tests/onnx2coreml.py", line 52, in <module> coreml_model = coremltools.models.MLModel(spec)

File "/anaconda2/envs/Pytorch2CoreML/lib/python2.7/site-packages/coremltools/models/model.py", line 216, in __init__ self.__proxy__ = _get_proxy_from_spec(filename)

File "/anaconda2/envs/Pytorch2CoreML/lib/python2.7/site-packages/coremltools/models/model.py", line 106, in _get_proxy_from_spec
warnings.warn(

NameError: global name 'warnings' is not defined

Don't know why warnings is not defined, but it looks like an exception occurred in _MLModelProxy(filename). Probably again some issue with dependencies / versions...

Code that i'm executing

import copy
import coremltools

from onnx_coreml import convert
from onnx import onnx_pb

"""
IMAGE_NET_MEAN = [0.485, 0.456, 0.406]
IMAGE_NET_STD = [0.229, 0.224, 0.225]

"""

scale = 1.0 / (0.226 * 255.0)
red_scale = 1.0 / (0.229 * 255.0)
green_scale = 1.0 / (0.224 * 255.0)
blue_scale = 1.0 / (0.225 * 255.0)

args = dict(is_bgr=False, red_bias = -(0.485 * 255.0) * red_scale  , green_bias = -(0.456 * 255.0) * green_scale , blue_bias = -(0.406 * 255.0) * blue_scale)
        
model_file = open('NIMA_Aesthetics.proto', 'rb')
model_proto = onnx_pb.ModelProto()
model_proto.ParseFromString(model_file.read())
coreml_model = convert(model_proto, image_input_names=['0'], preprocessing_args=args)

spec = coreml_model.get_spec()

# get NN portion of the spec
nn_spec = spec.neuralNetwork
layers = nn_spec.layers # this is a list of all the layers
layers_copy = copy.deepcopy(layers) # make a copy of the layers, these will be added back later del nn_spec.layers[:] # delete all the layers

# add a scale layer now
# since mlmodel is in protobuf format, we can add proto messages directly
# To look at more examples on how to add other layers: see "builder.py" file in coremltools repo
scale_layer = nn_spec.layers.add()
scale_layer.name = 'scale_layer'
scale_layer.input.append('input1')
scale_layer.output.append('input1_scaled')
params = scale_layer.scale
params.scale.floatValue.extend([red_scale, green_scale, blue_scale]) # scale values for RGB
params.shapeScale.extend([3,1,1]) # shape of the scale vector 

# now add back the rest of the layers (which happens to be just one in this case: the crop layer)
nn_spec.layers.extend(layers_copy)

# need to also change the input of the crop layer to match the output of the scale layer
nn_spec.layers[1].input[0] = 'input1_scaled'

coreml_model = coremltools.models.MLModel(spec)

print(coreml_model.get_spec().description)

coreml_model.save("NIMA_Aesthetics.mlmodel")
@manuelcosta74

This comment has been minimized.

Copy link
Author

@manuelcosta74 manuelcosta74 commented Sep 25, 2018

Seems that comemltools model.py is missing import warnings. After adding it, here is the error

/anaconda2/envs/Pytorch2CoreML/lib/python2.7/site-packages/coremltools/models/model.py:110: RuntimeWarning: You will not be able to run predict() on this Core ML model.Underlying exception message was: Error compiling model: "Error reading protobuf spec. validator error: Layer 'scale_layer' consumes an input named 'input1' which is not present in this network.".
return None

With the warning it is easier to understand the problem. Root node is called '0' in this network and not 'input1'...

@manuelcosta74

This comment has been minimized.

Copy link
Author

@manuelcosta74 manuelcosta74 commented Sep 25, 2018

Code that seems to convert onnx model made available in a previous post which was a conversion of NIMA based on MobileNetv2 architecture. There seems to be a small difference relative to the original model, but that is probably a precision issue.

Original model can be found here https://github.com/truskovskiyk/nima.pytorch

import copy
import coremltools
import os

from onnx_coreml import convert
from onnx import onnx_pb

"""
IMAGE_NET_MEAN = [0.485, 0.456, 0.406]
IMAGE_NET_STD = [0.229, 0.224, 0.225]

"""

scale = 1.0 / (0.226 * 255.0)
red_scale = 1.0 / (0.229 * 255.0)
green_scale = 1.0 / (0.224 * 255.0)
blue_scale = 1.0 / (0.225 * 255.0)

args = dict(is_bgr=False, red_bias = -(0.485 * 255.0)  , green_bias = -(0.456 * 255.0)  , blue_bias = -(0.406 * 255.0))
      
model_file = open('NIMA_Aesthetics.proto', 'rb')
model_proto = onnx_pb.ModelProto()
model_proto.ParseFromString(model_file.read())
coreml_model = convert(model_proto, image_input_names=['0'], preprocessing_args=args)

spec = coreml_model.get_spec()

# get NN portion of the spec
nn_spec = spec.neuralNetwork
layers = nn_spec.layers # this is a list of all the layers
layers_copy = copy.deepcopy(layers) # make a copy of the layers, these will be added back later
del nn_spec.layers[:] # delete all the layers

# add a scale layer now
# since mlmodel is in protobuf format, we can add proto messages directly
# To look at more examples on how to add other layers: see "builder.py" file in coremltools repo
scale_layer = nn_spec.layers.add()
scale_layer.name = 'scale_layer'
scale_layer.input.append('0')
scale_layer.output.append('input1_scaled')

params = scale_layer.scale
params.scale.floatValue.extend([red_scale, green_scale, blue_scale]) # scale values for RGB
params.shapeScale.extend([3,1,1]) # shape of the scale vector 

# now add back the rest of the layers (which happens to be just one in this case: the crop layer)
nn_spec.layers.extend(layers_copy)

# need to also change the input of the crop layer to match the output of the scale layer
nn_spec.layers[1].input[0] = 'input1_scaled'

print(spec.description)

coreml_model = coremltools.models.MLModel(spec)

coreml_model.save("NIMA_Aesthetics.mlmodel")
@manuelcosta74

This comment has been minimized.

Copy link
Author

@manuelcosta74 manuelcosta74 commented Sep 25, 2018

@aseemw just a final doubt.
How to configure shapeBias in "scale_layer" as alternative to preprocessing_args?

@aseemw

This comment has been minimized.

Copy link
Collaborator

@aseemw aseemw commented Sep 25, 2018

I didn't quite get your question :)
What do mean by "configuring" shapeBias?
If you mean removing the pre-processing arguments and just adding the Bias parameter in the scale layer, yes that can be done. Adding the bias parameter is very similar to adding the Scale parameter. See: https://github.com/apple/coremltools/blob/d07421460f9f0ad1a2e9cf8b5248670358a24a1a/coremltools/models/neural_network/builder.py#L839

@manuelcosta74

This comment has been minimized.

Copy link
Author

@manuelcosta74 manuelcosta74 commented Sep 25, 2018

Thanks @aseemw for all the help.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.