Preprocess an image before passing to a ResNet model

[jianpingliu]

I noticed that the preprocessing and the deprocessing only worked for the VGG model, and it also seems true for Apple's coremltools. In the original fast-neural-style code , the ResNet preprocess is done as

function M.resnet.preprocess(img)
  check_input(img)
  local mean = img.new(resnet_mean):view(1, 3, 1, 1):expandAs(img)
  local std = img.new(resnet_std):view(1, 3, 1, 1):expandAs(img)
  return (img - mean):cdiv(std)
end

while for VGG is done through

function M.vgg.preprocess(img)
  check_input(img)
  local mean = img.new(vgg_mean):view(1, 3, 1, 1):expandAs(img)
  local perm = torch.LongTensor{3, 2, 1}
  return img:index(2, perm):mul(255):add(-1, mean)
end

The difference is that for the ResNet model the image is in the range [0, 1] while for the VGG it is in the range [0, 255].

In your example, to use coremltools' API you defined the preprocessing and deprocessing as

    coreml_model = convert(
        model,
        [input_shape],
        input_names=['inputImage'],
        output_names=['outputImage'],
        image_input_names=['inputImage'],
        preprocessing_args={
            'is_bgr': True,
            'red_bias': -123.68,
            'green_bias': -116.779,
            'blue_bias': -103.939
        },
        image_output_names=['outputImage'],
        deprocessing_args={
            'is_bgr': True,
            'red_bias': 123.68,
            'green_bias': 116.779,
            'blue_bias': 103.939
        },
        unknown_layer_converter_fn=convert_instance_norm
    )

which is natural for the VGG model.

I wonder if there is a way to do the same thing for the ResNet model.

Thanks.

[opedge]

Yes, you can do it. You need to extract mean and std tensors, and use this values for '*_bias' and 'image_scale' keys. Division by std tensor is not fully compatible with CoreML preprocessing, because it accepts only one value for all 3 channels and not 3x1x1 tensor, so if use'll use average value it should be comparable with torch7 output.